For the past month, I've been working on getting the latest version of my company's flagship website up and running. The part that personally caused me the most grief was the commerce-related scripts. I spent an entire weekend stumbling through them before it became clear that even though I'd worked with asynchronous functions in JavaScript before, I had no idea what I was doing.

Golden retriever wearing a necktie sitting at a computer: "I have no idea what I'm doing"

My boss realized what the trouble was, and the following Monday, she sat me down and gave me a twenty-minute rundown of SQL transactions, JavaScript promises and how they're related to ES7's new async/await function, and how to properly handle async errors using the try/catch/finally pattern. While she was at it, she showed me how the famed Express router's error handling works, and all the gotchas that come with it. I'm pretty sure she saved me months of my future life trying to figure this stuff out.

My main takeaways were (in ascending order of confusion):

  • Error handling
    • I seriously did not know how to throw an error within a try/catch. It's literally throw 'Error message'.
    • If you're going to pass any variables besides an error into a catch, declare it outside of the try/catch and then set it within the try. Otherwise catch doesn't know about anything that happens within try.
  • Promises
    • The function you're waiting for should have a try/catch in which you invoke resolve() upon a success and reject() upon catching an error. (Don't forget to pass those functions in from the Promise invocation.)
    • You can avoid true callback hell by using then/catches (like this one: promiseFunction().then().catch((error)=>{})) infinite times in what's called a "promise chain".
    • Promise chains are easier to work with than callback hell, but they're still only one step removed from callback hell.
  • Async/Await
    • async is basically a promise wrapper that prevents you from having to deal with all the .then().catch().then() of promise chain error handling, further shielding you from the horrors of callback hell.
    • Instead, you keep functions to await in a try/catch, and they are run chronologically, like normal functions.
    • You can technically await straight promises, but you shouldn't need to...
    • ...unless you're awaiting the end of a string of promises that are called asynchronously, in which case you should put them in an array in await Promise.all();, because the corresponding await* function didn't make it into the ES7 release.
  • Express.js with Async/Await + Promises
    • For every CRUD operation, you should only have one async function that returns a promise, and that's for the request that hits the server endpoint.
    • Every other client-side async function only awaits these request functions. There should be no other promises in your data-related code, with the exception of Promise.all() for the reason mentioned earlier.
  • SQL Transactions and Try/Catch/Finally
    • On the server side, instead of doing a series of straight SQL queries that change the database, you can open a database connection in a try statement and create the series of queries, then commit the whole transaction only at the very end, after everything else checks out.
    • If any part of the transaction fails, you can catch the error and rollback the entire transaction so that no part of it ever happens.
    • (And in the finally, if you still have a connection, you should release it.)
    • This allows you to make several related queries at once without being afraid that only some of them will go through, which could mess up your database.
    • Non-database functions that are harder to undo, like order or email sending, should be at the very end of the try statement, just before you commit the transaction. So if any other part of the try fails, those critical functions won't be run.
  • Some serious Express gotchas
    • The error handler is the first function passed into the router with four parameters (error, request, response, next) instead of three (request, response, next).
    • If you're passing anything into your next() function, it's read as an error and it'll go straight to the error handler.

One of the reasons I took this job was because during my interview, everyone I talked to taught me something new. I've been learning a lot from people here who are not only very knowledgeable, but willing to share their knowledge with others. Hopefully I'll get to a point where I can pay it forward more often.

trekking through the amazon

A few months ago, I learned about the t2.nano, a new Amazon EC2 instance that was cheaper than my old t2.micro ($5 a month versus $10), so I tried switching over to it and promptly bricked the whole thing, which was why this blog wasn't working for a while.

I started a new job that required me to learn a lot more about Amazon services, so this week I decided to take another crack at migrating my blog instance over. This time I even properly documented everything. It took several long evenings (and a panicked rewriting of my commit history after I pushed up some config files to GitHub that I really shouldn't have, which resulted in my emailer promptly getting hacked and turned into a spamming device), but it's now on the smaller nano instance and purring like a kitten.

While cussing at my laptop, I learned a lot about:

  • SSHing into stuff and making that process easier
  • How RSA keys are handled between hosts
  • Cloning Amazon EC2 instances and EBS volumes
  • Copying files to and from remote hosts (scp)
  • Obscure Git commands that you only need to know when you've seriously ****ed some **** up

    I really ****ed some **** up I really ****ed some **** up

This was all basic web deployment stuff I've been wanting to get deeper into, since my main goal as a web developer is to be able to create and run applications by myself. While the frontend has always been my jam (I like making things people can immediately use), I figure that if I can be a one-stop shop, I'll be of much more use to smaller places that really need a web presence, such as nonprofits and small businesses.

People have already suggested other improvements I can make to the blog, like adding encryption. I could also make my Ghost instance a static site using Buster (a.k.a. 'Ghost Buster'), since storing it in an Amazon S3 bucket would be even cheaper-- around $3 a month. I'll consider some of these options. But before that, I'd like to just enjoy having it back up and running.

It was only a couple of years ago that I became a web developer, but in that time, it feels like I went to college again and learned twice as much as I had the first time around. I started this blog in 2014; I'm kind of surprised that I managed to do it, since I was blindly copy-pasting terminal commands from Stack Overflow and hoping for the best. I know so much more about how this all works now.

Edit: I still blindly copy-paste terminal commands from Stack Overflow and hope for the best, but not as often as I used to.

another year, another bot

My 2015 has been filled with long and arduous international and domestic travel, plenty of family business, a cross-country move, and a deferred to-do list a mile high. I haven't been keeping regular hours since the beginning of the year, so I'm glad that the dust has settled and I have a little free time to learn and work on new things again.

I finally made a version of the bang bot that is police-oriented (and less pervy-sounding). The Washington Post's fatal police shootings database is reliably updated, and it's on GitHub so it was much easier for me to get raw data from (otherwise I would have to do some manual scraping-- not hard, just time-consuming).

The bot's checking mechanism was the simplest one I could come up with:

  1. Pull the latest .csv file on WaPo's GitHub
  2. Check if the number of lines is greater than that of the last stored version, and
  3. Tweet that number if it is.

The implementation's still got a few kinks (apparently you can't use Twitter's API to tweet the same thing twice, which may be a problem soon), but my previous experience with Node made building it fairly easy.

It was the deployment that gave me some trouble. I used Heroku to run my app this time, and it took me a while to realize that the reason my files weren't properly saving was because Heroku workers don't do data persistence (derp). So I had to pick a data storage addon. I would have liked to use MongoDB to make the app JS end-to-end, but it wasn't free. And I'm pretty familiar with Postgres after having used it for a class earlier this year, but I ultimately went with Redis, another NoSQL choice that's fast and has an easy-to-use Node.js client. I hear that relational databases are the way to go if you have a data structure that's even remotely complicated, but I only had to store two things.

A screenshot of callback hell. Can you tell I'm using Node?

Besides storage, I still haven't quite gotten the hang of JavaScript's asynchronous processes yet (please refer to the callback hell in the screenshot above). I tried to keep things readable, but I think I'm gonna just bite the bullet next time and use async.

The reason I started this project was to get an update on Twitter whenever WaPo confirms another death by police shooting, and I'm glad I reached that goal. But seeing the bot tweet for the first time on production was the first time I had ever felt sad about seeing something I built actually work.

Now that I'm getting into full-stack development, I'm exploring what else I can do with open data. My final project for the back-end class that I took this spring used data made available by San Francisco's Human Services Agency. Similar municipal open data initiatives are happening in New York City, which is where I'm based now. I hope to build bigger and more useful applications with some of this information in the future.

time flies

Hi, my job has (surprise!) swallowed up most of the last few months. But now that the most intense part of this development sprint is over for me, I need to take stock of the things I'd been working on before I stopped having a life.

dev projects

  • bang bot - Twitter bot that tweets whenever someone is killed by a gun. I only barely got this Node app started over the summer.
  • sfpd and property assessment map - One of many projects for the Anti-Eviction Mapping Project. It's been in and out of development hell since the summer.
  • bunny - I wrote this absurdist text adventure while learning about Python last year. I've been meaning to finish it and turn it into a tiny but full-fledged Node+jQuery web app so people can play it outside of the command line.
  • [catmap - Essentially abandoned after I found out that someone took this basic idea and made something better, maybe even creepier.]

art projects

  • electronic music - After the past couple of years, I am finally comfortable assembling minimal stuff in Ableton Live. But the whole point was to make the music I hear in my head, and I can't do that yet, so I have to keep going.
  • novel - This is a long-running inside joke that I'd almost gotten done in a blast a few years ago. Never managed to write the last five or so chapters, and might as well. I have a surprising number of friends who want to see it published (at least as an e-book), but the manuscript is typewritten, so they'll have to OCR it first.
  • another collaboration with joel - When we first started dating, we made a gif site together. We've been meaning to do more of that, especially now that we both know more about computers and now that he's finally ventured into the SF/Oakland art scene. We're currently busy enough trying to get hitched, though.

A lot of traveling and personal events are happening over the next year, so I hope I manage to finish at least some of the things on this list.

armies of robots of lines of code

Lately I've been building a bot that tweets whenever a goal is scored in the World Cup. I've always wanted to make a twitter bot, and was pleasantly surprised at how easy it was.

The twit library for Node.js got me tweeting within minutes. I hooked it up to this super useful World Cup JSON API, wrote a hilariously complicated function to determine how many Os to tweet in the word "GOL" for each team, pushed it up to a tiny Heroku worker, and BAM! Now a silly idea I tweeted last month has become reality, and I learned a few things out of it.

Working on the gol bot has been fun, but as someone who tries to use technology to do good, I'd like to tackle more useful and thought-provoking work with this newfound knowledge. For example, I was thinking of making a bot that tweets the word "Bang" as often as someone in the United States is killed by a gun, which is more than once an hour. I could also make bots that do the same for other countries and have a site that live-updates all of them side-by-side. Other ideas are welcome.