trekking through the amazon

A few months ago, I learned about the t2.nano, a new Amazon EC2 instance that was cheaper than my old t2.micro ($5 a month versus $10), so I tried switching over to it and promptly bricked the whole thing, which was why this blog wasn't working for a while.

I started a new job that required me to learn a lot more about Amazon services, so this week I decided to take another crack at migrating my blog instance over. This time I even properly documented everything. It took several long evenings (and a panicked rewriting of my commit history after I pushed up some config files to GitHub that I really shouldn't have, which resulted in my emailer promptly getting hacked and turned into a spamming device), but it's now on the smaller nano instance and purring like a kitten.

While cussing at my laptop, I learned a lot about:

  • SSHing into stuff and making that process easier
  • How RSA keys are handled between hosts
  • Cloning Amazon EC2 instances and EBS volumes
  • Copying files to and from remote hosts (scp)
  • Obscure Git commands that you only need to know when you've seriously ****ed some **** up

    I really ****ed some **** up I really ****ed some **** up

This was all basic web deployment stuff I've been wanting to get deeper into, since my main goal as a web developer is to be able to create and run applications by myself. While the frontend has always been my jam (I like making things people can immediately use), I figure that if I can be a one-stop shop, I'll be of much more use to smaller places that really need a web presence, such as nonprofits and small businesses.

People have already suggested other improvements I can make to the blog, like adding encryption. I could also make my Ghost instance a static site using Buster (a.k.a. 'Ghost Buster'), since storing it in an Amazon S3 bucket would be even cheaper-- around $3 a month. I'll consider some of these options. But before that, I'd like to just enjoy having it back up and running.

It was only a couple of years ago that I became a web developer, but in that time, it feels like I went to college again and learned twice as much as I had the first time around. I started this blog in 2014; I'm kind of surprised that I managed to do it, since I was blindly copy-pasting terminal commands from Stack Overflow and hoping for the best. I know so much more about how this all works now.

Edit: I still blindly copy-paste terminal commands from Stack Overflow and hope for the best, but not as often as I used to.

another year, another bot

My 2015 has been filled with long and arduous international and domestic travel, plenty of family business, a cross-country move, and a deferred to-do list a mile high. I haven't been keeping regular hours since the beginning of the year, so I'm glad that the dust has settled and I have a little free time to learn and work on new things again.

I finally made a version of the bang bot that is police-oriented (and less pervy-sounding). The Washington Post's fatal police shootings database is reliably updated, and it's on GitHub so it was much easier for me to get raw data from (otherwise I would have to do some manual scraping-- not hard, just time-consuming).

The bot's checking mechanism was the simplest one I could come up with:

  1. Pull the latest .csv file on WaPo's GitHub
  2. Check if the number of lines is greater than that of the last stored version, and
  3. Tweet that number if it is.

The implementation's still got a few kinks (apparently you can't use Twitter's API to tweet the same thing twice, which may be a problem soon), but my previous experience with Node made building it fairly easy.

It was the deployment that gave me some trouble. I used Heroku to run my app this time, and it took me a while to realize that the reason my files weren't properly saving was because Heroku workers don't do data persistence (derp). So I had to pick a data storage addon. I would have liked to use MongoDB to make the app JS end-to-end, but it wasn't free. And I'm pretty familiar with Postgres after having used it for a class earlier this year, but I ultimately went with Redis, another NoSQL choice that's fast and has an easy-to-use Node.js client. I hear that relational databases are the way to go if you have a data structure that's even remotely complicated, but I only had to store two things.

A screenshot of callback hell. Can you tell I'm using Node?

Besides storage, I still haven't quite gotten the hang of JavaScript's asynchronous processes yet (please refer to the callback hell in the screenshot above). I tried to keep things readable, but I think I'm gonna just bite the bullet next time and use async.

The reason I started this project was to get an update on Twitter whenever WaPo confirms another death by police shooting, and I'm glad I reached that goal. But seeing the bot tweet for the first time on production was the first time I had ever felt sad about seeing something I built actually work.

Now that I'm getting into full-stack development, I'm exploring what else I can do with open data. My final project for the back-end class that I took this spring used data made available by San Francisco's Human Services Agency. Similar municipal open data initiatives are happening in New York City, which is where I'm based now. I hope to build bigger and more useful applications with some of this information in the future.

time flies

Hi, my job has (surprise!) swallowed up most of the last few months. But now that the most intense part of this development sprint is over for me, I need to take stock of the things I'd been working on before I stopped having a life.

dev projects

  • bang bot - Twitter bot that tweets whenever someone is killed by a gun. I only barely got this Node app started over the summer.
  • sfpd and property assessment map - One of many projects for the Anti-Eviction Mapping Project. It's been in and out of development hell since the summer.
  • bunny - I wrote this absurdist text adventure while learning about Python last year. I've been meaning to finish it and turn it into a tiny but full-fledged Node+jQuery web app so people can play it outside of the command line.
  • [catmap - Essentially abandoned after I found out that someone took this basic idea and made something better, maybe even creepier.]

art projects

  • electronic music - After the past couple of years, I am finally comfortable assembling minimal stuff in Ableton Live. But the whole point was to make the music I hear in my head, and I can't do that yet, so I have to keep going.
  • novel - This is a long-running inside joke that I'd almost gotten done in a blast a few years ago. Never managed to write the last five or so chapters, and might as well. I have a surprising number of friends who want to see it published (at least as an e-book), but the manuscript is typewritten, so they'll have to OCR it first.
  • another collaboration with joel - When we first started dating, we made a gif site together. We've been meaning to do more of that, especially now that we both know more about computers and now that he's finally ventured into the SF/Oakland art scene. We're currently busy enough trying to get hitched, though.

A lot of traveling and personal events are happening over the next year, so I hope I manage to finish at least some of the things on this list.

armies of robots of lines of code

Lately I've been building a bot that tweets whenever a goal is scored in the World Cup. I've always wanted to make a twitter bot, and was pleasantly surprised at how easy it was.

The twit library for Node.js got me tweeting within minutes. I hooked it up to this super useful World Cup JSON API, wrote a hilariously complicated function to determine how many Os to tweet in the word "GOL" for each team, pushed it up to a tiny Heroku worker, and BAM! Now a silly idea I tweeted last month has become reality, and I learned a few things out of it.

Working on the gol bot has been fun, but as someone who tries to use technology to do good, I'd like to tackle more useful and thought-provoking work with this newfound knowledge. For example, I was thinking of making a bot that tweets the word "Bang" as often as someone in the United States is killed by a gun, which is more than once an hour. I could also make bots that do the same for other countries and have a site that live-updates all of them side-by-side. Other ideas are welcome.

21st century innocence

Anamanaguchi - Endless Fantasy

Most people to whom I show the above video think it is ridiculous. Of course it's ridiculous. It's some weirdly Americanized kawaii iOS 7 vision of the 90s as reimagined by someone who grew up in the 90s but is now technically an adult. It's cheesy and dumb, like drunk hipsters who jump in front of an exploding firework just to take a badly-focused Instagram photo that will get them 50 likes. And what the hell kind of entitled jerks spent a bunch of money to send a piece of pizza to space?

In spite of myself, I dig this song. As for the video, I obviously don't think anyone should make that their life all the time. But it reminds me of the one kind of fun I can no longer have: the carefree joy I felt when I still believed I could do anything, when I was still blissfully ignorant of how the world worked.

I'm glad I know better now. But sometimes, on the days so tiring that I need to use my last shred of faith in humanity just to go outside, I'd give anything to get that innocence back.

method to the madness

"As a project drags on, my git commit messages get less and less informative."

Much of my programming experience has been gained through trial and error. For a teenager doing raw frontend development, this is not a totally terrible strategy. In fact, if you have the patience to nudge a box a few pixels over and over then eyeball it in different platforms and sizes to see what looks best, being a frontend dev might be for you.

However, for things that can't be eyeballed, and with a large framework like Ember, that kind of manual plugging takes forever. For a while, I was honestly just Googling my problems and trying everything that Stack Overflow was suggesting one by one. But if it's 3 a.m. and you barely have enough functioning brain cells to maintain your sanity, much less keep track of which things worked, you can't do this.

So what I've started to do is keep a dev diary.

Writing entries for a dev diary is like an extended version of writing explanations for a problem set answer, where you also include everything that went wrong. I write down:

  1. The desired goal, or the problem I want to solve
  2. A description of the trouble I'm running into (this might be the same as step 1)
  3. My theory on why I'm having trouble
  4. A course of action I'm going to try based on what I think is going wrong
  5. Whether that course of action worked
  6. (Repeat steps 3-5, with adjustments as necessary, until problem is solved or goal is achieved)
  7. A summary of what I did right in TL;DR form that I usually put at the beginning of the entry

Much like a real diary, this helps me organize my thoughts so I can sleep at night without trying to keep anything in my head, then return to the task in the morning without having forgotten what I've already done.

There's also the added bonus of having a beautiful written record of the knowledge I've gained, instead of a bunch of uninformative git commits called "Still broken", "Less broken", "Fixed this, now trying to fix this other thing", and "OMG WTF". I highly recommend it.