Merry Christmas! If you've been paying attention to New England specific weather reports, you might think that Katie and I are holed up somewhere since apparently Massachusetts is a frozen hellscape. But no! Since last year we've been continuing the Swap Holidays plan, and this Christmas we're in Rumson, NJ. It was a hell of a drive. We left Saturday morning during an ice storm, and I spent 7 and a half exhausting hours making our way through Mass, Connecticut, and New Jersey to arrive here.
We chilled on Saturday evening, both exhausted, and went to bed fairly early. Christmas Eve was pretty nice. Katie and I went for a drive, and she's getting pretty good with the manual! She made a "student driver" sign, since she was intimidated whenever someone pulled up behind her. After putting 4 more years of wear on my syncros/clutch, we ran a few errands. Katie commissioned watercolors of her grandmother's old home, where her mom and uncle grew up, and we traveled around trying to buy frames. While doing so, we also stopped at William Sonoma so I could buy a gift for her grandmother. The highlight, though, was a store called "Nothing Bundt Cakes", which sold... bundt cakes. I made sure to stop, and after confirming from the front desk girl that they did, in fact, sell only bundt cake, we sampled the lemon, which was wicked good.
After picking up some dried cranberries for Maggie, we helped a little bit around the house getting things ready, and took the dog for a walk in the neighboring town. That night, we went out for dinner at a fancy (jacket and tie) restaurant called Nicholas. We did a 6 course tasting menu, which was great, since Katie and I just ordered opposites on the menu so we could try everything. At the end of the night, the waiter came over with little gift bags and was all "here's some banana bread... for the ladies". I was like "what the hell, where's my banana bread, why the sexist banana bread guy". In my head, of course. Had it just been Katie and I, I totally would have asked for some, but I wasn't paying the bill. Oh well. I ate it when we got home, anyway. Take that creepy fancy restaurants that try to gender dessert breads.
Yesterday morning, we slept in pretty late, and Katie's mom made eggs, bacon, and biscuits, which was great. After that, and a fair amount of coffee, we opened presents! I got a book called "Wine Cats", which consists entirely of pictures of cats, with small anecdotes about the cats. It is the Best Book. Once we were done with presents, we cleaned up a bit, and Maggie continued to get things ready for dinner. It was at this point that I decided to retreat somewhere that I couldn't be found, since there's a limited amount of social interaction I'm prepared to handle, and I feel like I needed to recharge my batteries for dinner. Luckily, there was a task at hand, one of my own creation, and something I needed to fix quickly.
I had the bright idea of upgrading my Ghost (blogging platform) installation a few days ago. While in the middle of the upgrade, something terrible happened, and I lost my shell. When I re-connected, everything was broken. My blog was down. I left it like this, because I was busy doing Christmas things, and didn't have time to fix it. It was the worst. Leaving a job unfinished gnaws at me, and though I know no one actually reads this blog, I didn't like the idea of it being down. After running 33 npm/yarn commands, manually performing a database migration, and rewriting my supervisor scripts so the damn thing would persist on reboot, we were back in business. Hooray hosting things yourself!
While I was in the guts of the server, I decided to address some performance issues that had been nagging me. Last month, I purchased a couple of reserved instances from AWS. Reserved instances aren't really anything other than a billing discount. You essentially commit to having an on-demand instance of whatever type of server you purchase for a 12 month period. I bought a t2.micro for my mail server, and a t2.small to run my blog and password vault. For deployment reasons I can't just run one t2.medium, so this was the closest I could get to maximum resource utilization. The mail server is running just fine! Unfortunately, the blog/vault server has been a bit creaky, and has been exhibiting OOM-like symptoms occasionally when I try to connect to it.
A cursory examination of top showed that CPU was totally fine, and that the biggest consumers of memory were sqlservr and mysql. Ok. Free -mt showed that there was almost a full GB being used by buffer cache, which again, makes it seem like there's plenty of available memory for the box. The only thing I could come up with is that occasionally something happens that starts to consume more memory, and the box starts to thrash. Probably like a cron'd vacuum. So I decided to add some swap, to help alleviate memory pressure during that time. I generally set swap around the same size as RAM, and I set swappiness to 10 and vfs_cache_pressure to 50. All that'll do is make sure the machine will only start to swap when things get really dicey, and that the kernel will prefer to keep file system pointers/inodes cached over other data, which will help if I need to manually intervene when something goes wrong. So far it's been super healthy, and I'm hoping I can get through 2018 with this instance configuration. With the reserved instances, it should end up costing close to $15/month, which is way better than the close to $40 I was paying before.
After figuring this stuff out, I helped get things ready for dinner, which was delicious, and we had some drinks and played Codenames, which is a game you should play only with large groups. If you do that, it'll be a lot of fun, otherwise you'll hate everyone in the room. Katie is sick, so we went to bed fairly early. In a half hour or so, we'll begin the long drive back to Somerville, where I'm looking forward to chilling for a week or so. Who knows? Maybe I'll finally be able to accomplish my goal of going Google Free by spinning up a cost-effective self-hosted Dropbox replacement. No word on whether I'll go AWS free at some point, but baby steps. At least I own the data and the private/encryption keys for these boxes, instead of trusting them to a third party. Now that I'm getting the cost under control, I'm pretty pleased that I made the decision to start self hosting services. I'm a lot more comfortable paying for tools, than being the product for a huge megacorporation.
-Chris