War Stories: Kafka Partition Lag

Recently, I and a number of colleagues spent the better part of a week chasing down some baffling behavior in a kafka consumer. After a routine1 cluster upgrade, we observed that one of the partitions in a deal publication topic was lagging further and further behind, negatively affecting our production processes.

By the end of the week, we’d chased down the issue, and have determined that it was the result of the confluence of a number of factors which had been lying dormant until the right combination of circumstances arose. I think the combination is subtle and interesting enough to peel back the curtain a bit and see what it was that bit us.

Also, this might qualify as therapy… You’ve been warned.

  1. Not actually all that routine, as we blanked out all of our existing topics, chasing a clean start. This had the effect of changing the number of partitions on this topic.

Replace While Loop With Enumerator

Sometimes, it looks like it is not possible to avoid using an accumulating array, a pattern that feels unnatural in Ruby.

Recently, I’ve need to chase down and unroll pagination links over a JSON / REST api. I don’t know how many pages there will be, and it’s probable (but not guaranteed) that I need to retrieve and use all of the content. Since each page is dependant on results from the previous page, there is no obvious Enumerable parallel. Here, I’ll demonstrate a quick refactoring that will provide in a clean, lazy enumerable object.

Use your environments

I may be repeating something you already know, but this is not something I see talked about a lot, and I’ve run into this more times than I should.

Every so often, when I’m combing through a Rails codebase, I’ll find code that looks something like this:

def some_method
  if Rails.env.production?
    # Do something for real
    # Do something fake

This might be tied up in a Twitter callback, or it may be performing some other sort of expensive network computation. Or, it might be a switch to turn off a feature while it’s still in development. There are good reasons for applying this pattern … but the way it’s applied leaves much to be desired.

Isolating Validations in ActiveModel

I’m working on a rails project where I’m experimenting with a much more object-oriented approach than Rails usually encourages. I’m intending on writing more in depth about that, when I can better gather my thoughts about the experience. I found something recently that I can write a brief post about and hopefully get the ball rolling.

I’m trying to keep a fairly hard-line divide between my persistence layer and the actual domain objects. What this means for today’s purposes is that my domain objects know nothing about ActiveRecord (and some of them never will).

This lead me to a philisophical quandry when trying to figure out validations. My domain objects are already imbuded with ActiveModel attributes, because they communicate with the view layer in ActionPack. (ActiveModel is, essentially, the reference API that ActionPack expects.) However, validations can cross both persistence and domain concerns. I’ve also been pretty heavily burned in the past by the mess that is conditional validations.

In debating whether to embed my validations in the domain model, or in the ActiveRecord objects that my Repositories map the domain entites to, I realized that I had a third option: Separate the validations. Structuring the validations as their own class opens up a lot of the traditional OO flexibility that I’ve been aiming for in this application, and allows me to ignore the validations in situtations where they are not needed, without a lot of stub/mock gymnastics.

getting started

It’s been a long time since I tried to regularly maintain a blog; I once had a wordpress blog under the same name, but it suffered from the bitrot that hits a lot of neglected self-hosted wordpress installs.

Rather than apologize for not posting and trying to get that old beast into shape, I’m starting from scratch here. Compared to the various social media sites out there, starting a blog in this day and age seems odd. For the most part though, I want to talk about and explore some technical ideas in deeper depth. I’ll see how well I can keep this up, this time through.

I’ve been doing software professionally for a decade now. Most of that time, I’ve been embedded in an environment that doesn’t much promote oversharing. About two weeks ago, however, I started as a developer at Living Social. I’ll be working on improving their internal tools and services. Some of it will be green field, and some of it will be rescuing older applications.

I’m not the only one talking about the things I’m planning on starting with here; I’m far from a primary source. What I am is an aggressive early adopter … and a cranky old-school UNIX curmugeon and VI holy warrior.

It’s probably not wise to start working on a blog before heading off on holiday vacation, but I wanted to get at least a little something in place. There are a couple of things I’m going to write about sometime in January, and I wanted to get the baseline established. For now, I have got to finish packing.