I Spent 40 Hours Fixing a 12ms Spike

It wasn't a database deadlock. It wasn't a network timeout, a race condition, or a null pointer hiding three call stacks deep. It was a single misplaced line of code that woke up the Android Garbage Collector at exactly the wrong millisecond, once every thirty seconds, for a duration of twelve milliseconds. That's it. That was the whole crime. And it took forty hours to catch, because the entire time, it looked like nothing was wrong at all.

The Ghost

The bug didn't have a stack trace. It didn't have an error message. It had a feeling, the kind QA testers describe with their hands more than their words: "it just kind of... stutters. Right there. During that transition. Not every time."

That last part is what makes this genre of bug so miserable. A crash is a gift. A crash gives you a stack trace, a line number, a repeatable trigger. This was the opposite of a gift. It was a single dropped frame, once every thirty seconds, only during one specific screen transition, only sometimes, and only noticeable if you were staring at the screen at exactly the right moment with the muscle memory of someone who has spent too many years caring about frame pacing.

On a 120Hz display, a frame budget is roughly 8.3ms. Miss it once and nobody notices. Miss it in a pattern, though, and your thumb notices before your eyes do. Users can't tell you what's wrong. They just stop trusting the app, one imperceptible flinch at a time.

First Suspects, All Innocent

The instinct on any performance bug is to reach for the obvious suspects first, and I did, one by one, watching each of them turn out to be innocent.

Network. Disabled the radio entirely, ran the app in airplane mode with cached data. Stutter still there, exactly once every thirty seconds. Not network.

Main thread work. Wrapped every suspicious method in Trace.beginSection() / Trace.endSection() and pulled a systrace during the transition. Nothing in the UI thread's own work was spiking. The layout passes were clean, the measure/draw cycle was well within budget. Whatever was stealing the frame wasn't something I had written directly into the render path.

Animation overdraw. Turned on GPU overdraw debugging, checked for excessive layering during the transition. A little wasteful, sure, the kind of thing every Android app is a little guilty of, but not enough to account for a hard stutter. Ruled out.

Three suspects down, and the crime scene was somehow cleaner than before I started, which is its own special kind of despair. The bug wasn't hiding in any single method. It was hiding between methods, in something ambient.

The Pattern That Cracked It

The "every thirty seconds" detail kept nagging at me, because periodic problems in software are almost never actually periodic on their own. Something else is periodic, and this bug is just a symptom riding along on top of it. Timers don't fire themselves into existence. Something was ticking every thirty seconds, and the stutter was a passenger.

I pulled a systrace with GC events explicitly enabled, this time watching the full width of a two-minute session instead of a single transition, looking for anything with that thirty-second cadence. And there it was, a ART Background GC event, landing with suspicious regularity, spaced almost exactly thirty seconds apart.

Android's garbage collector doesn't run on a clock out of spite. It runs when allocation pressure crosses a threshold. Something in this app was allocating enough garbage, at a steady enough rate, to trip that threshold roughly every thirty seconds, and the concurrent GC pause, small as it was, was landing on top of a UI transition often enough for a human thumb to feel it.

The Heap Dump

Systrace told me when. It didn't tell me what. For that I needed a heap dump, taken right before one of the periodic collections, compared against a second dump taken right after the next one, run through Android Studio's Memory Profiler with the diff view.

The diff was almost boring in how obvious it was in hindsight. A steady, linear climb in Rect object allocations, thousands of them, none of them referenced by anything that should have outlived a single frame. Short-lived garbage, exactly the kind the generational collector is built to handle cheaply, except there was so much of it that "cheap" still wasn't free.

I traced the allocation source with the profiler's callstack view and found it: a custom onDraw() override on a View used inside a RecyclerView item, instantiating a new Rect() on every single draw call to compute a clip boundary for a rounded-corner effect.

override fun onDraw(canvas: Canvas) {
    val bounds = Rect(0, 0, width, height) // allocated every draw call
    canvas.clipRect(bounds)
    super.onDraw(canvas)
}

Scrolling a list at 120Hz calls onDraw() up to 120 times a second, per visible item. On a list with a dozen visible rows, that's well over a thousand short-lived Rect allocations per second during any scroll, feeding the young generation heap fast enough to trigger a background collection roughly every thirty seconds of moderate scroll activity. The collection itself was cheap, twelve milliseconds, barely a blip on paper. But it didn't care what the main thread happened to be doing when it landed, and often enough, what the main thread happened to be doing was the exact transition animation someone in a usability session couldn't quite put their finger on.

The Fix Fit in One Line

private val bounds = Rect() // allocated once, reused every draw call

override fun onDraw(canvas: Canvas) {
    bounds.set(0, 0, width, height)
    canvas.clipRect(bounds)
    super.onDraw(canvas)
}

Move the allocation out of the hot path. Reuse the object. Mutate it in place instead of recreating it sixty or a hundred times a second. The diff is three lines. The bug took forty hours to find. That ratio is not a coincidence, it's the entire genre.

Rebuilt, reran the same systrace capture over the same two-minute session. The periodic background GC events were gone. Not smaller. Gone. The allocation rate during scroll dropped enough that the collector simply had no reason to fire on that cadence anymore. The stutter, the one nobody could quite describe but everybody could feel, went with it.

Why Forty Hours Was the Correct Amount of Time

There is a very reasonable argument against spending forty hours on a twelve-millisecond hitch that ninety-five percent of users would never consciously register. Ship it. Move on. Nobody's writing a support ticket about a single dropped frame during a screen transition.

That argument is correct about the ticket volume and wrong about everything that matters. Nobody files a bug report for "the app feels slightly less premium than the other app on my home screen." They just quietly prefer the other app, and they couldn't tell you why if you asked them. Frame pacing is one of the few technical qualities that gets evaluated by literally every user, on every single interaction, without a single one of them being consciously aware they're doing it. Get it wrong consistently enough and the app just feels cheap. Get it right consistently enough and it feels, for lack of a more precise engineering term, expensive.

"Good enough" would have shipped after the first three suspects came back clean. Nothing was crashing. Nothing was timing out. The metrics dashboard was green. Good enough had already been achieved, twice over, by any measure a product manager would sign off on.

Great required believing that a feeling nobody could articulate was still a real bug, worth a systrace, worth a heap diff, worth forty hours chasing a single reused Rect. It required treating twelve milliseconds, once every thirty seconds, as a crime scene instead of a rounding error. That obsession is not efficient. It is not, by any reasonable calculation, a good use of an engineer's week.

It is, however, the entire difference between an app that runs and an app that disappears into the hand.