The first time our mobile app loaded a user’s transaction history, it took 12 seconds. Twelve. Whole. Seconds. I watched our CEO stare at the loading spinner during a demo, and I could see the math happening in his head: If this takes 12 seconds for one user, what happens when we have 10,000 users checking their balance at 9 AM on payday? He didn’t say anything. He just raised an eyebrow. ...
I thought I was being clever. When we migrated our services to Google Kubernetes Engine with auto scale profile optimized, I looked at our resource specs and saw an opportunity. Our pods were requesting 100m CPU but had limits set to 1000m. Ten times headroom! Surely we could tighten that up and save some money. So I did what seemed logical: I kept the limits high (just in case of traffic spikes) but dropped the requests even lower. 50m here, 25m there. The cluster was happy. Our costs went down. I patted myself on the back for being such a savvy engineer. ...
I used to think CPU and memory were the same. Not literally, of course. I knew one was for processing and one was for… well, memory. But when it came to Kubernetes resource limits, I treated them identically. Set a request, set a limit, let the scheduler do its thing. If the app needs more, it uses more, right? Wrong. Very, very wrong. And I learned this lesson at 2 AM on a Tuesday, when our primary API service went from “healthy” to CrashLoopBackOff in about 30 seconds. No warning. No graceful degradation. Just… dead. Then alive. Then dead again. ...