Scenarios: incidents and cross-team

Production breaks and teams point at each other. This chapter is a dense bank of real chat exchanges for exactly those moments, shown from both chairs: the one who shipped the bad change and the one who caught it, the team asking for ownership and the team dodging it. Use it as a lookup. Find the row that matches your Slack thread, borrow the shape, and keep the move: soften the person, sharpen the substance, always carry a fact or a next step.

Your deployment caused a production issue

  • Your deploy spikes error rate. Q: "Errors jumped right after your 2:14 release, is that you?" A: "Yes, that is mine. Rolling back now, ETA three minutes, will post when 5xx is back to baseline."
  • Owning it in the incident channel. Q: "Who deployed last?" A: "Me, the checkout service at 14:12. I own this. Rollback in progress, will update every five minutes."
  • Caught a bad config after ship. Q: "Payments are timing out in eu-west." A: "That is my config change, I set the pool size to 5 instead of 50. Reverting the env var now, no code deploy needed."
  • PM asks how bad it is. Q: "How many users are hit?" A: "About 4% of checkout traffic since 14:12, roughly 900 sessions. Rollback lands in two minutes, then it is zero."
  • Explaining the miss without excuses. Q: "How did this pass CI?" A: "It passed unit tests but we had no integration test on the timeout path. Adding one before I re-ship so it cannot recur silently."
  • Someone offers to help mid-fire. Q: "Want me to jump on the rollback?" A: "Yes please, can you watch the p99 dashboard while I run the revert? Ping me if it does not drop by 14:25."
  • Resisting the urge to hotfix live. Q: "Can you just patch it forward?" A: "I would rather roll back first to stop the bleeding, then fix forward calmly. Reverting now, fix PR in an hour."
  • Post-rollback confirmation. Q: "Are we green?" A: "Green. 5xx back to 0.1% as of 14:24, matching the morning baseline. Keeping the incident open for 30 minutes to be sure."
  • Owning it to a customer-facing team. Q: "Support is getting tickets about failed uploads." A: "That is my deploy, sorry for the noise. Fixed as of 14:24. Tell them to retry now, and I will write the customer note."
  • Manager checks in privately. Q: "You good? Need cover?" A: "I am good, rollback done, root cause is clear. I will run the retro Thursday and own the action items."
  • Declining to rush the retro. Q: "Can you send the postmortem tonight?" A: "I can send a two-line timeline tonight and the full retro with actions by Wednesday. The draft would be rushed if I do it at 8pm."
  • A teammate points out you skipped canary. Q: "Did this go straight to 100%?" A: "It did, and that is the real lesson. I should have canaried. Adding a 10% stage to this pipeline as the first action item."
  • Reassuring on-call it is contained. Q: "Is this still growing?" A: "No, it stopped growing at 14:24 when the revert finished. Nothing new since. Safe to lower the alert to a watch."
  • Being honest you are not sure yet. Q: "Do you know root cause?" A: "Not fully. I know the trigger was my deploy and rollback fixed it. The exact why is still open, I will confirm in the retro, not guess now."

Rolling back another engineer's change

  • You spot their deploy is the cause. Q: "(you, opening)" A: "Hey Sam, checkout 5xx started right at your 13:40 deploy. I am going to roll it back to stop the bleeding, then we can look together. Heads up, not blame."
  • Rolling back while they are offline. Q: "Sam is at lunch, do we wait?" A: "No, I will revert now and leave a full note in the channel and a DM. Stopping the incident comes first, courtesy second."
  • Asking first when it is safe to wait. Q: "Should I revert Priya's migration?" A: "It is a slow burn, not a fire, so ping her first. Priya, your 11am migration is doubling read latency, ok to roll back while you look?"
  • They push back on the rollback. Q: "Do not revert, I can fix forward in ten." A: "I hear you, but we are losing checkouts every minute. Let me revert to green now, then you fix forward calmly and re-ship. Same outcome, less bleeding."
  • Softening the message you had to revert. Q: "(DM to the author)" A: "Reverted your feature-flag PR at 13:52, sorry to step on it. It was tripping the rate limiter in prod. Nothing wrong with the code, just the rollout. Want to pair on re-shipping?"
  • You are the author being rolled back. Q: "I reverted your change, it was spiking latency." A: "Thank you, that was the right call. I would rather be green and re-ship than defend a live regression. What did the graph show?"
  • Explaining the revert in the retro. Q: "Why did we revert instead of patch?" A: "Revert was a known-good state one click away. A forward patch was untested under load. Fastest safe path was back, then forward."
  • Rolling back a shared library bump. Q: "The 2.4.0 bump broke three services." A: "Pinning us back to 2.3.1 across all three now. I will open one issue on the library repo with the repro so the owner sees it, not three angry pings."
  • When the author disagrees it was them. Q: "It cannot be my change, it is just a copy tweak." A: "Could be, but the timing lines up to the second with your deploy. I will revert and we watch: if errors stay, I owe you an apology and we keep digging."
  • Reassuring a junior after reverting them. Q: "I feel terrible, I broke prod." A: "You did not break prod, a gap in our canary let it through. Everyone here has been reverted. Let us fix the pipeline so the next one is caught."
  • Reverting a vendor SDK update. Q: "The new Stripe SDK is throwing on refunds." A: "Rolling us back to the prior SDK pin now. I will file with their support with the stack trace so it is on their radar, and we hold the upgrade."

Disagreeing about incident severity

  • You think it is bigger than called. Q: "This is a SEV3, just some slow queries." A: "I would call it SEV2. p99 on login is 8 seconds and climbing, users cannot get in. Happy to be wrong, but let us staff it like a SEV2 for now."
  • You think it is smaller than called. Q: "Declare SEV1, the dashboard is red." A: "The dashboard is red but it is one internal admin panel, no customer impact. I would hold at SEV3 and skip the exec page until we see user harm."
  • Manager wants to downgrade too early. Q: "Can we close this out, looks quiet?" A: "Errors are quiet but we have not confirmed root cause, so it could recur. I would keep it open as a SEV3 watch for another hour, then close."
  • Pushing back on an all-hands page at 3am. Q: "Page the whole team now." A: "It is one region and traffic is low at 3am. Let me and on-call take it, and we escalate the moment it spreads. No need to wake ten people yet."
  • Agreeing to raise severity with a reason. Q: "You really think this is SEV1?" A: "Yes, payments are fully down in two regions and revenue is stopped. That is the SEV1 line for us. I will declare and pull in the payments lead."
  • Naming the disagreement calmly in the channel. Q: "Everyone good with SEV3?" A: "One dissent, gently: I read this as SEV2 because of the auth impact. Can we align on user harm as the tie-breaker before we set it?"
  • You are the IC being challenged. Q: "This should be a SEV1, not SEV2." A: "Make the case, what user impact are you seeing that I am missing? If it is total payment loss I will raise it in the next minute."
  • Severity argument during the call. Q: "We are wasting time debating a number." A: "Fair, the number decides who we wake and what we tell execs, so it matters. Let me set SEV2 now, we can adjust in ten if impact grows. Moving on."
  • Refusing to inflate to get attention. Q: "Just call it SEV1 so people respond faster." A: "I get the pull, but if we cry SEV1 on a SEV3 the next real one gets ignored. Let me escalate through the on-call chain instead, same speed, no inflation."

Don't be confused: severity is not the same as urgency. Severity measures how much harm is happening (who is hurt, how badly); urgency is how fast you must act. A slow data-corruption bug can be low urgency but high severity. Argue severity with impact numbers, not with how stressed the room feels.

Someone blames you for an incident

  • Blamed in the channel, not your change. Q: "This is from your service, right?" A: "Let me check before we settle that. My last deploy was Tuesday and the graph turns at 09:10 today. Pulling the deploy log now so we are on facts, not guesses."
  • Blamed and it actually was you. Q: "Pretty sure your deploy did this." A: "You are right, it was my 09:05 change. Owning it, rolling back now. Thanks for catching the timing fast."
  • Blamed publicly, you want it moved to DM. Q: "You keep breaking prod lately." A: "Let us fix the outage first, then I am happy to go through my last three deploys with you in a DM. Right now I want eyes on the rollback."
  • A vague accusation with no evidence. Q: "Feels like the search team caused this again." A: "Maybe, but I do not want to guess at a team. What is the actual signal? If it is search latency I will pull our traces and we will know in five minutes."
  • Blamed for a config you did not own. Q: "You set this timeout, right?" A: "That value came from the platform default, not from me. Here is the commit, it is owned by infra. Let us loop them in rather than trade guesses."
  • Staying calm when it gets sharp. Q: "Your team never tests anything." A: "That stings, and I want to get to the fix. If our tests missed this, I want to know exactly where so we close the gap. Can you show me the failing path?"
  • You caused it and got defensive first. Q: "Are you sure it is not your deploy?" A: "I checked again and it is mine, I was wrong a minute ago. Sorry for the pushback. Reverting now."
  • Blame in the retro, keeping it blameless. Q: "So this is on the new hire?" A: "The change was theirs, the gap that let it reach prod is ours as a team. Let us talk about the missing canary, not the person. That is where the fix is."
  • Redirecting blame to the system, not a person. Q: "Whose fault is the data loss?" A: "The honest answer is that our deploy process let an unmigrated schema ship. No single name fixes that. The action item is a migration gate in CI."

You believe another team caused it

  • Opening the cross-team thread carefully. Q: "(you, opening to the other team)" A: "Hey, our checkout errors line up exactly with a spike in 429s from the pricing API at 10:02. Could be us calling it wrong, could be a limit change on your side. Can we look together?"
  • Presenting evidence without accusing. Q: "Why do you think it is us?" A: "Two things: our calls to your endpoint went from 40ms to 3s at 10:02, and your status page shows a deploy at 10:01. Not blaming, just the two data points I have. What do you see?"
  • They deny it, you hold the facts. Q: "Nothing changed on our end." A: "Understood, though our traces show your endpoint returning 503 since 10:02. Here is a request id: req_8f2a. Can you grep your logs for it so we are on the same call?"
  • Asking for their timeline. Q: "Can you share your deploy log for the auth service?" A: "That would help a lot. If your last change was before 09:00, this is almost certainly on us and I will drop it. If it was 10:01, we have our answer."
  • You are wrong, walking it back gracefully. Q: "That was our old caller, not your API." A: "You are right, it was our stale connection pool holding dead sockets. Sorry for the ping, thanks for looking. The fix is on our side."
  • Avoiding a public pile-on. Q: "Should I call out the DB team in the incident channel?" A: "Let us DM their on-call first with the query plan. If it is theirs they will jump on it faster without an audience. Public only if we get no response."
  • Framing it as shared, not their fault. Q: "This is clearly the CDN team's problem." A: "It shows up at the CDN edge, yes. But our cache headers might be forcing it. Let me bring them in as partners: here is what we see, here is what we changed."
  • Getting agreement on the boundary. Q: "Where does your part end and ours begin?" A: "Our responsibility ends when the message hits your queue. It sat there 40s before your consumer picked it up. Can we confirm the consumer lag on your dashboard?"
  • When both teams contributed. Q: "So who owns this one?" A: "Honestly both of us: our retry storm plus your tighter rate limit. Let us split the actions, we add backoff, you document the new limit. One retro, two owners."

Asking another team to take ownership

  • Handing off a bug that is theirs. Q: "(you, opening)" A: "This one traces cleanly to the notifications service: your worker drops messages when the payload is over 256KB. Repro and logs in the ticket. Can your team pick it up? Happy to pair on the repro."
  • They try to hand it back. Q: "Can you just fix it in your service?" A: "I could patch around it, but the bug lives in your queue consumer and a workaround here hides it from the next caller. Better it is fixed at the source. What do you need from me to take it?"
  • Asking for an owner, not a favor. Q: "Who should own the flaky auth timeout?" A: "It is in the auth service so I think it is yours, but I want to confirm rather than assume. If it is, can you name an owner and a rough week? If not, point me the right way."
  • Making the ask easy to say yes to. Q: "We are slammed, can this wait?" A: "It can wait two weeks, it is not on fire. I have written the repro, the fix location, and a test that fails. It should be a small pickup. Can we put it in your next sprint?"
  • You are being asked to take ownership. Q: "This bug is in your service, can you own it?" A: "Looks like it, yes. Give me an hour to confirm the repro, then I will take it and give you an ETA. Thanks for the clear write-up."
  • Declining ownership with a reason and a redirect. Q: "Can your team own the shared dashboard?" A: "We cannot take it on this quarter, we are down two people. The observability team owns the dashboard platform though, they are the right home. I will intro you."
  • Setting the boundary for future asks. Q: "Can you handle all the vendor webhook failures?" A: "We will own the ones inside our ingestion path. Failures at the vendor edge belong with the integrations team. Let me write that split into the runbook so it is clear next time."
  • Getting a commitment, not a maybe. Q: "Yeah we will look at it sometime." A: "Appreciate it. Can we make it concrete so it does not slip: an owner name and a target sprint? Even 'Ravi, sprint after next' is enough for me to plan around."

A team is not answering your dependency request

  • Gentle first nudge. Q: "(you, following up)" A: "Hey, following up on my request from Monday for the read-only DB grant. No rush if it is queued, just confirming it landed with someone. Ticket is PLAT-812."
  • Second nudge with the cost of waiting. Q: "(you, second follow-up)" A: "Circling back on PLAT-812. We are blocked on this for the launch and lose about two days for each day it waits. Can someone give me a yes/no or an ETA today?"
  • Asking who owns it, not chasing air. Q: "Who owns access grants these days?" A: "I have pinged the channel twice with no bite. Rather than keep shouting, can you point me to the actual owner or the on-call rotation for platform requests?"
  • Moving from channel to a named person. Q: "(DM after silence)" A: "Sorry to DM directly. My request in the platform channel has sat two days and I am blocked. You do not have to own it, just tell me who does and I will take it there."
  • You are the team being chased. Q: "Can someone please look at PLAT-812?" A: "Sorry for the silence, we dropped it. Looking now. You will have the grant by end of day or a clear reason why not. Thanks for the patience."
  • Setting a deadline politely. Q: "(you, adding a date)" A: "To keep us unblocked, I need this by Thursday to hit the launch. If Thursday is not doable, tell me and I will escalate for help rather than let the date slip quietly."
  • Offering to reduce their work. Q: "We do not have bandwidth to build the endpoint." A: "Understood. Would it unblock us if you just gave me read access and I query directly for now? Smaller lift for you, and I stop pinging. We can build the proper endpoint later."
  • When silence forces an escalation. Q: "(you, warning before escalating)" A: "Heads up, if I do not hear back on PLAT-812 by tomorrow I will raise it with your lead, only because we are blocked, not to go over your head. I would rather solve it with you."

Escalating a blocker

  • First escalation to your own manager. Q: "(you, to your manager)" A: "I need help unblocking. The data team has not answered my access request in three days and the launch is Friday. I have chased twice. Can you nudge their lead, or point me at a better path?"
  • Escalating with a clear ask, not a complaint. Q: "What do you need from me?" A: "One thing: get the pricing team to commit an owner and a date for BUG-440 by tomorrow. I have the repro and the fix ready, I just need them to say yes."
  • Framing the impact for a skip-level. Q: "Why is this on my radar?" A: "Because it now risks the Q3 launch date. A dependency has been unanswered for a week. I am not asking you to fix it, just to help me get an owner named this week."
  • You are the manager receiving the escalation. Q: "I am blocked and getting no response from platform." A: "Thanks for flagging before it blew up. Send me the ticket and your two chase timestamps. I will talk to their lead today and get you a name by tomorrow morning."
  • Escalating a disagreement, not a person. Q: "Sounds like you two just cannot agree." A: "We agree on the facts, we disagree on who owns the fix. I need a tie-breaker on ownership, not a referee for a fight. Can you and their lead make the call?"
  • Keeping the escalated party in the loop. Q: "(you, DM to the other team)" A: "Flagging so it is not a surprise: I raised our blocker with the two leads today because of the Friday date. Nothing personal, still hoping we sort it directly. Same ticket, PLAT-812."
  • Escalating up when down did not work. Q: "Did you try the on-call first?" A: "Yes, pinged on-call Tuesday and the channel twice, no reply in three days. I am at the point where I need a lead to reassign it. Here are the timestamps."
  • Backing off when the block clears. Q: "Looks like they just responded." A: "They did, we are unblocked, closing the escalation. Thanks for standing by. I will note in the retro that requests sat three days so we fix the intake, not the people."
  • Escalating bad news you cannot fix alone. Q: "Can you just handle it?" A: "I have done what I can from my seat and it is still stuck above my access. That is exactly why I am escalating: this needs a decision at your level, today, to hold the date."

Don't be confused: escalating is not tattling. Tattling seeks blame; escalating seeks a decision or a resource you cannot get yourself. Keep the escalation about the blocker and the date, tell the other party you are doing it, and stop the moment it clears. For the fuller playbook see Chapter 11.

👉 That covers the fires and the friction. Next we turn to the quieter but just as loaded moments: getting credit right and handing feedback well. On to Chapter 54.