Flashcard deck

Two to five cards are written per completed topic, in front,back,chapter format. The deck lives at output/flashcards.csv in this directory and is reproduced below so it is readable without leaving the book.

How to use it

Import into Anki as a CSV with a comma field separator and three fields mapped to Front, Back and a tag. Or use it as a written self-test: cover the right column, answer out loud, and only then look. Reading a card and thinking "yes, I know that" is recognition, not recall, and recognition is not what an interview measures.

The cards are deliberately verbose on the back. A three-word answer is easy to recall and useless in a room; the backs are written at roughly the length you would actually speak.

The deck

front,back,chapter
"What does Kafka exactly-once actually guarantee?","At-least-once delivery plus deduplication plus atomic visibility, scoped to Kafka. Idempotent producer dedupes retries by PID and sequence number; transactions make output writes and the offset commit atomic; read_committed consumers respect the last stable offset. Anything outside Kafka needs its own idempotency key.",04-streaming-apis
"Why must max.in.flight.requests.per.connection be <= 5 with an idempotent producer?","The broker only remembers the last five sequence numbers per producer per partition. With more in flight, a retry can arrive after the window has moved and the broker cannot tell a duplicate from a gap.",04-streaming-apis
"Consumer lag is climbing but the processor is healthy and error-free. What do you check?","The last stable offset. An upstream transactional producer with an open transaction blocks read_committed consumers past it, so lag climbs while errors stay flat.",04-streaming-apis
"What is the default concurrency of Reactor's flatMap, and why is that a problem?","256, which is the Queues.SMALL_BUFFER_SIZE constant rather than a considered limit. Against a downstream sized for 20 concurrent calls it is a self-inflicted load test. Always pass an explicit concurrency derived from the downstream's capacity via Little's Law.",04-streaming-apis
"flatMap vs concatMap vs flatMapSequential","flatMap: concurrent, output in completion order. concatMap: one inner at a time, source order, no concurrency. flatMapSequential: concurrent like flatMap, output buffered into source order.",04-streaming-apis
"Why does an L4 load balancer break gRPC?","gRPC multiplexes RPCs over one long-lived HTTP/2 connection, and an L4 balancer picks a backend once per connection. All of a client's RPCs pin to one backend and newly scaled pods receive nothing.",04-streaming-apis
"Four fixes for gRPC load balancing","Client-side round_robin over a headless service; an L7 proxy or mesh that balances per stream; xDS lookaside balancing; or MaxConnectionAge on the server so connections recycle.",04-streaming-apis
"Why is prefill compute bound and decode memory bandwidth bound?","Prefill does O(sequence length) FLOPs per byte of weight read; decode does O(batch size). The H100 ridge point is around 300 FLOPs per byte, so prefill sits right of it and decode far left.",05-ai-llm
"KV cache size formula","2 x layers x kv_heads x head_dim x seq_len x batch x bytes_per_element. Llama 3 70B fp16 is 320 KiB per token, so 2.5 GiB for an 8k context.",05-ai-llm
"What limits concurrency in LLM serving?","KV cache memory, not weights and not compute. Weights are a fixed cost paid once; cache is the per-user cost and scales with context length.",05-ai-llm
"The one test that splits a RAG failure in five minutes","Manually place the known-correct passage in the context and re-run. Answer becomes right: retrieval bug. Answer stays wrong: grounding bug.",05-ai-llm
"Why is a reranker the wrong fix for low recall?","A reranker reorders what retrieval returned. If recall@50 is 0.6, the ceiling after reranking is 0.6. Fix the first-stage retriever, then rerank for precision at small k.",05-ai-llm
"Three biases of LLM-as-judge, and their mitigations","Position bias (randomise order), verbosity bias (normalise or instruct against length), self-preference bias (use a different model as judge).",05-ai-llm
"Why can a container using 35 percent of its CPU limit still be throttled?","Throttling is a function of instantaneous parallelism against a 100 ms quota window, not of average utilisation. A 1 CPU limit with 16 runnable threads burns the quota in 6.25 ms and freezes for 93.75 ms.",08-compute-kernel
"Why might removing CPU limits improve p99 latency?","CFS freezes every thread in the cgroup once the quota is consumed within a period. Removing the limit removes the freeze; requests still guarantee a proportional share under contention.",08-compute-kernel
"First metric to check for a slow-but-healthy pod","container_cpu_cfs_throttled_periods_total divided by container_cpu_cfs_periods_total, then throttled_seconds correlated against p99 latency.",08-compute-kernel
"Three ways to prevent a cache stampede","Request coalescing (singleflight or a lock) so one caller recomputes; probabilistic early expiration (XFetch) so clients desynchronise; stale-while-revalidate so the stale value serves while one background refresh runs.",09-caching-edge
"Why does jittering TTLs not fix a hot-key stampede?","Jitter desynchronises different keys populated at the same time. On one key every client reads the same stored expiry and misses together.",09-caching-edge
"Why is a cache stampede self-amplifying?","The herd slows the origin, which lengthens the miss window, which lets more requests arrive and miss, which enlarges the herd.",09-caching-edge
"SSE or WebSocket for LLM token streaming, and why?","SSE. The traffic is unidirectional, so WebSocket's advantage is unused, and SSE stays inside HTTP so load balancers, auth, tracing and compression keep working. It also gives reconnection and Last-Event-ID resumption for free.",09-caching-edge
"Three infrastructure gotchas that break SSE","Proxy buffering (disable with X-Accel-Buffering: no or proxy_buffering off), load balancer idle timeouts (send heartbeat comments), and the HTTP/1.1 six-connections-per-origin limit (fixed by HTTP/2).",09-caching-edge
"How do you cancel a generation over SSE?","AbortController on the fetch closes the connection, the server observes the disconnect and stops generating. Durable cancellation across a partition needs an explicit POST /cancel with the request ID.",09-caching-edge
"Why does PKCE apply to confidential clients that already have a secret?","The secret authenticates the client application; it does not bind the code to the specific authorization request. PKCE closes authorization code injection, which is why RFC 9700 and OAuth 2.1 require it universally.",10-security
"PKCE vs state vs nonce","PKCE binds the code to the client. State binds the callback to the browser session (CSRF). Nonce binds the ID token to the authorization request. You want all three.",10-security
"What does PKCE not protect?","The access token. A stolen bearer token is fully usable until it expires. Sender-constrained tokens (DPoP, RFC 9449, or mTLS binding, RFC 8705) are the control for that.",10-security
"Burn rate, defined","Observed error ratio divided by (1 minus the SLO target). Burn rate 1 exhausts the budget exactly at the end of the window; 14.4 exhausts a 30-day budget in about two days.",12-sre-observability
"The standard multi-window burn-rate alert set","Page at 14.4x over 1h with a 5m short window (2 percent of budget); page at 6x over 6h with a 30m short window (5 percent); ticket at 1x over 3d with a 6h short window (10 percent).",12-sre-observability
"Why two windows in a burn-rate alert?","The long window establishes significance; the short window confirms it is still happening, so the alert clears when the incident does instead of firing for hours afterwards.",12-sre-observability
"Error budget minutes per 30 days at 99.9 and 99.99 percent","99.9 percent is 43 minutes 12 seconds. 99.99 percent is 4 minutes 19 seconds.",12-sre-observability
"Three 99.9 percent dependencies called in series give what availability?","0.999 cubed, about 99.7 percent, which is 2h 10m a month rather than 43m. You cannot be more available than the product of your hard dependencies.",12-sre-observability
"Canary vs A/B test","Canary asks whether the version is safe: operational metrics, minutes to hours, asymmetric decision, automatic rollback, owned by the deploy system. A/B asks whether the change is better: product metrics, days to weeks, pre-registered hypothesis and power analysis, owned by the experimentation platform.",13-deployment-delivery
"Why compare a canary against a contemporaneous baseline cohort?","The whole fleet has a different scale so percentiles are not comparable, and last week has a different traffic mix. The baseline must be a freshly deployed cohort of the old version, at the same size, running at the same time.",13-deployment-delivery
"What can a canary not catch?","Slow-burn data corruption, scale-dependent failures such as pool exhaustion at full traffic, coordination bugs needing both versions to interact, and anything with a period longer than the bake time.",13-deployment-delivery
"Minimum detectable effect for a payments canary at 200 QPS","Detecting 0.1 to 0.2 percent error rate needs about 23,500 requests per arm. A 1 percent canary is 2 QPS, so 3.3 hours. A 30-minute 1 percent canary cannot detect a doubling.",13-deployment-delivery
"Why does a Cassandra delete write a tombstone instead of removing data?","Replicas reconcile by last-write-wins with no coordinator. Silently removing a row would let anti-entropy repair copy it back from a replica that missed the delete, resurrecting it permanently.",03-storage
"What is gc_grace_seconds for, and what breaks if you lower it?","It keeps tombstones alive long enough for repair to carry them to every replica. Default 10 days. Lower it below your verified full-repair cycle and deleted rows resurrect silently.",03-storage
"Why did a Cassandra range query start timing out?","Tombstone accumulation in the scanned range. The reader must merge every tombstone in memory. The WARN log prints live rows and tombstone cells for the exact query.",03-storage
"Two silent sources of Cassandra tombstones","TTL expiry, which generates one tombstone per expired cell, and writing null in an UPDATE, which writes a cell tombstone. Both are far more common than explicit DELETE.",03-storage
"Structure for any leadership role-play question","First move, information I would gather, line I would not cross.",16-leadership
"Comment taxonomy for code review","blocking: / suggestion: / nit: / question: / praise:. Most review harm comes from ambiguity rather than tone; an author who cannot tell a nit from a blocker treats everything as blocking.",16-leadership
"Review order to state out loud","Correctness, then design and boundaries, then tests, then readability, then nits.",16-leadership
"SCOR, and why not STAR","Situation, Complication, Options, Result and reflection. The Options slot is what makes a story sound like judgment rather than a resume bullet; STAR has no slot for it.",16-leadership
"Little's Law, and the two ways to use it","L = lambda x W: concurrency = throughput x latency. Sizing: 3000 QPS at 50 ms needs 150 in flight. Inverted, finding the ceiling: a pool of 50 at 50 ms caps you at 1000 QPS no matter how many app servers you add.",02-distributed-systems
"The connection-pool death spiral","A downstream slows, so W rises; L = lambda W means concurrency rises; the pool is finite so requests queue; queueing adds wait, so W rises again. Latency feeds back into concurrency, which is why saturation is a cliff rather than a slope.",02-distributed-systems
"Why run at 70 percent utilisation?","Wait scales as 1/(1-rho): 2x at 50 percent, 3.3x at 70, 5x at 80, 10x at 90, 20x at 95. The knee is 70 to 80 percent. It is arithmetic, not conservatism.",02-distributed-systems
"What is write skew?","Two transactions read overlapping rows, decide, and write disjoint rows. No write-write conflict, so snapshot isolation lets both commit and an invariant over the set breaks silently. PostgreSQL REPEATABLE READ is snapshot isolation and permits it.",02-distributed-systems
"Three fixes for write skew, best first","Express the invariant as a database constraint (a counter row with a CHECK, or an exclusion constraint) so there is nothing to skew; materialise the conflict with SELECT ... FOR UPDATE; or use SERIALIZABLE, which needs application retry logic on SQLSTATE 40001.",02-distributed-systems
"Is serialisability stronger than linearisability?","Neither. Orthogonal. Linearisability is recency on a single object; serialisability is isolation across objects with no real-time requirement. Strict serialisability is both.",02-distributed-systems
"The first sentence of any DynamoDB answer","Access patterns first, schema second. The key schema is the query plan, and you cannot change it later without rewriting the data.",03-storage
"Why single-table design?","Not to save on table costs. It is so one Query against one partition returns a heterogeneous set of related items, which is DynamoDB's only mechanism for retrieving related data in one round trip. The join happens at write time.",03-storage
"GSI vs LSI","GSI: own partition key, own capacity, eventually consistent, addable later. LSI: shares the base partition key, strongly consistent, must be created with the table, and caps a partition key value at 10 GB. Default to GSIs.",03-storage
"Do virtual threads make WebFlux obsolete?","They remove the main reason most teams adopted it (scaling I/O concurrency) without losing stack traces, debuggers or thread-locals. They do not replace demand signalling across a network boundary, which is what request(n) gives you. Default to virtual threads on JDK 21+; use reactive for real streaming backpressure.",04-streaming-apis
"What is pinning, and what changed?","A virtual thread that cannot unmount from its carrier. Native frames and class initialisers pin. synchronized pinned through JDK 23 and no longer does in the common cases as of JDK 24 (JEP 491), so the answer depends on your runtime version.",04-streaming-apis
"You switch to virtual threads and throughput is flat. Why?","Threads were not the bottleneck. Check the connection pool (Little's Law caps you at N/L), then a saturated downstream, then pinning. Also: the old thread pool was an accidental rate limiter, so add an explicit semaphore per downstream.",04-streaming-apis
"Why doesn't caching fix GraphQL N+1?","The problem is 50 lookups for 50 different keys, which is batching not caching. GraphQL is a POST with the query in the body so HTTP caching does not apply without persisted queries. And client-shaped queries are unique by construction, so a response cache is cold.",04-streaming-apis
"Two rules for a DataLoader batch function","Return results in the same order and length as the key array, with null for misses (databases return their own order and omit missing rows, which silently misattributes records). And never make the loader a module-level singleton, because its memoisation crosses users.",04-streaming-apis
"The five slots in a context budget","System instructions, tool schemas, retrieved context, conversation history, and the output reserve, which comes off the top because input and output share the window. Order them stable to volatile so the prefix caches.",06-context-agents
"Prompt caching economics","A read costs about 0.1x input price; a write costs 1.25x on a short TTL or 2x on a long one. Break-even is two requests short-TTL, three long-TTL. It is a prefix match, so one timestamp in the system prompt disables it entirely.",06-context-agents
"Where does the cache breakpoint go in a multi-turn conversation?","At the end of the most recently appended turn, not at the end of the system prompt. Within a few turns the history dwarfs the system prompt, so that is where the tokens are.",06-context-agents
"What is the recall cliff in filtered vector search?","Below roughly a percent filter selectivity, HNSW traversal both slows and loses recall, because most visited neighbours fail the predicate and the graph can be disconnected with respect to the filter. Raising ef_search does not reliably fix it; exact search over the surviving set does.",07-search-ranking
"Why never post-filter an ACL?","Two reasons. Recall: with a 1 percent filter, a top-100 yields one survivor. Security: the number of results you drop leaks the existence of documents the user cannot see, which is information disclosure even when content is never returned.",07-search-ranking
"Why is interleaving more sensitive than A/B?","The comparison is within-impression rather than between-population, so the user is their own control and user-behaviour variance drops out. Published validations report one to two orders of magnitude fewer impressions for the same power.",07-search-ranking
"Interleaving vs A/B, in one line","Interleaving picks the ranker; A/B decides whether to ship it. Interleaving measures relative ranker preference only and cannot see revenue, latency or retention.",07-search-ranking
"Diagnostic order for a slow-but-healthy pod","Shape first (all pods or one, tail or uniform, what changed). Then CFS throttling, GC and major faults, per-dependency latency AND pool wait, DNS and ndots, node steal time and run queue, disk and network saturation, probe config, then profile.",08-compute-kernel
"Why is ndots:5 a latency problem?","Any hostname with fewer than five dots is tried against each search domain first, so resolving api.stripe.com issues three or four failing queries before the real one, doubled for A and AAAA. Fix with a trailing dot, a lower ndots in dnsConfig, or NodeLocal DNSCache.",08-compute-kernel
"You cannot revoke a stateless credential without...","...reintroducing state. The design question is where to put it and how much. Shrink the window first with 5 to 15 minute access tokens, then do the real revocation at the refresh boundary, which already talks to the issuer.",10-security
"Watermark vs jti denylist","A per-user tokens_valid_after watermark is one entry per user, written only on an event, and it maps onto the cases that matter (password change, account disable, global logout). A jti denylist grows with revoked tokens but can revoke one session while leaving others alive.",10-security
"Refresh token rotation with reuse detection","Each refresh issues a new refresh token and invalidates the old. If an already-used token is presented again, you cannot tell theft from a lost response, so revoke the entire family. The legitimate user re-authenticates; the attacker is locked out.",10-security
"Redux: what is server state?","Anything that originates from an API, can change without this browser doing anything, or would be different after a page reload. Typically 60 to 70 percent of a large store. It belongs in a server cache library, not a reducer.",11-frontend
"What do you lose migrating Redux to a query cache?","Normalised entity consistency. Entity adapters keep one order in one place so an edit updates every screen; a query cache is keyed by request, so the same entity can live in two entries and you must invalidate both.",11-frontend
"What is metrics cardinality, and what is the cost model?","Series = the product of every label's distinct values, and each active series costs a few kilobytes of memory plus index. The cost model is series count, not sample rate. One unbounded label (a user ID) makes it unbounded.",12-sre-observability
"Why is a cardinality OOM worse than it looks?","Because recovery is slow: WAL replay of millions of series takes minutes during which nothing is scraped and no alerting rule evaluates, and a memory-constrained pod can OOM again during replay and crash-loop.",12-sre-observability
"The cardinality policy that actually works","Three layers. A rule with a rationale (metrics are for aggregates; identifiers belong in traces and logs, linked by exemplars). Enforcement in the collection path (metric_relabel_configs, sample_limit) so a bad target fails loudly. And alerting on head series plus the churn rate.",12-sre-observability
"Cognitive load, operationally","Not workload. Count what a team must hold: services on call for, distinct domains, runtimes, integrations, stakeholder groups. The indicators are onboarding time, bus factor per area, interrupt rate, and deploy coupling.",14-architecture-patterns
"How do you validate a proposed team boundary?","Co-change analysis over six months of commits. If the cut runs through directories that change together in a large fraction of commits, it converts an in-team change into a cross-team negotiation and the boundary is wrong.",14-architecture-patterns
"The five extraction forces for a microservice","Independent scaling, independent deploy cadence for separate teams, fault isolation in-process bulkheads cannot provide, genuine polyglot need, regulatory or residency separation. Codebase size, clean boundaries and team autonomy are not on the list.",14-architecture-patterns
"How do you enforce modular-monolith boundaries?","Three mechanisms, all required. Compile-time or CI dependency rules (module system, ArchUnit, import-linter) so a cross-boundary import fails the build. One database schema and role per module so a module physically cannot read another's tables. And code ownership on each module's public API.",14-architecture-patterns
"Why double-entry rather than a balance column?","Because it gives you a checkable invariant: the sum of every entry is zero, per currency, always. That turns errors from silent and unbounded into detectable within one reconciliation cycle, and it gives you the audit trail.",15-design-answers
"How do you prevent an overdraft race in a ledger?","Make the check part of the write: UPDATE balances SET balance = balance + delta WHERE account = ? AND balance + delta >= 0, and treat zero affected rows as insufficient funds. Read-check-then-write is a write-skew bug.",15-design-answers
"What is the real bottleneck in a payments ledger?","The hot revenue or fee account that every transaction credits. Row-level lock contention on it caps throughput long before the database is otherwise stressed. Fix with sharded counters, or no projection at all for accounts nobody needs a real-time balance for.",15-design-answers
"The reliability-investment reframe","You are not asking for capacity, you are pointing out you already spend it invisibly at a worse exchange rate. Open with the number: unplanned work as a percentage of capacity, and its trend.",16-leadership
"The sentence that makes a capacity ask credible","And if the number hasn't moved by the review date, we should stop rather than keep spending. It converts a request into an experiment, and it is what makes the second ask easy.",16-leadership
"Error budget policy, and the catch","While inside the SLO the team ships at full speed; when the budget is exhausted, feature work pauses until it recovers. The catch is that leadership must sign it before the budget runs out, not during the incident.",16-leadership
"RTO vs RPO","RTO is time to restore; RPO is how much data you may lose. Independent: a system can fail over in 30 seconds and lose an hour of writes, or take 8 hours and lose nothing.",17-dr-multiregion
"Why can't you have RPO zero across regions?","Zero RPO needs synchronous replication, which makes every write wait for the remote acknowledgement. Light in fibre gives roughly 1 ms per 100 km round trip, so London to Virginia adds ~75 ms per write. Practical answer: synchronous in-region, asynchronous cross-region, so your RPO is your replication lag.",17-dr-multiregion
"The three DR dependencies everyone forgets","Identity provider, DNS (and its TTL, which bounds failover speed), and the secrets manager. Your real RTO is the max of your critical dependencies' RTOs, not your own.",17-dr-multiregion
"The reverse-due-diligence mechanic","Ask the same question of three people and compare. What does success look like at six months, and who decides whether it happened? Divergent answers mean nobody has agreed what the role is for. Write the answers down between rounds.",18-offer-and-questions
"How to ask a hard question without being adversarial","Ask about the past rather than the present, ask for a specific instance rather than a characterisation, and give permission to be honest. Walk me through your last incident and whether the action items shipped beats is reliability a problem here.",18-offer-and-questions
"The staff design-round structure","5 min clarify (requirements as numbers), 3 min capacity math out loud, 3 min API contract, 5 min data model and partition key, 10 min architecture at container level, 12 min deep dive (let them pick), 5 min failure modes and degradation, 2 min tradeoffs.",01-interview-mechanics
"The single biggest differentiator in a design round","Doing capacity math out loud and then using the result to justify a decision. It takes three minutes, almost nobody does it, and it converts every later choice from preference into consequence.",01-interview-mechanics
"Why does RRF beat weighted score blending?","BM25 is unbounded and corpus-dependent; cosine is bounded and compressed; min-max normalisation is computed against the candidate set so the same document normalises differently per query; and score distributions vary by query so a fixed weight is wrong for most. Rank means the same thing everywhere.",05-ai-llm
"What does k=60 do in reciprocal rank fusion?","Damps the difference between top ranks. At k=60 rank 1 contributes only ~1.15x what rank 10 does, so a document must rank decently in several lists rather than topping one. That rewards agreement between retrievers over confidence within one.",05-ai-llm
"The two metric families for RAG","Retrieval: recall@k, NDCG, MRR, context precision. Generation: faithfulness (every claim traceable to the retrieved context) and answer relevance. Recall@k is the ceiling; no prompt work gets you above it. One combined accuracy number cannot direct any work.",05-ai-llm
"Three LLM-judge biases and their fixes","Position bias (evaluate both orderings and require consistency), verbosity bias (instruct against length and monitor score-length correlation), self-preference bias (use a different model family as judge). And validate the judge against human labels before trusting it.",05-ai-llm
"How much does GQA buy you?","The grouping factor, directly, because KV cache size is linear in KV head count. 64 query heads with 8 KV heads instead of 64 is an 8x smaller cache, so 8x more concurrent sequences at the same memory.",05-ai-llm
"What did PagedAttention fix?","Fragmentation. Naive allocation reserves max sequence length per request because the kernel wants contiguous memory, wasting 60-80 percent. Paging allocates fixed blocks on demand via a block table, so waste is at most one partial block, and blocks can be shared for a common prefix (prefix caching).",05-ai-llm
"Why does batching transform decode but not prefill?","Amortisation. In decode the GPU reads every weight from HBM to produce one token per sequence, so reading the weights once serves the whole batch. In prefill the GPU is already saturated with arithmetic from a single long prompt, so batching just queues them.",05-ai-llm
"LLM cost levers, in order","Measure per feature with an outcome field; prompt caching (usually broken by something dynamic in the prefix); context trimming, especially retrieval k; model routing cheap-first with escalation; batch APIs; then semantic caching and distillation last.",05-ai-llm
"Why does escalation rate matter more than price ratio in model routing?","Escalated requests pay for both calls. At 30 percent escalation with a fifth-price model you save about half; at 60 percent you save almost nothing and have added latency to most requests.",05-ai-llm
"The lethal trifecta","Private data, exposure to untrusted content, and a way to communicate externally. An agent with all three is exploitable by indirect prompt injection. Remove any one leg and the attack cannot complete; the cheapest leg is usually egress.",05-ai-llm
"Why don't prompt-level injection defences work?","Instruction and injected text share one channel, so you are asking the model to make a probabilistic judgement about which text is more authoritative, and the attacker has unlimited attempts. Every published prompt-level defence has been broken. Architecture is the defence.",05-ai-llm
"What is the dual-LLM pattern?","Privilege separation. A quarantined model reads untrusted content and returns only constrained structured output; a privileged model with tools and private data acts on that structure and never sees the raw text. Cost: the privileged model has less context.",05-ai-llm
"Why is per-step accuracy useless for agents?","It compounds. 95 percent per step over 20 steps is 0.95^20, about 36 percent task success. Use task-level success against a checkable end state, including an assertion that nothing else changed.",05-ai-llm
"What is pass^k and why does it matter?","The fraction of tasks that succeed on all k independent attempts. pass@1 asks whether the agent can do the task; pass^k asks whether it reliably does. Agents degrade sharply as k rises, and consistency is what decides shippability.",05-ai-llm
"Where do most agent failures actually live?","Tool design, not model capability. Wrong tool selected means the description does not say when to call it; wrong arguments means the schema is too permissive. Tool-layer fixes routinely move pass@1 20 points with no model change.",05-ai-llm
"Why is fixed-size chunking usually wrong?","It cuts where the token counter says rather than where the meaning ends, so a procedure gets split and neither half is retrievable or usable. Recursive character splitting is strictly better at the same cost; structure-aware is better still.",05-ai-llm
"The highest-return chunking change","Prepending the heading path to each chunk. One line, free at query time, and it fixes the dominant failure in structured corpora: an orphaned fragment that contains the answer but nothing connecting it to the question.",05-ai-llm
"The tail-at-scale arithmetic","Fan out to N servers each with probability p of being slow, and the aggregate is slow with probability 1-(1-p)^N. At N=100 and p=1 percent, 63 percent of requests are slow. The tail at the leaf becomes the median at the root.",02-distributed-systems
"What percentile must you control under fan-out?","The one your fan-out demands: solve (1-p)^N >= target. At N=60 and a 99 percent target you need each backend's p99.98, not its p99.",02-distributed-systems
"Hedged requests, and the risk","Send to one replica; if no response by the p95, send a duplicate and take the first answer, cancelling the loser. Extra load is bounded at a few percent. The risk is that under overload hedging amplifies, so gate it on the observed hedge rate.",02-distributed-systems
"Why is exactly-once delivery impossible?","Two Generals. Over an unreliable channel no finite protocol lets both parties agree a message was received, because the acknowledgement can be lost. What you can build is exactly-once effects: at-least-once plus deduplication.",02-distributed-systems
"Five things an idempotency key mechanism needs","Scope the key by tenant and operation; store the result not just the key; handle in-flight explicitly with a 409; hash the request so the same key with different parameters is a 422; and propagate the key downstream.",02-distributed-systems
"The dual-write problem and its fix","Writing to a database then publishing an event are two operations with no shared transaction, so a crash between them leaves them permanently inconsistent. Fix: the transactional outbox, writing the event in the same transaction, with an at-least-once publisher and idempotent consumers.",02-distributed-systems
"What is a watermark?","An assertion flowing through a stream that no events earlier than T are still expected, so a window can close. It is a heuristic, which makes it a completeness-versus-latency dial rather than a guarantee.",04-streaming-apis
"Your Flink job consumes normally and emits nothing. Why?","A stalled watermark. An operator's watermark is the minimum across its inputs, so one idle partition freezes the whole job: no window fires, no error, normal-looking lag. Fix with an idleness timeout on the source; alert on watermark lag.",04-streaming-apis
"What must be true of your sink if you use allowed lateness?","It must be idempotent or upsert-capable, because a late event causes the window to re-fire with an updated result. An appending sink double-counts.",04-streaming-apis
"News feed: push, pull or hybrid?","Hybrid, and the reason is the follower distribution, not preference. Reads outnumber writes ~100:1 so precompute at write time, but one account with 50M followers makes a single post 50M timeline writes. Push below a threshold, pull above it, merge at read time.",15-design-answers
"Why is the celebrity pull path cheap?","A celebrity's recent posts are identical for every follower, so that cache entry is written once and read by millions. And the number of celebrities any one user follows is small, so the merge is bounded.",15-design-answers
"CVSS vs EPSS vs KEV","CVSS scores intrinsic severity, EPSS predicts probability of exploitation in the next 30 days, KEV is CISA's catalogue of confirmed active exploitation. Three different questions; a priority is the combination plus your exposure.",10-security
"The patching order","KEV first, because observed exploitation beats any prediction. Then EPSS times exposure. Then CVSS times exposure times data sensitivity. 'We patch all criticals in seven days' without exposure context is the junior answer.",10-security
"What bounds your patching velocity?","Inventory accuracy. You cannot patch what you do not know you run, and most organisations fail at that step rather than at applying the patch. An SBOM registry that answers 'who uses this library' in minutes is the first investment.",10-security
"An engineer missed three commitments. First move?","Not the conversation. An hour of preparation, because it is a symptom with about six causes and at least two of them are management failures: interrupt load, and someone not feeling able to say work was slipping.",16-leadership
"How does a missed-commitments conversation end?","One specific change from them, one specific change from you, a check-in date close enough to be real, and a written summary the same day. No surprises at review time only works if the conversations happened and were recorded.",16-leadership
"Zero trust in one sentence for a director","Being on our network currently means being trusted, so a phished laptop can reach the customer database. Zero trust checks every request against who you are, what device you are on, and whether you should have that specific access, every time. One compromised laptop stops being one compromised company.",10-security
"The zero trust sequencing, and why","Identity, device, workload, network, data. Identity first because everything downstream needs a reliable answer to who is asking, and SSO is the one security project users like, which makes it fundable. Network is fourth because segmenting by IP is brittle; you want to segment by workload identity.",10-security
"Zero-downtime reindex, in one line","Applications read and write through an alias, never a concrete index name, so the switch is one atomic _aliases call and the rollback is the same call reversed.",03-storage
"Two settings that halve reindex time","Zero replicas and refresh_interval -1 during the bulk load, because both multiply indexing work and neither is needed while nobody reads the index. Restoring them before the swap is the step people forget.",03-storage
"How do you handle writes during a reindex?","Dual-write from the application, or repeated delta passes filtered on updated_at with version_type external so they are idempotent and cannot overwrite newer documents with older ones, or replay from the source of truth if you have CDC.",03-storage
"Deploy vs release","Deployment moves bits onto infrastructure; release exposes behaviour to users; a feature flag decouples them. Deploy continuously, release deliberately.",13-deployment-delivery
"The biggest benefit of decoupling deploy from release","Not rollback speed, though 4 seconds beats a 25-minute redeploy. It is that trunk-based development becomes possible, because unfinished work merges behind a disabled flag, which removes long-lived branches and merge hell as a category.",13-deployment-delivery
"What does NOT decouple with a feature flag?","Database schema (needs expand-contract), cached and serialised data (version cache keys with the schema, or the old path reads a format it cannot parse), published events (consumers deploy before producers), and irreversible external side effects.",13-deployment-delivery
"The four feature-flag types","Release (days to weeks, deleted after rollout), ops/kill switch (permanent infrastructure), experiment (owned by the A/B platform), permission/entitlement (permanent business rule). Only release flags need an expiry policy.",13-deployment-delivery
"Contract testing, in one line","Consumers declare what they need in executable form; consumer tests run against a mock generated from that; the provider's CI replays every consumer's contract against the real provider. The two sides never run at the same time, so no shared environment is needed.",13-deployment-delivery
"What contract testing cannot catch","Emergent behaviour. Two services can each satisfy their contracts and produce a wrong outcome together, like marking an order shipped before payment settled. Keep three to five end-to-end tests plus synthetic monitoring for that.",13-deployment-delivery
"Why 40 services is tractable for contract testing","The interaction graph is sparse. Forty services typically have fewer than 100 real consumer-provider edges, not 1,560, so the work scales with edges rather than with the square of the node count.",13-deployment-delivery
"A director wants a date you cannot commit to. First move?","Ask what the date is anchored to. A contract, a customer commitment already made, a conference and a stretch target need completely different responses, and the real constraint often has a better answer than either party started with.",16-leadership
"How do you quote a date credibly?","With percentiles from historical cycle time, not story points: 60 percent confidence on the 29th, 90 percent on the 12th. Points measure imagined effort; cycle time measures what happened. And commit to 60-70 percent of theoretical capacity.",16-leadership
"Two teams building the same service. First move?","Verify the duplication is real, because they are often solving different problems that look alike. Then quantify the cost in engineer-years, confused consumers and any live correctness divergence, and translate it into whatever the decision-maker already said they wanted capacity for.",16-leadership
"How do you make a consolidation actually happen?","Give the losing team something real: they own the migration, their distinctive features get ported, their lead is named as a contributor. 'Your year of work is deleted' is why consolidations get agreed and then quietly not done.",16-leadership
"Promotion, one level short: the question that decides whose problem it is","Have they had the opportunity to demonstrate what is missing? If the gap is cross-team scope and every project you assigned was inside the team, the gap is yours, because at this level the evidence comes from the work someone is assigned.",16-leadership
"What do you promise in a promotion conversation?","The packet and your advocacy, never the outcome, because you do not control the calibration room. Plus a commitment to say in January if it is not tracking, so they do not find out in March.",16-leadership
"Serverless vs containers: the crossover","Around 30-40 percent average utilisation. Below it, per-request billing wins; above it, per-time billing wins because the time is fully used. Committed-use discounts push the crossover down to roughly 20 percent.",08-compute-kernel
"How do you size the container for the comparison?","Little's Law: concurrency equals throughput times latency. 50 requests/sec at 200 ms is 10 in flight, so about 2 vCPU with headroom, not a guess.",08-compute-kernel
"The constraint that overrides serverless cost math","Connection management. A function per invocation cannot hold a pool, so hundreds of concurrent functions exhaust the database. That is the most common way function architectures fail at scale, and it fails rather than merely costing more.",08-compute-kernel
"Where is the durability boundary?","fsync. write() returns when data is in the page cache, which is RAM, so it survives a process crash and nothing else. fsync pushes to the device and flushes the device cache. Commit path: append to WAL, fsync WAL, then acknowledge.",08-compute-kernel
"fsyncgate, in one line","On Linux a writeback failure can be reported once and the dirty pages then marked clean, so a retried fsync returns success against data that is gone. PostgreSQL 12+ panics on fsync failure rather than retrying.",08-compute-kernel
"synchronous_commit = off vs fsync = off","The first loses a bounded window (about 200 ms) of committed transactions and leaves the database consistent. The second risks corruption. Teams conflate them.",08-compute-kernel
"Why is etcd sensitive to disk latency?","Every Raft log append is an fsync before a follower can acknowledge, so commit latency is local flush plus a quorum round trip. A 10 ms fsync caps cluster write throughput regardless of network speed.",08-compute-kernel
"eBPF: the signature that means reach for it","The caller and callee disagree. Trace says the DB call took 340 ms, the DB says 4 ms. That gap is client-side and no application instrumentation can see it.",08-compute-kernel
"The four eBPF questions, in order","Off-CPU (offcputime, runqlat), block I/O (biolatency, biosnoop), network (tcpretrans), then application internals (funclatency on a uprobe). Off-CPU first, because most intermittent latency is time not spent on CPU, which a sampling profiler cannot see.",08-compute-kernel
"Why a CPU profiler misses intermittent latency","It samples on-CPU time. The latency is usually off-CPU: run-queue wait, lock, page fault, disk, or a TCP retransmit. Off-CPU flame graphs are the artifact that answers it.",08-compute-kernel
"Cache invalidation with many dependencies: the first deliverable","A table, per entity: change rate, tolerable staleness, and fan-out per change. Price changes 50k/day with zero tolerance and fan-out 1. A category changes 20/day, tolerates an hour, fan-out 80,000. One strategy cannot be right for both.",09-caching-edge
"Tags or versioned keys?","Fan-out decides. Tags (surrogate keys) below roughly 1,000 objects per change: precise, no garbage. Versioned keys above it: one INCR beats 80,000 deletes, at the cost of superseded entries sitting until eviction.",09-caching-edge
"The subtle bug in dependency-index invalidation","The index expiring before the entries it tracks. The purge finds an empty set, deletes nothing, and pages stay stale until their own TTL. Give dependency sets a TTL strictly longer than the entries.",09-caching-edge
"Why CDC for cache invalidation instead of app-emitted purges?","An app-emitted purge is a line of code someone can forget: the admin tool, the importer, the migration, the manual data fix. CDC reads the write-ahead log, so it captures every write path including the ones that bypass your service.",09-caching-edge
"Why keep a TTL when purging works?","A purge is a message and messages get dropped. A purely purge-driven cache with an infinite TTL has no self-healing path: one lost message means one wrong value served forever.",09-caching-edge
"When a knowledge graph beats a vector index","Multi-hop questions where no chunk contains the chain; global questions about the corpus where no value of k works; relationship questions like dependency impact; and anywhere the reasoning path must be auditable.",06-context-agents
"The move that makes a knowledge graph affordable","Noticing how much of it already exists in structured systems: service catalogue, tracing data, org directory, ticket system. LLM extraction only for relationships that exist purely in prose.",06-context-agents
"GraphRAG global search, mechanically","Leiden community detection over the graph, hierarchically, then a pre-generated LLM summary per community per level. A global question maps over those summaries and reduces. No vector-index equivalent exists.",06-context-agents
"The three shard key tests","Cardinality (can you split?), frequency (is one value dominant?), monotonicity (does every insert land on one shard?). A key can pass one and fail the others: _id has perfect cardinality and is the worst common choice.",03-storage
"Why a monotonic shard key is fatal","Every new document has the highest value, so every insert lands in the top chunk, which lives on one shard. You get an N-shard cluster with single-shard write throughput.",03-storage
"What is a jumbo chunk?","A chunk past the size limit that cannot be split because every document in it shares one shard-key value. The balancer then refuses to move it. It is the failure mode of poor frequency spread.",03-storage
"Compound shard key: the routing rule","Only queries containing the prefix fields route. A query on the suffix alone broadcasts, exactly like a compound index.",03-storage
"Why a column rename takes five deploys","A rolling update is not atomic. Both versions serve traffic for minutes, so a bare RENAME throws on every pod not yet replaced. And rollback is a deploy backwards, so old code must work against the new schema too.",13-deployment-delivery
"The event compatibility ordering rule","Additions: producers deploy first, consumers ignore the unknown field. Removals: consumers deploy first, then the producer stops emitting. It is the reverse of intuition and getting it backwards takes down every consumer at once.",13-deployment-delivery
"Why did the rollback fail on a cache format change?","The new version left state the old one cannot read, and with a TTL the cache does not self-heal. Fix: version the cache key so the two cannot collide, and treat an unknown payload version as a cache miss rather than an exception.",13-deployment-delivery
"The five-minute habit that prevents rollback failures","Ask what state this version writes that the previous one cannot read: schema, cache, sessions, published events, object storage, queue messages. Schema migrations get reviewed; a serialisation change in a cached object does not.",13-deployment-delivery
"INP's three phases","Input delay (main thread busy), processing (your handler), presentation (style, layout, paint). The phase split is the diagnosis. Presentation is often largest and is invisible in a JavaScript profiler.",11-frontend
"Why INP replaced FID","FID measured only the delay before the first interaction's handler started. A page could score well while every interaction after the first took half a second to show a result. Teams that 'fixed FID' often have the worst INP.",11-frontend
"INP thresholds","Good under 200 ms, poor above 500 ms, at the 75th percentile of real user sessions. The reported value is roughly the worst interaction, with one outlier discounted per 50 interactions.",11-frontend
"The INP fix that requires nothing to get faster","Paint the pending state first, yield, then compute. INP measures time to the next paint, not time to complete the work, so this alone can take an interaction from 400 ms to 30 ms.",11-frontend
"The micro-frontends test","Does the teams' work co-render on the same page? If teams own separate routes, split by route: real deploy independence, no shared cascade or React, because two routes never render at once. Same page means genuine micro-frontends.",11-frontend
"The constraint that usually kills micro-frontends","Module Federation's singleton: true means one React across all fragments, so every team upgrades together, which is the coordination you were removing. Drop it and you ship two Reacts and hooks break across the boundary.",11-frontend
"Chaos engineering: what makes it an experiment","A steady-state metric in user-visible terms, a hypothesis, a chosen blast radius, and automated abort conditions. Without a hypothesis there is no result, only an incident.",12-sre-observability
"Why run chaos experiments during business hours?","You want the people who understand the system awake and watching. Running at 3am minimises the customers affected and also the people capable of noticing. If it is too risky to run at 2pm, it is too risky to run.",12-sre-observability
"What chaos experiments actually find","Fallback code that has silently rotted, because it is the least-executed code in the system. Alerting that cannot detect the failure at small blast radius. Runbooks that no longer match. Two of the three are observability findings.",12-sre-observability
"Latency or failure injection first?","Latency. Dependencies rarely die cleanly, they slow down, and systems handle slow far worse than dead: a clean failure trips a breaker, while slowness fills thread pools and propagates backpressure.",12-sre-observability
"The repository pattern, precisely","A collection-like interface over persistence, one per AGGREGATE ROOT, not one per table. Per-table is a DAO with a fashionable name and gives up the boundary that made the pattern worth having.",14-architecture-patterns
"The test for whether a repository earns its cost","Is there an invariant that would be violated if code could load a partial version of this thing? Order with line items and a cancellation rule: yes. A reporting endpoint producing a screen: no, that is a query object.",14-architecture-patterns
"The strongest argument against the repository pattern","Your ORM already implements it (SQLAlchemy Session, EF DbContext are Unit of Work), and it leaks on exactly the properties that matter: cardinality, indexes, locking, N+1. Those decide whether the system works.",14-architecture-patterns
"The CQRS ladder","1: separate handlers (hours). 2: separate read models, same DB (days). 3: separate read store, same transaction (weeks). 4: async projections (months, and consistency changes). Almost everyone asking imagines 4 and needs 1.",14-architecture-patterns
"Which CQRS rung changes consistency?","Only rung 4. Rungs 1 through 3 are strongly consistent. And rung 4 is a product decision, not an engineering one: whether a customer can place an order and not see it for two seconds.",14-architecture-patterns
"CQRS and event sourcing","Independent patterns, frequently deployed together. Conflating them is why teams think CQRS costs months. Most teams asking for event sourcing want an audit trail, which an append-only audit table gives without making replay the recovery path.",14-architecture-patterns
"Converting SCOR to STAR","Complication becomes the Task in first person. Options move to the front of Action, one sentence each with its cost, and you keep all of them. Resolution splits: what you did is Action, what happened is Result. Add an explicit 'what was mine to decide'.",01-interview-mechanics
"Why prepare in SCOR and deliver in STAR","STAR is the scoring rubric; SCOR is a narration order. STAR has no slot for the alternatives, and at staff level the decision is the content. Preparing in SCOR forces the complication and options to exist; they survive the conversion.",01-interview-mechanics
"What makes a scar-tissue story work","Three sentences, under 30 seconds, one number only someone present would know, and it returns to the technical point. 'It caused an outage' is generic; '40 seconds at 100 percent CPU' is a memory. Three or four per interview, not one per answer.",01-interview-mechanics
"Down-levelled offer: the first move","Do not accept, decline or negotiate. Ask which of three things happened: the loop calibrated you lower, the req is scoped lower, or it is an anchor. Ask without mentioning compensation.",18-offer-and-questions
"The highest-conversion move in a level negotiation","Offering an additional conversation with a staff engineer focused on the named gap. It converts your assertion into something they can verify, costs them an hour, and is hard to refuse.",18-offer-and-questions
"The question that actually decides a down-levelled offer","The promotion path: who has gone senior to staff on this team, how long, when are the cycles, what work builds the evidence. 'Definitely possible for strong performers' means no path exists; price the offer as though the level is permanent.",18-offer-and-questions
"Why you cannot roll back a mobile release","Google Play will not let you decrease a staged rollout percentage and users who have the build keep it; App Store phased release pauses but does not reverse. Halting freezes the affected population, it does not shrink it.",13-deployment-delivery
"What turns a 3-day mobile incident into a 20-minute one","A server-side kill switch on the affected feature. On the server a bad deploy is fixed by rollback; on mobile a bad binary is fixed by a flag or it is not fixed for days. Hence: ship every feature dark, enable server-side.",13-deployment-delivery
"The mobile segmentation that is decisive most often","Upgrade versus fresh install. If fresh installs are clean and upgrades crash, it is a migration or restored-state bug that no internal test could catch, because test devices are clean installs.",13-deployment-delivery
"Why the mobile alert fires late","Staged rollout mathematically dilutes a segmented failure. A crash hitting 100 percent of one OS version at 20 percent rollout looks like noise in the aggregate. Alert per OS version and per upgrade path, or staging hides the problem it was meant to bound.",13-deployment-delivery
"Multilingual search: clarify this word first","Nine separate corpora with same-language queries, or cross-lingual where a French query matches an English document? Different systems. Cross-lingual needs one shared embedding space and is the interesting case.",15-design-answers
"Why hybrid rather than dense-only for product search","'iPhone 15 Pro Max 256GB' is exact-match, and a dense encoder puts the 128GB variant at near-identical similarity, so it confidently returns the wrong product. Exact-identifier queries are a large share of commercial traffic.",15-design-answers
"The filtered-ANN recall cliff","HNSW traversal assumes a connected graph; filtering removes nodes and disconnects regions, so the search cannot reach qualifying neighbours that exist. At ~1 percent selectivity you can lose most recall, silently.",15-design-answers
"Filtered vector search: the routing rule","Exact scan under ~10,000 estimated candidates (exact is faster there); partitioned index where the filter is a partition key; widened ef otherwise. Route by cardinality estimate, do not pick one algorithm.",15-design-answers
"Multilingual docs: how many vectors per product?","One. Embed the canonical description and keep per-locale lexical fields. Nine embeddings of one product are near-duplicates competing for the same slots, hurting diversity and inflating the index ninefold.",15-design-answers
"The two freshness paths in search","Text and embeddings: GPU pipeline, under an hour, ~50k changes/day. Price and stock: attribute store, under 5 seconds, ~2M changes/day, applied at RANKING time. Re-embedding on price change costs more GPU than the serving fleet.",15-design-answers
"The number that decides a recsys serving design","30 million item-feature reads per second (60k RPS x 500 candidates). Not servable remotely, so item features live in the serving process: 2M items x 256 fp16 = ~1 GB, broadcast to every replica.",15-design-answers
"The funnel rule","Each stage may only reduce the set, and recall lost at retrieval can never be recovered downstream. Which is why retrieval recall is measured separately from ranking quality.",15-design-answers
"Why two-tower retrieval works","The item tower is expensive and offline (produces the ANN index); the user tower is cheap and per-request. No user-item interaction at retrieval time, which is exactly what makes an ANN index possible and why retrieval ranks worse than the ranker.",15-design-answers
"Where diversity belongs, and why","In the re-rank after scoring, not in the model. A pointwise ranker scores items independently and cannot express 'twenty hiking boots is worse than twelve boots and eight related items'. That is a set-level constraint.",15-design-answers
"What happens if you skip exploration","Short-term engagement improves; over months the training data becomes the model's own past decisions, items never shown never get positive signal, and the effective catalogue narrows. Invisible in daily metrics, expensive to reverse.",15-design-answers
"Hedged requests, and when to use them","Send a second request at the p95 and take the first response. Costs ~5 percent more load, removes the tail. Justified when your p99 is dominated by dependencies' tails rather than your own work, which is the tail-at-scale argument.",15-design-answers
"Exactly-once: the correction to make first","Exactly-once DELIVERY is impossible: a worker dying before ack is indistinguishable from never running. What you build is exactly-once EFFECT: at-least-once dispatch plus idempotent execution.",15-design-answers
"The idempotency key that actually works","Derived from job identity plus scheduled instant (job_id:scheduled_for), never a fresh uuid4 per attempt. Same occurrence, same key; different occurrence, different key. A per-attempt UUID deduplicates nothing.",15-design-answers
"Why SKIP LOCKED and an idempotency key, not just one","SKIP LOCKED stops two schedulers dispatching simultaneously. It does not stop a scheduler crashing after enqueue and before marking dispatched, which re-dispatches on the next poll. The key covers that.",15-design-answers
"The lease bug most implementations have","Not checking ownership on every heartbeat. A worker paused by GC or VM migration wakes up believing it still owns the job after a takeover. The heartbeat must be a conditional update, and zero rows updated means abort immediately.",15-design-answers
"Ticketing: the reframe","Not a scale problem. 50,000 seats is 10 MB and 8,300 RPS is unremarkable. It is a contention problem: 500,000 people want the same rows in the same second. And 90 percent cannot succeed, so most of the system's job is rejecting people clearly.",15-design-answers
"The ticketing guarantee, versus the optimisation","A unique partial index on (event_id, seat_id) where status is sold. That is the guarantee and it holds even if Redis is wrong. Redis holds and conditional updates are optimisation to avoid hitting it constantly.",15-design-answers
"Conditional update vs SELECT FOR UPDATE","Both correct. FOR UPDATE serialises every attempt on a hot seat behind a lock held across a round trip. UPDATE ... WHERE status='available' RETURNING is one atomic statement; zero rows means someone else got it, which is a normal outcome.",15-design-answers
"The admission rate formula","(seats_remaining / expected_conversion) / hold_duration_seconds. 50,000 seats at 40 percent conversion with 600 s holds is about 208 users/sec. It is a control loop, not a constant, and when seats run out you stop admitting and tell the queue.",15-design-answers
"Notifications: the split that drives everything","Three traffic classes on separate topics: transactional, social fan-out, campaign. A priority field does not help when the consumers are already busy with a 200M campaign. The real failure is a password reset queued behind marketing.",15-design-answers
"A celebrity posts to 50M followers. What is enqueued?","A resumable fan-out JOB, not a message. Cursor pagination in 10k chunks, checkpoint per chunk, bulk preference filtering with one multi-get, voluntary yielding, and the payload stored once by reference (50M copies of 2 KB is 100 GB).",15-design-answers
"The four sources of duplicate notifications","Producer retries, queue redelivery (by design), fan-out chunk replay after a crash, and two services noticing the same event. One derived key, sha256(event_id:user_id:channel), covers all four via SET NX.",15-design-answers
"Preference evaluation order, and why","Legal (consent, unsubscribe), then explicit user choice, then quiet hours in the user's timezone, then frequency caps. Cheapest to most expensive, because most notifications are rejected and rejecting early is free.",15-design-answers
"Quiet hours: drop or defer?","Defer. Dropping means the user never learns about it, which is almost never the intent. The exception is content that is worthless later, which should carry an explicit expiry instead.",15-design-answers
"Why bounce handling is a correctness requirement","Email throughput is limited by reputation, not bandwidth. SES suspends above ~5 percent bounce or ~0.1 percent complaint, so an unhandled hard bounce degrades deliverability for transactional mail too. Hence separate IP pools per traffic class.",15-design-answers
"Ad clicks: the first question to ask","Billing or dashboards? Dashboards tolerate 0.5 percent error and need seconds; billing needs exactness over 24 hours and tolerates hours. Two paths reading one topic, with divergence between them as a monitored metric.",15-design-answers
"Why partition clicks by click_id, not campaign_id","Campaign partitioning gives a hot partition for the biggest advertiser (ad spend is heavily skewed). click_id partitioning is uniform and makes dedupe state local. The aggregation then does a keyBy shuffle, one hop, much cheaper.",15-design-answers
"The three-tier lateness policy","Within the watermark: normal window firing. Within allowed lateness: window re-fires with an updated count, so the sink must upsert not append. Beyond: side output to object storage for batch reconciliation. Never a silent drop.",15-design-answers
"ON CONFLICT DO UPDATE SET, not +=","The window emits a COMPLETE count for that window, so overwriting is correct and adding double-counts on every checkpoint replay. This one distinction is the most common bug in these pipelines.",15-design-answers
"withIdleness: why it is the highest-value Flink config line","The watermark is the minimum across all sources, so one idle partition out of 200 holds the global watermark back and every window everywhere stops firing, while the job reports healthy.",15-design-answers
"Why batch dedupe must be deterministic","MIN_BY(..., ingest_time) always keeps the earliest occurrence, so a rerun produces byte-identical output. A non-unique tie-break means the invoice changes between runs, which an auditor will find.",15-design-answers
"LLM gateway: what to ship first","Usage recording and cost attribution. It is what gets funded, it is read-only so it can ship as a wrapper before anything sits in the request path, and it builds the case for the rest with data.",15-design-answers
"Gateway failover: the distinction that matters","Same model, different provider (direct -> Bedrock -> Vertex) is identical output and can be automatic. A smaller model is cheaper, faster and DIFFERENT, so it is opt-in per route. Silent downgrade during an incident is unattributable quality loss.",15-design-answers
"Where LLM cost actually goes","Prompt structure, not model choice. Stable system prompt and few-shot examples FIRST, variable content LAST, so the provider prompt cache is usable. For a 20k-token prefix and a 200-token question that removes most of the input cost.",15-design-answers
"Why an LLM gateway is connection-bound","Each request holds a connection for seconds. 2,000 RPS at 8 s average is 16,000 concurrent connections, mostly idle. That dictates async I/O, streaming pass-through rather than buffering, and cancelling upstream on client disconnect.",15-design-answers
"Gorilla compression, and why it matters","Delta-of-delta timestamps plus XOR float encoding takes 16 bytes per sample to about 1.37. That 10x is why metrics need a purpose-built store rather than Postgres, and why the in-memory index rather than the samples is the memory constraint.",15-design-answers
"How a metrics system actually dies","Cardinality, suddenly. It is multiplicative across labels, so adding user_id takes a 40,000-series metric to 80 billion. Ingesters OOM, and because ingest is sharded by series hash the bad series spread evenly, so every shard dies at once.",15-design-answers
"'I need user_id on my metric'","Ask what question they are answering. It is 'the p99 spiked, which request?'. The answer is exemplars: a trace id attached to a histogram bucket sample, so you can jump from the spike to a trace without the id becoming a label.",15-design-answers
"The alert-path independence rule","Alert queries must be answerable from recent local data only. An alert ranging over 30 days cannot fire during an object-storage incident, which is exactly when you need it. Enforce it in rule review, not at runtime.",15-design-answers
"Active-active: the question that forces the design","Coordinate on every write (75 ms quorum from us-east across 5 regions, 180 ms from ap-southeast) or do not coordinate and resolve conflicts. There is no third option. So: what is the write latency budget and what does a conflict cost?",15-design-answers
"Why hybrid logical clocks, not wall clocks","With wall clocks, skew decides which write survives: a region 200 ms fast wins every conflict silently. HLC advances a logical counter when the physical clock does not move, so order is monotonic regardless of skew. It does not fix LWW's lost write.",15-design-answers
"Version vectors: what they actually give you","Detection, not resolution. Comparing tells you dominance or genuine concurrency. Per REGION, not per client, because a per-client vector grows without bound. On concurrency you return siblings or merge.",15-design-answers
"The consistency routing rule for a multi-region store","CRDTs for counters, sets, flags. LWW+HLC for profiles and documents. Home-region ownership for uniqueness or monotonicity. Consensus group for balances. One consistency model for the whole store is the mistake.",15-design-answers
"Session tokens in an eventually consistent store","The client carries the version vector it last saw; the serving region waits briefly or reads from the region that has it. Gives read-your-own-writes and monotonic reads across region changes without global coordination.",15-design-answers
"The forgotten cost of active-active","Cross-region egress. 50k writes/sec x 2 KB to 4 peers is 34 TB/day, often a bigger line item than compute. Batching and compression are required, not optimisations. At 10x, selective replication replaces the full mesh.",15-design-answers
"Greedy vs batched dispatch","Batched. Two requests 200 ms apart can each take the other's best driver: about 40 percent more total wait in the simple case. Batching solves an assignment problem optimally (Hungarian, O(n^3)), costing 2-4 seconds of latency.",15-design-answers
"Why H3 hexagons rather than geohash squares","Hexagons have six equidistant neighbours. Squares have four at d and four at d*sqrt(2), which distorts every radius query and heatmap. And geohash prefixes differ across boundaries, so a prefix query silently misses nearby drivers.",15-design-answers
"How to compute 1M ETAs per dispatch batch","You do not. Haversine pruning first, which is EXACT because straight-line is a strict lower bound on road distance, removing 80-90 percent. Then a cached cell-to-cell matrix. Then real routing for the top 20 per request. Then an ML correction.",15-design-answers
"Why dispatch cost is not just ETA","Acceptance probability (a decline costs 15 s and a re-match), a fairness term weighted by waiting time (or marginal locations get passed over forever while the average looks fine), heading (a U-turn is real time), and soon-to-be-free drivers.",15-design-answers
"RAG access control: why the failure is different","A retrieval mistake is not a bad answer, it is a data leak laundered through a model so it appears as the assistant's own words with no provenance. Authorisation is a property of the whole pipeline, not a filter on retrieval.",15-design-answers
"Why two authorisation layers in RAG","Pre-filter makes the candidate set usable but is stale by the index sync interval. Post-filter re-checks each chunk against a fresh read, catching a revocation from two minutes ago. Post-filter alone gives a 2-percent-access user 2 chunks out of 100.",15-design-answers
"Grants vs revocations","Asymmetric. Revocations must propagate in minutes because failing to revoke is a leak; additions can wait hours because failing to grant is an inconvenience. Push revocations, batch additions.",15-design-answers
"Where RAG access systems actually leak","Not the retrieval filter. The cache (an answer keyed by question alone), conversation history (turn 5 still holds turn 1's content after a revocation), citations (a title is information), and prompt logs (an unpermissioned copy of the corpus).",15-design-answers
"The thing to raise before a permissioned RAG launches","It surfaces pre-existing over-permissioning: documents technically readable by everyone but practically undiscoverable become discoverable. Those incidents look like RAG failures and are ACL hygiene failures. Access review is part of the project.",15-design-answers
"Rate limiter: why not Redis INCR per request","It adds ~0.5 ms same-AZ round trip to every request and makes one Redis a hard dependency for the whole API. It is worst on hot keys, where 1,500 RPS means 1,500 coordinated round trips on one key.",15-design-answers
"Why token bucket over sliding window","Capacity and refill rate express burst and sustained rate as separate parameters, which is what an API product sells ('1,000/min, bursting to 100'). No window algorithm can state that. Also immune to clock skew: refill uses elapsed monotonic time.",15-design-answers
"Two-tier rate limiting, and the overshoot bound","Local token buckets decide; a central authority grants leases proportional to observed demand with per-key node heartbeats. Worst-case slack is active_nodes x rate x refill_interval: about 2 percent at 100 ms and 10 nodes on a 1,000/min limit.",15-design-answers
"Fixed window's fatal flaw","100 requests at 11:59:59 and 100 at 12:00:01 is 200 in two seconds against a '100 per minute' limit. The boundary permits double the intended rate, always, and clients that find it will exploit it.",15-design-answers
"Fail open or fail closed? Compare consequences","Rate limiter: fail OPEN, because failing closed turns a Redis outage into a total API outage. Permission check: fail CLOSED, because failing open leaks data. The rule is not a default, it is which consequence is worse.",15-design-answers
"Why a single sequence wastes a GPU","Decode is memory-bandwidth-bound: every token reads all 16 GB of weights, so 7.8 ms at 2 TB/s. The compute used is 2 TFLOPS of 312 available: 0.7 percent utilisation. That is why batching is not an optimisation.",15-design-answers
"Static vs continuous batching","Static returns when ALL sequences finish, so outputs of [20,45,800] keep every slot busy for 800 steps: ~17 percent utilisation. Continuous schedules at the iteration level, so finished sequences leave and waiting ones join every step.",15-design-answers
"Why chunked prefill exists","A 4,000-token prefill is ~410 ms of compute, during which every decoding sequence stalls: 12 missed tokens at 30 tok/s, a visibly stuttering stream. Chunking into 512-token pieces interleaved with decode trades a little TTFT for smooth TPOT.",15-design-answers
"What PagedAttention actually fixes","Over-reservation. Allocating contiguous KV for the declared max length wastes 60-80 percent. Fixed 16-token blocks with a per-sequence block table bound waste to the last partial block, under 4 percent, and enable prefix sharing.",15-design-answers
"The biggest LLM serving win people omit","Prefix sharing. 100 requests with the same 2,000-token system prompt is 25 GB of KV cache without it and 256 MB with it: about 200 more concurrent sequences, larger than anything the scheduler contributes.",15-design-answers
"Set the batch limit from memory or from the SLA?","From the SLA. Memory may allow 460 sequences, but at that batch the decode step is compute-bound at ~25 ms, giving 40 tok/s/user against a 30 floor. Sizing to memory gives great throughput numbers and a stuttering user experience.",15-design-answers
"The agent platform's core security decision","Authorise every tool call as the invoking USER, never as the platform. Structural rather than probabilistic: the worst case becomes the user doing something they could already do manually, which is bounded and auditable.",15-design-answers
"Does a step cap bound agent cost?","No. Context grows each step, so step 25 with a 100k-token context costs far more than step 1. You need four bounds: steps, token budget, wall clock, and a cost cap. The token budget is the one teams omit.",15-design-answers
"Three agent tool isolation tiers","In-process for platform-written pure functions (defends against nothing, is fast). Container with a network policy for team-written tools. MicroVM for anything model-generated, always, because that code is untrusted input in a shared kernel otherwise.",15-design-answers
"Why side_effects must be an enforced field","Replaying send_customer_email sends a second email. So the manifest declares side_effects, and the replay engine only ever uses the recorded result for external ones. A convention here fails once and destroys trust in the tool.",15-design-answers
"Why LLM CI is not normal CI","The system is non-deterministic and there is often no single correct answer, so the gate is 'did quality drop by more than noise', which requires measuring the noise floor by running the unchanged baseline several times.",15-design-answers
"Why re-run the baseline in the same job","Comparing to a stored score conflates your change with provider drift. And pairing removes between-case variance, the dominant term: 200 paired cases detect what 800 unpaired would. It costs double the calls and quarters the sample needed.",15-design-answers
"Eval set size, honestly","At 80 percent baseline, SE is 4 percent at n=100, so a 5-point regression is undetectable. 500 gets you ~5 percent, 2,000 gets ~2.5 percent. Teams gate on 50 cases and believe they have a gate.",15-design-answers
"LLM-as-judge: the three biases","Position (prefers the first option: run both orders), verbosity (prefers longer: control for length), self-preference (prefers its own family: use a different judge). Then validate against human labels; below ~80 percent agreement, gating is worse than not.",15-design-answers
"The eval check that catches what ships","Segment-level gating. A change that improves English 4 percent and destroys Portuguese 15 percent shows as a net win. Safety failures are absolute, not statistical: one injection success blocks regardless of the aggregate.",15-design-answers
"Cost attribution is three different asks","Showback (what did you spend), chargeback (move the money, needs finance-dispute accuracy), and unit economics (cost per outcome). The third is what leadership actually wants even when they ask for the first two.",15-design-answers
"Why compute LLM cost at ingest, not query time","Prices change. Query-time computation means every historical report silently changes when a provider adjusts pricing, month-over-month comparison becomes meaningless, and invoice reconciliation is impossible. Version the price table.",15-design-answers
"The top accuracy problem in cost attribution","Calls that bypass the gateway. If 30 percent of spend is direct, no pipeline makes it correct. The fix is organisational: provider keys issued only to the gateway, so a direct call is impossible rather than discouraged.",15-design-answers
"Cost per team vs cost per outcome","'$84k a month' invites a blunt cut. '$0.20 per resolved ticket against a $6 human handling cost' is a business case, and the same table reveals features costing more than they return, which a per-team report cannot.",15-design-answers
"Why content-defined chunking","Fixed 4 MB chunks: any insertion shifts every subsequent boundary, so a 2 GB file re-uploads. A rolling hash cuts on content, so an insertion changes one chunk. 8 MB instead of 2 GB, 250x less. Bound min and max size or low-entropy data explodes the chunk count.",15-design-answers
"File sync conflicts: the position","Keep both, with a conflicted copy. Last-writer-wins silently destroys work someone spent hours on, which users never forgive, and general file sync cannot merge arbitrary binary formats. Dropbox, Drive and OneDrive all reached this independently.",15-design-answers
"The hard part of file sync","Metadata, not bytes: ~875 billion chunk records and 5 trillion file records. And the production trap is refcounting, which races with in-flight uploads. Mark-and-sweep with a grace period, the same reasoning as gc_grace_seconds.",15-design-answers
"Erasure coding vs replication at exabyte scale","Reed-Solomon (10,4) tolerates 4 losses at 1.4x overhead against 3x for triple replication, at equal or better durability. At 3.5 EB logical that is over 5 EB of difference. The cost is read amplification and expensive repair, hence a hot-chunk cache.",15-design-answers
"Derive BM25 in three failures","Raw term frequency lets common words dominate, so weight by IDF. It grows linearly, implying 100 mentions is 10x more relevant than 10, so saturate. Long documents accumulate matches by accident, so normalise by length relative to average.",07-search-ranking
"What k1 and b do","k1 is the saturation rate: 0 is binary presence, 1.2 (default) saturates fast, 3 stays near-linear. b is length normalisation strength: 0 is none, 1 is full, 0.75 is the default. Both are empirical TREC defaults that generalise well.",07-search-ranking
"BM25 saturation, with a number","At k1=1.2 the first occurrence is worth 1.20 of a maximum 2.2, more than half the achievable score, and going from 10 occurrences to 100 buys about 0.13. The first mention says the doc is about the topic; the rest is confirmation.",07-search-ranking
"Can you compare BM25 scores across queries?","No. The scale depends on the query's IDF values and the corpus, so 14 on one query and 6 on another says nothing. Which is why hybrid fusion uses reciprocal rank rather than a weighted sum of incomparable scores.",07-search-ranking
"Multi-field BM25: the trap","Summing per-field scores gives a separate first-occurrence saturation bonus per field, over-rewarding documents that mention the term everywhere shallowly. BM25F combines frequencies before saturating; Elasticsearch's cross_fields approximates it.",07-search-ranking
"Refresh vs flush vs merge","Refresh is visibility (new searcher over the buffer, not necessarily durable). Flush is durability (write, fsync, truncate translog). Merge is efficiency. A document can be searchable and not durable; the translog covers that gap.",07-search-ranking
"Why Lucene segments are immutable","Modifying an inverted index in place means rewriting a postings list of ten million entries. Immutability makes indexing an append, constant time regardless of index size. The cost: searches touch every segment, and updates leave tombstones.",07-search-ranking
"How to speed up a 100M-document bulk load","Disable refresh_interval and set replicas to 0, then restore both. A 1-second refresh over 33 minutes creates ~2,000 small segments, each merged repeatedly up the tiers with 5-10x write amplification, and replicas double all of it. Typically 2-3x.",07-search-ranking
"When force-merge is actively harmful","On an index still receiving writes. It produces a segment above max_merged_segment (5 GB), which the merge policy then never touches again, so its deleted documents can never be reclaimed. Permanent degradation; only fix is a reindex.",07-search-ranking
"Why segment count affects search latency","A query runs against every segment with a fixed per-segment cost per term: a term dictionary lookup and skip-list setup. An 8-term query against 50 segments does 400 lookups; against 5 segments, 40. Check segment count before tuning the query.",07-search-ranking
"HNSW in one sentence","A skip list in metric space: layered graph, sparse long hops on top, dense local links at layer 0, with layer membership decaying exponentially. Search enters at the top, greedily walks toward the query, drops a layer, repeats.",07-search-ranking
"The three HNSW parameters, by when you pay","M is edges per node: build time, permanent, costs memory. efConstruction is graph quality: a one-time build cost with permanent benefit and ZERO query cost. efSearch is per query, trading recall against latency. Spend on efConstruction.",07-search-ranking
"HNSW memory formula","Roughly d x bytes_per_component + 8 x M. At 50M vectors, d=768, M=32: ~168 GB fp32, ~53 GB int8. Quantisation saves 2,300 bytes/vector where halving M saves 141, so quantisation is the lever by an order of magnitude.",07-search-ranking
"Why HNSW has no real delete","Removing a node breaks the graph and repairing it cascades, so implementations soft-delete: the node stays and is traversed as a routing node but excluded from results. Memory is never reclaimed and effective recall falls as the deleted fraction grows.",07-search-ranking
"When NOT to use HNSW","Under ~100k vectors, brute force with SIMD is single-digit ms, exact, with no build, parameters or delete problem. When memory binds, IVF-PQ compresses to under 100 bytes/vector. When it will not fit RAM, DiskANN.",07-search-ranking
"NDCG, derived","Cumulative gain (sum of grades) is order-insensitive, so add a log2(rank+1) discount for decaying attention, giving DCG. Normalise by the ideal ordering so queries are comparable. The 2^rel - 1 numerator makes 'perfect vs good' a bigger gap than 'good vs marginal'.",07-search-ranking
"Why report recall@k separately from NDCG","They can move in opposite directions. Recall is whether retrieval found it; NDCG is whether ranking placed it well. Retrieval recall is a hard ceiling: no reranker can surface what was never retrieved. Only measuring NDCG means you cannot tell which stage to fix.",07-search-ranking
"When MRR is wrong","Exploratory search. [relevant, irrelevant x3] and [relevant x4] have identical MRR and very different NDCG, because MRR ignores everything after the first hit. MRR is for known-item search where there is one right answer.",07-search-ranking
"The silent cause of offline-online divergence","Evaluation set bias. If the judged pool came from the old ranker's results, the new ranker's genuinely new documents are unjudged and scored as irrelevant by default, so it measures worse precisely because it found something new. Fix: TREC-style pooling across both systems.",07-search-ranking
"How do you know your offline metric is worth anything?","Measure the correlation between offline and online deltas across a set of past changes. If it is weak, the offline metric is not a decision tool and gating on it produces false confidence. Most teams have never done this.",07-search-ranking
"Why the ranking funnel exists","Arithmetic. A cross-encoder at ~5 ms per query-document pair over 10M documents is 14 hours per query; over 50 candidates it is 35 ms batched. Six orders of magnitude. Every stage shrinks the set enough for the next stage's model to be affordable.",07-search-ranking
"The funnel invariant","Each stage can only shrink the set, so recall lost at retrieval is unrecoverable downstream. Which is why retrieval recall is measured separately: high recall with low NDCG is a ranking problem, low recall is a retrieval problem.",07-search-ranking
"Bi-encoder vs cross-encoder","Bi-encoder encodes query and item separately and compares by dot product, so the item side precomputes and an ANN index works, at the cost of no query-item interaction. Cross-encoder encodes them together with full attention: much better, nothing precomputable.",07-search-ranking
"The biggest mistake in multi-stage training","Training each stage on random negatives. In production the reranker sees 50 plausible candidates from the previous stage; if it learned relevant-versus-absurd it never learned the distinction it needs. Mine hard negatives from the previous stage's own output.",07-search-ranking
"How many candidates should reach the heavy ranker?","Measured: the curve bends around 50-100. In one case 20 gave NDCG 0.712 at 18 ms, 50 gave 0.741 at 35 ms, 100 gave 0.749 at 68, and 200 gave 0.752 at 134. Doubling to 200 bought 0.003 for 66 ms.",07-search-ranking
"Popularity bias vs the feedback loop","Popularity bias is static: popular items have more interactions so a model trained on counts predicts popularity. The feedback loop is dynamic: the model's own decisions generate its next training set, so bias compounds over cycles.",07-search-ranking
"Why the feedback loop is invisible","Engagement metrics stay flat. A system can go from 340k items appearing in any top-20 to 61k over a year with CTR, session length and conversion all steady, because the head is genuinely engaging. The first symptom is usually a supply-side complaint.",07-search-ranking
"Thompson sampling over epsilon-greedy for exploration","Sample efficiency at the same slot cost. Epsilon-greedy spends budget uniformly including on items with 50,000 impressions where nothing is left to learn. Thompson draws from each item's posterior, so wide-posterior items get explored.",07-search-ranking
"Why IPS alone cannot fix the feedback loop","An item never shown has propensity zero and infinite weight: there is no observation to reweight. IPS corrects position bias within what was shown. Exploration generates the propensity variation IPS needs, so they are complements.",07-search-ranking
"Is concentrated exposure automatically bias?","No. If 5 percent of items genuinely are what most users want, uniform exposure is worse. The test is whether impression Gini EXCEEDS a relevance-based Gini, and establishing that needs unbiased exposure data, i.e. exploration first.",07-search-ranking
"Multilingual: the two orthogonal decisions","Lexical topology (per-language indexes, for analysis and IDF) and vector topology (one shared multilingual embedding space, for cross-lingual matching). Conflating them produces a design where neither is right.",07-search-ranking
"The subtle failure when merging results across indexes","BM25 IDF is computed per index, so the same term is worth ~28 percent more in a small French index than a large English one, purely because the index is smaller. Merging by score favours the smaller language. Fuse by rank with RRF instead.",07-search-ranking
"Language detection on short queries","Unreliable. 'Paris hotel' is ambiguous across languages, product names are language-neutral, and bilingual users code-switch. The user's declared locale is the stronger signal; detection supplements it rather than overriding it.",07-search-ranking
"How many vectors for a product in nine locales?","One. Nine translations are nine near-duplicate vectors competing for the same result slots, hurting diversity and inflating the index ninefold for no recall gain. Cross-lingual matching comes from the shared embedding space.",07-search-ranking
"Two-tower: why the lack of interaction is the point","No query-item interaction inside the model is exactly what lets the item side be encoded offline and put in an ANN index. A model with interaction cannot precompute anything. The cost is that it retrieves rather than ranks.",07-search-ranking
"The logQ correction, and why it matters","In-batch negatives sample items in proportion to frequency, so popular items are over-penalised and the model learns an anti-popularity bias. Subtracting log P(sampled) from each logit cancels it. Omitting it makes retrieval worst on head queries.",07-search-ranking
"The trap in hard negative mining","The model's top unlabelled results are disproportionately unlabelled POSITIVES, because labels are sparse. Using them as negatives teaches the model that correct answers are wrong. Skip the top ~10 and sample below.",07-search-ranking
"Three ways feature-store parity fails","Point-in-time leakage (training uses today's value to predict a 3-month-old click). Implementation skew (SQL in training, Python in serving, different null and timezone semantics). Missingness mismatch (training is backfilled, production has timeouts).",07-search-ranking
"The fastest signal for feature skew","Null-rate divergence. A broken join or a timed-out lookup shows up in the null rate before it shows up in the distribution of the values that did arrive. Alert on PSI above 0.2 too, but check nulls first.",07-search-ranking
"Why you cannot roll out a two-tower model gradually","A new model puts every item vector in a different space, so old and new vectors are not comparable and mixing them scores meaninglessly with no error anywhere. Full re-embed, atomic alias swap, and both towers deployed together.",07-search-ranking
"The three latency ratios worth memorising","Memory is ~200x slower than L1. Random NVMe is ~200x slower than memory. A same-datacenter round trip (~500 us) is ~5,000 memory accesses. Ratios survive hardware changes; absolute values do not.",02-distributed-systems
"What follows from a 500 us datacenter round trip","Splitting a service adds that permanently to the happy path, costing what 5,000 memory accesses would. And N+1 patterns are fatal at any N: 50 sequential calls is 25 ms of pure waiting with the CPU idle.",02-distributed-systems
"Sequential vs random on NVMe","1 MB sequential is ~50 us; 1 MB of 4 KB random reads is ~5,000 us. About 100x per byte. That is why every high-throughput storage system is log-structured: LSM trees, WALs and Kafka convert random writes into sequential appends.",02-distributed-systems
"Is compression worth it before a network call?","Almost always. Compressing 1 KB is ~2 us and sending it over 10 Gbps is ~0.5 us, but the round trip is 500 us, three orders of magnitude larger. Across regions at 150 ms it is not close.",02-distributed-systems
"Why cross-region latency cannot be optimised","CA to Netherlands is ~150 ms round trip; light in fibre over 8,900 km gives a floor of 89 ms. Within a factor of 1.7 of physics. It is a placement decision: the data is near the user or the request waits.",02-distributed-systems
"The counter-intuitive caching arithmetic","The benefit is dominated by the MISS rate. With a 40 ms query and a 0.5 ms cache, 50 to 90 percent hit rate is 4.5x better, and 90 to 99 percent is another 5x. The last few percent are worth more than the first fifty.",02-distributed-systems
"Linearizability vs serializability","Linearizability is single-object and real-time: once a write completes, later reads see it. Serializability is multi-object transactions equivalent to SOME serial order, saying nothing about which. Strict serializability is both.",02-distributed-systems
"What CAP's 'C' actually means","Linearizability specifically, not serializability and not consistency generally. The more useful result is that causal consistency is the strongest model achievable in an always-available system.",02-distributed-systems
"The four session guarantees","Read your writes, monotonic reads, monotonic writes, writes follow reads. These are what users actually perceive: nobody notices a linearizability violation between strangers, everybody notices their profile showing the old name.",02-distributed-systems
"How to implement read-your-writes","Version tokens, not sticky routing. The client carries the version it last saw; the replica satisfies it, waits briefly, or falls back to the leader. Sticky routing breaks on replica failure or a rehash, and the degradation is a wrong read rather than a slow one.",02-distributed-systems
"Choose a consistency model for the system?","No: per operation. Username registration needs consensus (uniqueness is a real invariant), payments need strict serializability, posting needs session guarantees, comments need causal, like counts can be a CRDT. Two of seven pay for consensus.",02-distributed-systems
"Where does snapshot isolation sit on the ladder?","It does not. It is a transaction isolation level weaker than serializable that permits write skew. PostgreSQL REPEATABLE READ is snapshot isolation and permits it; SERIALIZABLE uses SSI and prevents it.",02-distributed-systems
"Why not hash modulo N?","Going from 4 nodes to 5, hash%4 and hash%5 agree for 1 key in 5, so ~80 percent move. For a cache that is a near-total miss and an origin stampede. Consistent hashing bounds it to K/N.",02-distributed-systems
"The real reason for virtual nodes","Failure redistribution, more than balance. Without them a failed node's entire keyspace goes to its one clockwise neighbour, which then serves double load and often fails too. With 160 scattered positions the keys spread across all survivors.",02-distributed-systems
"Rendezvous hashing, and when to prefer it","Hash (key, node) for every node and take the max. Provably minimal disruption, good balance with no tuning parameter, and top-k replica selection for free. O(N) per lookup, which is irrelevant below a few hundred nodes.",02-distributed-systems
"Jump hash: the constraint","Buckets must be a numbered range you only grow or shrink at the END. You cannot remove bucket 3 from ten. Unusable for arbitrary node failure; ideal for a fixed shard count.",02-distributed-systems
"Does consistent hashing solve hot keys?","No. It distributes KEYS, not load. One celebrity key taking 40 percent of traffic lives on one node by construction, and no vnode count changes that. Fixes: key splitting, hot-key replication, a client-side cache, or power-of-two-choices.",02-distributed-systems
"Bounding disruption is not surviving it","Growing a cache fleet 20 to 30 nodes moves a third of keys, taking origin load from 12k to 74k requests/sec against 25k capacity. The answer is adding two nodes at a time and pre-warming them, not a better hash function.",02-distributed-systems
"Back-of-envelope: the three disciplines","State assumptions explicitly so they can be corrected. Work in round numbers (100,000 seconds per day). And DERIVE the architecture from the result: if the numbers and the design are unrelated, the arithmetic was decoration.",02-distributed-systems
"The sizing question to ask before designing distributed","Does it fit on one node? 82 GB of index means nine replicas each holding everything, so no scatter-gather and no tail-at-scale problem. And notice int8 quantisation is what kept it under the threshold: the optimisation removed the need to shard.",02-distributed-systems
"Which assumption is load-bearing?","Name it. In a search sizing, tripling users changes nothing structural; tripling the catalogue takes it past one node and forces sharding. Saying which number decides the architecture is worth more than the numbers themselves.",02-distributed-systems
"Raft in three subproblems","Leader election (randomised timeout, term increment, majority vote), log replication (leader pushes entries with a prevLogIndex/prevLogTerm consistency check), and safety (five properties, of which Leader Completeness is load-bearing).",02-distributed-systems
"Why the up-to-date vote check is a safety property","A committed entry is on a majority; any two majorities intersect; so at least one voter holds it and refuses to vote for a candidate whose log is behind. That intersection argument is the whole safety proof.",02-distributed-systems
"Raft's commit restriction (Figure 8)","A leader may NOT commit an entry from a previous term just because it is now on a majority: it can still be overwritten by a later leader. Leaders commit old entries indirectly, by committing one from their own term, hence the no-op on election.",02-distributed-systems
"Pre-vote, and why production needs it","A partitioned node keeps incrementing its term and rejoins at term 847 while the cluster is at 12, forcing the healthy leader to step down for nothing. Pre-vote asks 'would you vote for me' without changing state first.",02-distributed-systems
"Raft cluster sizing","Five nodes, odd, within one region. Even sizes are strictly worse: four tolerates the same single failure as three while waiting for an extra ack. Cross-region puts a 60-200 ms floor on every write.",02-distributed-systems
"The real Raft bottleneck","Disk, not network. Every committed entry needs an fsync on the leader and each acking follower, so a 10 ms fsync caps throughput regardless of network speed, and delays heartbeats behind log writes, causing spurious elections.",02-distributed-systems
"Lamport vs vector clocks","Lamport gives a total order consistent with causality and CANNOT detect concurrency (a lower timestamp does not mean happened-before). Vector clocks detect concurrency exactly and give only a partial order. Not interchangeable.",02-distributed-systems
"The two ways wall clocks fail","Skew (machines disagree, so the later-timestamped write may have happened first) and non-monotonicity (NTP corrects by stepping, which can move a clock backwards). With LWW, the worst-configured node wins every conflict, silently.",02-distributed-systems
"What TrueTime actually buys","An interval rather than an instant, so Spanner commits at a timestamp then WAITS OUT the uncertainty before releasing locks, making it definitely past everywhere. Costs ~2 epsilon per commit: tighter clocks are literally faster transactions.",02-distributed-systems
"Hybrid logical clocks, and their limit","Physical time in the high bits, logical counter in the low bits, advanced when the physical clock does not move. Monotonic, causal, human-readable, no hardware. What it cannot do is bound the error, so it cannot support commit-wait.",02-distributed-systems
"Does a better clock fix last-writer-wins?","No. HLC removes the pathologies (backwards steps reordering writes, effects preceding causes) and does not stop a concurrent write being discarded, because that is what LWW means. Use vector clocks or a CRDT instead.",02-distributed-systems
"Document vs term partitioning","Term partitioning routes a query to only the shards holding its terms, which sounds ideal, and its write path is a scatter (a doc with 200 terms touches many shards) with irreducible hot shards on common terms. Everyone uses document partitioning.",07-search-ranking
"How to choose a shard count","By target shard size, 20-50 GB, and let the count fall out. A fixed count is wrong at every scale except the one it was chosen for. And shard count is a latency decision: 50 shards at 1 percent slow each means ~40 percent of queries hit a slow shard.",07-search-ranking
"The highest-value search-latency optimisation","Reducing effective fan-out, not per-shard tuning. Partition-aware routing plus tiering (query the best 5 percent first, fall through only when insufficient) took one case from 50 shards to 2, so slow-shard probability went 39.5 percent to 2.",07-search-ranking
"The cold-start cascade","A restarted shard has an empty page cache, so queries go 15 ms -> 400 ms, and in a broadcast that makes EVERY query slow. Timeouts fire, retries multiply load, healthy shards saturate, and a rolling restart becomes a fleet outage.",07-search-ranking
"The one line that prevents the cold-start outage","The health check must fail while the shard is cold, so the load balancer does not route to it. A process that has started is not a process that can serve.",07-search-ranking
"LambdaRank's insight","You cannot differentiate NDCG, so instead of defining a loss and deriving a gradient, define the gradient directly and scale it by the metric change from swapping that pair. Swapping positions 1 and 2 matters far more than 49 and 50.",07-search-ranking
"Why GBDT still wins for feature-based ranking","Ranking features are heterogeneous tabular data. Trees are scale-invariant, handle missing values natively, capture interactions without crosses, train in minutes, and are interpretable. Neural wins on RAW TEXT, which is the cross-encoder stage.",07-search-ranking
"Two feature-engineering rules for LTR","Log-scale anything heavy-tailed, or the model spends capacity on the top 0.1 percent of items. And never feed a raw identifier as a numeric feature: seller_id = 88412 implies an ordering that does not exist and the model just memorises sellers.",07-search-ranking
"The most common LTR implementation error","Omitting the `group` parameter, which tells the ranker which rows are the same query. Without it the model compares documents across queries, which is meaningless. It trains with no error and silently produces a much worse model.",07-search-ranking
"Before training any ranker","Serving-time feature logging plus a judged evaluation set. A team that trains a ranker before it can measure whether the ranker helped has built something it cannot improve, and teams reliably do these in the opposite order.",07-search-ranking
"SLI vs SLO vs SLA","SLI is the measurement, SLO the internal target, SLA the customer contract with a consequence. The SLA target must be LOOSER than the SLO, so you find out and act before a customer is owed money. Equal targets give up the warning margin entirely.",12-sre-observability
"What an SLO actually targets","How bad the service is allowed to get, not how good it should be. That inversion is what makes the error budget work, and it is why an unspent budget means over-investment paid for out of feature work.",12-sre-observability
"How to explain the cost of a nine","In minutes. 99.99 percent is 4m19s per 30 days, which means a human cannot be in the recovery path at all: a page, a wake-up and a login exceeds the whole budget. That lands better than any argument about diminishing returns.",12-sre-observability
"The three SLI shapes","Request-based (good/total requests; the default, under-weights a 3am outage). Windows-based (good minutes; weights every minute equally). User-based (users with a good experience; most honest, hardest, catches one user seeing 100 percent failure).",12-sre-observability
"Why you cannot average percentiles","A percentile is an order statistic over a distribution; you cannot recover the union's from the components'. 1,000 requests at 10 ms and 10 at 5,000 gives per-instance p99s averaging 2,505 while the true p99 is 10. Off by 250x.",12-sre-observability
"The correct percentile aggregation","Sum the histogram BUCKETS across instances (counts are additive), then compute the quantile: histogram_quantile(0.99, sum by (le) (rate(..._bucket[5m]))). The `sum by (le)` is the entire correction.",12-sre-observability
"How accurate is histogram_quantile?","Only as accurate as your bucket boundaries near the target percentile, because it interpolates linearly within the containing bucket and assumes uniformity inside it. Prometheus defaults suit a ~100 ms service and are useless for a 3 ms one.",12-sre-observability
"Two alerts, not one, for latency","The correctly aggregated fleet percentile deliberately HIDES a single bad instance, which is right for an SLO and useless for triage. 'Is one host slow or all of them' is the first question in a latency incident, so alert on outliers separately.",12-sre-observability
"Coordinated omission","If a load generator waits for a response before sending the next request, during a stall it sends nothing, so the worst latencies are never recorded and the reported p99 is optimistic by orders of magnitude. Most load tools have it by default.",12-sre-observability
"Composite availability: the two rules","Serial (all required) multiplies availabilities. Parallel (any suffices) multiplies UNavailabilities. And for small numbers, total unavailability is approximately the SUM, accurate to three decimals and doable in your head.",12-sre-observability
"What the dependency ceiling tells you","Whether the SLO was ever achievable. Five deps at 99.95/99.99/99.9/99.5/99.9 sum to 0.0076 unavailability, so the ceiling is 99.24 percent. Missing a 99.9 SLO against that is a planning error, not an execution one.",12-sre-observability
"Where availability work actually pays","Removing a dependency from the critical path, not making it more reliable. Because unavailability ADDS, the worst dependency dominates: making a 99.5 percent service degradable bought 0.005 in three days; taking the database from four nines to five buys 0.00009.",12-sre-observability
"Classify every dependency","Required (the request is meaningless without it), degradable (works less well), or asynchronous (not needed at all). Most dependencies teams treat as required are degradable, and reclassifying is where the availability is.",12-sre-observability
"The independence trap in redundancy","Two providers at 99.9 percent is 99.9999 only if failures are independent. At a 10 percent correlated fraction it is 99.99, two orders of magnitude worse. So reduce correlation rather than adding replicas.",12-sre-observability
"The correlation people miss","A shared deployment pipeline. It looks like redundancy on the architecture diagram and takes both regions out on one bad rollout. Also shared control plane, IAM, DNS, certificate authority and correlated demand.",12-sre-observability
"Composite SLO for a multi-service journey","Measure the journey directly rather than composing component SLOs. Composition is wrong because failures correlate, not every request touches every service, and a component SLO measures its own traffic (a service skipped by 40 percent of requests contributes 0.4x its unavailability).",12-sre-observability
"Error budget vs error budget policy","The budget is arithmetic nobody disagrees with. The policy is what happens when it is exhausted, which is a commitment. Most SLOs in production have the first and not the second, which makes them reports.",12-sre-observability
"What makes an error budget policy work","Being signed BEFORE it is needed. A policy negotiated during an incident, or three weeks before a launch, is a negotiation from the weaker position and produces whatever the more powerful party wants.",12-sre-observability
"Scope the freeze narrowly","Feature deploys only. Reliability fixes, security patches, rollbacks and flag-gated work continue. A freeze that stops all deploys is obviously wrong, so it gets overridden and the policy loses authority.",12-sre-observability
"The clause that gets product to sign","The symmetric one: if the budget stays above 50 percent for two windows, take more deployment risk or tighten the SLO. A policy that only ever constrains product gets resisted by product, reasonably.",12-sre-observability
"How often should the override be used?","Twice a year is a working policy. Monthly means the SLO is wrong and should be renegotiated rather than routinely overridden. And engineering should NOT hold a veto: the policy makes the decision deliberate and accountable, not impossible.",12-sre-observability
"The detail that makes canary analysis valid","The control must be a FRESHLY DEPLOYED instance of the current version, not the running production fleet. Production has warm caches, JIT and pools; the canary has none, so it looks worse for reasons that are not the code.",13-deployment-delivery
"How canary analysis decays","Invalid control produces false positives, false positives get 'fixed' by loosening thresholds, and then a real regression passes. In one case the latency threshold had been raised to 15 percent and the actual regression was 12.",13-deployment-delivery
"Why not compare means in a canary?","Latency is heavily right-skewed so a mean is dominated by the tail, and p99 values do not aggregate across instances. Use a non-parametric test (Mann-Whitney U), which assumes nothing about normality.",13-deployment-delivery
"The canary gate conjunction","Significance AND effect size. With enough samples a 0.3 percent regression is significant and irrelevant; effect size alone fires on noise. p < 0.05 and |effect| > 5 percent, with the effect threshold tuned per metric.",13-deployment-delivery
"Keep business metrics out of the canary gate","At 5 percent over 30 minutes they are noise. Gating on them causes constant false rollbacks, and the team's response is always to lower the overall threshold, which degrades everything. They belong in the A/B test.",13-deployment-delivery
"NODATA is a failure, not a pass","A change that breaks the metrics exporter produces no canary metrics, and a naive scorer sees no failures and promotes a version it could not measure.",13-deployment-delivery
"MDE scales as 1/sqrt(n)","Doubling bake time improves the minimum detectable effect by about 40 percent, and detecting a regression half the size takes four times as long. You cannot fix a bad MDE by adding a few minutes.",13-deployment-delivery
"Why p99 is much harder to gate than p50","Only about 1 percent of samples are near the 99th percentile, so with 40,000 canary requests the p50 is informed by all of them and the p99 by about 400. That is 1 percent of the power, so ~10x the MDE.",13-deployment-delivery
"The ritual canary","One whose MDE exceeds any regression you care about. It runs, it passes, everyone feels safer, and it detects nothing. Signs: nobody can state the MDE, the bake time was copied, it has never failed except on hard errors.",13-deployment-delivery
"What to do with a statistically useless canary","Do not delete it: hard-failure detection needs one sample, not ten thousand. Relabel it a smoke test, publish that the service has no regression gate, and move regression detection to a longer A/B test or post-rollout monitoring.",13-deployment-delivery
"Derive bake time, do not choose it","Ask the smallest regression that would matter, measure baseline variance, solve for n, divide by the canary request rate. At 800 rps with CV 1.4, detecting 5 percent needs ~12k samples so 5 minutes; 2 percent needs 77k so 32 minutes.",13-deployment-delivery
"Shadow traffic vs canary","A canary serves real users and its failures are visible; a shadow serves nobody and its failures are invisible. Safe on the response path, dangerous on the side-effect path.",13-deployment-delivery
"The shadow side-effect rule","The seam must be a design property, not something improvised in shadow mode. If the payment gateway is called directly from scattered call sites, there is no safe way to shadow the write path.",13-deployment-delivery
"Recording no-ops, not silent ones","A shadow that records what it would have done reports 'this version would have charged $40 where production charged $40', which compares intended SIDE EFFECTS rather than only responses.",13-deployment-delivery
"The shadow header","Propagate X-Shadow-Request through every downstream hop. Without it, shadow traffic is indistinguishable from real traffic in every downstream service's dashboards and corrupts their error-rate and latency SLIs.",13-deployment-delivery
"What shadow traffic cannot tell you","Whether the change is good. Nobody sees the response, so no conversion or engagement signal, and any metric depending on user response is measured under the OLD model's click distribution.",13-deployment-delivery
"The randomisation unit rule","Randomise at the level at which the experience is consistent and the effect operates. Getting it wrong INVALIDATES the result rather than adding noise, because observations are no longer independent.",13-deployment-delivery
"Interference in experiments","When treatment affects control outcomes. Treatment sellers win sales FROM control sellers, so the measured effect is roughly double the true one. Fix: randomise by market so competition happens within a variant, at a large power cost.",13-deployment-delivery
"The cost of peeking","Checking daily over two weeks takes the false positive rate from a nominal 5 percent to about 30, so one in three wins is noise. Fix: sequential testing with always-valid p-values, costing 10-25 percent more samples.",13-deployment-delivery
"Guardrails need inverted statistics","For the primary metric you control false positives (do not claim a fake win). For a guardrail you are detecting HARM, so control false negatives: looser alpha and a non-inferiority framing, 'is it worse by more than X'.",13-deployment-delivery
"Sample ratio mismatch","Expected 50/50, observed 50.4/49.6 over 400k users is p = 0.0003, which is not chance. Bot filtering differing by variant, a crash losing treatment users, or dropped instrumentation. It invalidates the experiment entirely: find the bug.",13-deployment-delivery
"A rate win is not a business result","Checkout conversion +12 percent with revenue per order -12 percent is revenue per user DOWN 1 percent. Conversion rate is the most commonly mis-used primary metric, because a rate improvement must be checked against value per unit.",13-deployment-delivery
"Cell vs shard","Sharding partitions data behind a SHARED application tier, so a bad deploy or poisoned cache hits every shard. A cell is a complete independent instance including its own database, which is what contains software and data failures.",17-dr-multiregion
"What cells actually protect against","Blast radius: bad deploys (rolled cell by cell), poison inputs that crash-loop a service, data corruption (one restore), noisy neighbours, resource exhaustion, and bad config pushes. None of those are infrastructure failures.",17-dr-multiregion
"AZ redundancy vs cellular isolation","Orthogonal. AZs protect against INFRASTRUCTURE failure; cells protect against SOFTWARE and DATA failure. Three AZs sharing one database cluster has the first and none of the second, and the headline outages are overwhelmingly the second class.",17-dr-multiregion
"The cell router problem","It is the single point of failure the architecture creates. Keep it a lookup not logic, keep the mapping static and cacheable, fail static rather than closed, and ideally resolve the cell at DNS or in the client so it is not in the data path at all.",17-dr-multiregion
"The floor on cell size","Your largest tenant. A cell has to hold them, so if one customer is 3 percent of traffic you cannot have 5 percent cells unless you isolate them. Dedicated cells for the top few customers is a common and sensible asymmetry.",17-dr-multiregion
"How a cell architecture erodes","Shared components that seem harmless. A 'shared read replica for reporting' is convenient and creates something whose overload crosses every cell. Any shared component needs its own availability argument.",17-dr-multiregion
"Story portfolio, not answers","The question space is 30-plus questions and nobody has 30 stories. Sixteen stories, each tagged with 3-5 themes, covers it, because the same events contain conflict, influence, failure and technical judgement. The skill is knowing which thread to pull.",16-leadership
"Rehearse numbers, not sentences","Fix 4-5 figures per story (scale, before, after, one detail only a participant would know, duration) and let the prose vary. Identical wording sounds recited; identical numbers with different wording sounds like a memory.",16-leadership
"The three story-portfolio gaps","Underperformance (most senior engineers have never had the conversation), changed-my-mind (needs a SPECIFIC piece of evidence, not 'I became more open to X'), and something you chose NOT to do.",16-leadership
"The three story lengths","30 seconds for the scar-tissue version inside a technical answer, 90 seconds for the standard behavioural answer, 3 minutes for 'tell me more'. Giving the 3-minute version when 90 seconds was asked reads as an inability to calibrate.",16-leadership
"Underperformance has six causes","Unclear expectations, missing skill, wrong role fit, motivation, personal circumstances, environment. THREE of the six are the manager's failure, and unclear expectations is the most common: the person is genuinely surprised anyone is unhappy.",16-leadership
"The underperformance conversation opener","Three dated specific examples, then the pattern they form, then 'is that consistent with how you think it's going?' And offer that if they are surprised, you have not been clear enough and that is on you.",16-leadership
"The no-surprises rule","If a formal performance process is the first time the person hears there is a problem, that is a management failure regardless of their performance. They should be able to predict the formal step before it starts.",16-leadership
"The two symmetric underperformance failures","Vague feedback, which leaves someone working hard on the wrong things for months, and waiting three months hoping it resolves before a sudden formal process. Both are cruel, and vagueness is the cruellest thing a manager does.",16-leadership
"The DORA four, and the central finding","Deployment frequency and lead time (throughput); change failure rate and recovery time (stability). The finding is that they are NOT in tension: high performers are better at all four, because the practices that make deploys frequent make failures rare.",16-leadership
"The DORA AI caveat, stated precisely","Recent reports show throughput up and stability DOWN with AI adoption, a different shape from the original finding. Correlational survey data, and the useful framing is amplification: strong testing gets faster, weak testing gets faster at shipping defects.",16-leadership
"The metric to add to DORA first","Interrupt rate, unplanned work as a share of capacity. It explains what all four leave unexplained (why a team that should have capacity does not), is invisible in delivery metrics, and above ~20 percent the team cannot plan at all.",16-leadership
"The rule leads break under pressure","Do not report delivery metrics upward as performance. A director asking 'how is the team doing' wants a number and these are the number available, and the moment they are used for evaluation they become a target and measure gaming instead.",16-leadership
"Measure recovery time in two parts","Detection and remediation. 'We recover in 20 minutes' often hides that 15 were before anyone noticed, and detection is fixed by alerting while remediation is fixed by pipeline work. Detection is frequently the larger and more actionable half.",16-leadership
"Is there an individual productivity metric?","No. Lines of code, PR count, story points and commits have all been tried, all gamed, and all degraded what they measured. SPACE's central argument is that no single metric captures it. Manager judgement supported by artifacts is the answer.",16-leadership
"Why the review queue doubles after AI tooling","The constraint was never typing speed, it was review, testing and deployment. Faster generation does not remove the bottleneck, it moves load onto it. The queue doubling is the system showing you where the constraint actually was.",16-leadership
"The four counters to a doubled review queue","Cap PR size first (review effectiveness collapses past ~400 lines and generated PRs are large, so it fixes depth and quality together). Raise test requirements on generated code. Require authors to explain it as their own. Label AI-assisted PRs.",16-leadership
"Why 'add reviewers' is the wrong answer","It is a staffing answer to a systems problem: it works briefly, consumes the capacity the tooling was meant to free, and does nothing about PR size or review quality. Human review does not scale with generation speed; only automation does.",16-leadership
"The line not to cross on review queue depth","Lowering the review bar to clear it. Tempting because the queue is visible and the defects are not yet, and it converts a throughput problem into a quality one that surfaces six weeks later as incidents.",16-leadership
"PM committed to a date in front of a customer. First move?","Say nothing in the meeting. Contradicting your PM in front of a customer costs more than any date and cannot be undone. If asked directly: 'I want to check the sequencing and come back with specifics this week.'",16-leadership
"The question nobody asks about a promised date","What does the customer actually need by then? Frequently narrower than what was promised, and it converts 'we cannot make that date' into 'here is what we can have on it'.",16-leadership
"Make the date process conversation a trade","Ask for a one-day sanity check on external commitments, and offer forecast ranges proactively in return. A request for approval over a PM's commitments is a power move and gets resisted; a trade is something both parties want.",16-leadership
"Why escalating first destroys the relationship","The PM finds out, and every subsequent commitment is made deliberately without you. If you escalate, escalate together, and escalate the DECISION ('which of these three options') rather than the person.",16-leadership
"Inheriting a low-morale team: first move","Two weeks of 1:1s and no changes, and say that out loud. What they are bracing for is another change imposed by someone who was not there, so announcing that you will not change anything yet does more than anything else in week one.",16-leadership
"The five causes of post-reorg low morale","Loss of autonomy (fixable immediately, highest leverage), loss of purpose, loss of people (grief, no intervention works), loss of trust (slowest), and workload. The wrong intervention makes the others worse.",16-leadership
"Weeks 3-6 after inheriting a team","Fix ONE thing completely and visibly: named by several people, entirely in your authority, done in under three weeks. Killing something beats adding something, because a new process is another change imposed by someone new.",16-leadership
"The angry person on a post-reorg team","Usually the best diagnostic instrument in the room and frequently the one who cared most. What changes them is being right about something and having it acted on, not being managed out or routed around.",16-leadership
"The boundary that makes a new lead credible","Naming what you cannot fix. 'Here is what I can change, here is what I will advocate for and might not win, here is what is settled.' People can work with that; they cannot work with promises to fix everything followed by quietly not.",16-leadership
"Ninety-day signals that it is working","Do people bring you problems unprompted? Have 1:1s shifted from complaints to plans? Has anyone said something critical in a group setting? Delivery is a lagging indicator and means little before month four.",16-leadership
"The multi-region write decision, in four shapes","Single-region write with global read; home-region per entity; global consensus; active-active with async replication. These are different systems, not points on a dial, and the read path is the easy half.",17-dr-multiregion
"Why home-region ownership is under-used","It gives NO conflicts by construction rather than conflicts resolved well, local latency for the ~85 percent of users in their home region, strong per-entity consistency without consensus, partial per-entity failover, and data residency for free.",17-dr-multiregion
"When active-active is not available","When an invariant breaks under concurrent writes: uniqueness, monotonicity, a balance, inventory. Two regions can both accept the username 'alice'. That is not a tuning question and no better conflict resolver fixes it.",17-dr-multiregion
"The multi-region data audit","Per entity: what breaks under concurrent writes, and what volume is it? In one case that produced a design where 2 percent of writes pay a cross-region cost, and they were exactly the ones where correctness was non-negotiable.",17-dr-multiregion
"Two-region consensus is broken","Losing either region loses quorum, so you pay full cross-region latency on every write and buy no availability. Three regions minimum, or do not use consensus.",17-dr-multiregion
"The cross-region latency asymmetry","A five-region quorum from us-east waits ~75 ms and from ap-southeast ~180 ms. The same system is 2.4x slower for some users, and that is a product property to surface deliberately rather than an implementation detail.",17-dr-multiregion
"BLUF, and what it is not","Conclusion first, then the ask, then the reasoning. It is not brevity: the reordered version is often the same length, and the brevity comes from the listener being able to stop early. The test: could they leave after two sentences and act correctly?",01-interview-mechanics
"Altitude is two dials, not one","Technical depth AND organisational scope. A CTO wants high depth and wide scope, so the answer is neither 'the connection pool was exhausted' nor 'we had an outage', it is 'this is the third incident from the same pattern'.",01-interview-mechanics
"The most under-used communication technique","Asking. 'Do you want the two-minute version or the detail?' costs three seconds and removes the guessing entirely. It feels like weakness and is the opposite: it shows you know the answer has several altitudes.",01-interview-mechanics
"Bounded uncertainty: all four elements","The answer, the confidence, the alternative, and when you will know. Engineers reliably give the first and third and omit the second and fourth, producing 'it might be this or that', which the listener correctly cannot act on.",01-interview-mechanics
"The highest-value phrase in a recommendation","'One thing that would change my mind.' It converts a preference into a judgement and invites the listener to supply information rather than argue.",01-interview-mechanics
"Why bluffing is the most expensive interview mistake","It is detected in one follow-up and it is retroactive: the question fails, prior answers get re-examined as possible bluffs, later ones are heard sceptically, and the note says 'confidently wrong'. One bluff costs more than three admissions.",01-interview-mechanics
"The four-part response to not knowing","Say you do not know, fast. Say what you know that is adjacent. Reason toward an answer and LABEL it as reasoning. Say specifically how you would find out. An unlabelled guess is a bluff regardless of intent.",01-interview-mechanics
"Not all unknowns are the same","A memorised constant deserves 'I'd look it up' and moving on, because reasoning at length about a default value is padding. An unfamiliar SYSTEM deserves the full four-part response. A legal question deserves 'I'd involve counsel'.",01-interview-mechanics
"Correcting yourself mid-answer","A positive signal, not a recovery: it demonstrates you monitor your own reasoning, which is what you want in someone whose design-review assertions others build on. Defending an answer you know is wrong is the negative.",01-interview-mechanics
"When to guess in an interview","When explicitly invited. 'What's your instinct?' is a test of whether you can commit to a position under uncertainty, and refusing reads as risk-aversion, which at staff level is its own negative. Guess, label the confidence, give the reason.",01-interview-mechanics
"Cross-encoder vs LLM reranker, by the numbers","A small cross-encoder is ~5 ms per pair, so 50 candidates is ~35 ms batched. An LLM reranker over the same 50 is 300 ms to 2 s and costs real money per query. That difference decides which surfaces each fits.",05-ai-llm
"Why a reranker beats a bi-encoder","A bi-encoder must place a document's vector before seeing any query, so it encodes what the document is ABOUT. A cross-encoder reads both together with full attention, so it can tell a document that restates the premise from one that answers it.",05-ai-llm
"How many candidates to rerank","The curve bends at 50-100. In one case: 20 gave NDCG 0.712 at 18 ms, 50 gave 0.741 at 35, 100 gave 0.749 at 68, 200 gave 0.752 at 134. Doubling to 200 bought 0.003 for 66 ms. Take 50, adaptive under load.",05-ai-llm
"Listwise, not pointwise, for LLM reranking","Pointwise asks for an absolute relevance score on a scale the model does not hold stably across independent calls. Listwise ordering is more reliable and is one call instead of twenty. Use a sliding window for lists over ~20.",05-ai-llm
"The LLM reranker correctness bug","It will occasionally return nine ids instead of ten, or one that was not in the input. Validate against the input set and fall back to the input order. A reranker that can invent a document is a correctness bug, not a quality one.",05-ai-llm
"Getting LLM quality at cross-encoder latency","Distillation: run the LLM reranker offline over a large query sample and train a small cross-encoder to match its scores. The student inherits much of the judgement at ~1/100 of the latency, with the teacher's cost paid offline.",05-ai-llm
"The under-used reranking lever","Do not rerank every query. An exact product code or navigational lookup already has BM25's top result correct, so a cheap intent classifier skipping 30-40 percent of traffic buys headroom for the queries that benefit.",05-ai-llm
"Three compaction strategies","Rolling window (preserves recency, loses the task). Hierarchical summarisation (lossy trace of everything, specifics gone). Structured state extraction (preserves what the schema NAMES, loses everything it does not).",06-context-agents
"Why a bigger context window does not remove compaction","Cost is linear in input tokens, prefill is linear, and 'lost in the middle' means information mid-context is used less reliably. An agent carrying 800k tokens of history can be WORSE at the task than one carrying an 11k summary.",06-context-agents
"The most common agent context failure","It has forgotten its own task. Pin the system prompt, the original task and a constraints list, never compacted. A few hundred tokens prevents a whole failure class, and it is the cheapest fix available.",06-context-agents
"Schema over summary, and the field to insist on","A summary preserves what the summariser found salient; a schema preserves what you named. The field worth arguing for is failed_approaches, because agent loops are the most common production failure and no step cap prevents them cheaply.",06-context-agents
"Summarise from originals, not from summaries","Errors and omissions compound at each level, so by the third level the actual working configuration has been replaced by 'the second approach worked'. Re-reading the originals is more expensive and keeps each level one lossy step from truth.",06-context-agents
"Make compaction lazy, not lossy","Store the full transcript externally by turn id and large tool results by reference, with a retrieval tool. Then a summary can say 'the working config is in turn 14' and the agent can fetch it.",06-context-agents
"Most agent failures are not model failures","They are tool design, context management or authorisation failures presenting as bad model behaviour. In a $4,000 overnight incident, every cause was tool or budget design, and a better model would have persisted longer and spent more.",06-context-agents
"Fixing an agent loop","Detect repeated identical (tool, args) pairs and INJECT A MESSAGE saying that approach is not working. Far more effective than raising the step cap. And check the tool: the cause is often 'no results' returned indistinguishably from results.",06-context-agents
"Errors as tool results, not exceptions","A framework that raises on a bad call ends the run; one that feeds the error back lets the model read it and correct, usually within one step. That converts a whole class of hard failures into self-correcting ones for free.",06-context-agents
"The retry that costs real money","A send_email call times out at the HTTP layer, the framework retries, the email goes twice. Tools declare side_effects, and only tools that are idempotent or accept a DERIVED idempotency key are ever auto-retried.",06-context-agents
"How sub-agents blow the budget","By receiving FRESH budgets. A 30-step agent spawning three sub-agents per step, each with 30 steps, is 2,700 steps and the parent's cap caught none of it. Sub-agents must inherit the parent's remaining budget.",06-context-agents
"Catching a silent wrong answer","Cite-or-abstain is the highest-leverage single move: require the answer to reference the tool results supporting it and treat an unsupported claim as a failure. That converts a silent failure into a visible one you can gate on.",06-context-agents
"The schema-migration outage mechanism","Not duration. Your ALTER queues behind a long-running transaction, and because lock requests are ORDERED, every subsequent query queues behind the ALTER. A 40-minute index build caused 22 minutes of total unavailability before it started.",03-storage
"The most important line in a migration script","SET lock_timeout = '2s'. Retried twenty times with backoff is much safer than one attempt at 60 seconds, because each attempt's queue is short-lived. Its absence is the difference between a failed migration and an outage.",03-storage
"What CREATE INDEX CONCURRENTLY costs","Two to three times the duration (two table scans plus two waits), it cannot run inside a transaction block, and on failure it leaves an INVALID index that the planner ignores while it still costs write maintenance until dropped.",03-storage
"Making SET NOT NULL safe on Postgres 12+","Add CHECK (col IS NOT NULL) as NOT VALID (brief lock, no scan), VALIDATE CONSTRAINT (scans under SHARE UPDATE EXCLUSIVE so reads and writes continue), then SET NOT NULL uses it and skips its own scan.",03-storage
"gh-ost vs pt-online-schema-change","pt-osc uses TRIGGERS on the original table, which run inside every write transaction and add latency. gh-ost reads the BINLOG instead, so no added write latency, plus interactive throttling and a postponed cut-over flag file.",03-storage
"Where schema migrations actually go wrong","The backfill, not the DDL. A single UPDATE over 180M rows holds a long transaction, generates enormous WAL, blocks autovacuum and lags replicas. Batch with FOR UPDATE SKIP LOCKED and a short sleep, and gate the next deploy on a completeness check.",03-storage
"2PC vs saga: different guarantees, not preferences","2PC gives atomicity and takes availability (locks held from prepare to commit, blocked if the coordinator dies). A saga gives availability and takes atomicity AND isolation.",04-streaming-apis
"Why 2PC blocks, and why 3PC does not fix it","A participant that voted yes has durably promised it can commit, so it cannot unilaterally abort. That is a proved property: no protocol is non-blocking under a single coordinator failure with asynchronous communication. 3PC assumes bounded delay, which networks do not give.",04-streaming-apis
"Compensation is not rollback","Rollback restores the previous state and nobody sees the intermediate. Compensation is a NEW transaction that semantically undoes it, and the intermediate was visible. A charge plus a refund is two lines on the customer's statement, which is a support call.",04-streaming-apis
"The most important saga design decision","Step ordering, so the irreversible action is LAST. Authorise early and capture late, so a shipping failure voids an authorisation the customer never sees rather than refunding a capture they do.",04-streaming-apis
"What sagas give up besides atomicity","Isolation. Another saga can read a state later compensated away: a dirty read across services. The countermeasures are in the 1987 paper (semantic locks, commutative updates, reordering) and are routinely dropped from modern retellings.",04-streaming-apis
"Before designing a saga, ask this","Is this a multi-step business process with compensations, or one database write plus one event publish? Usually the second, which is the transactional outbox: a table and a relay, against an orchestrator plus compensations plus intermediate-state handling.",04-streaming-apis
"Choreography or orchestration?","Orchestration beyond about three steps, on operational grounds: when a saga stalls at 3am, reading five services to reconstruct the event flow is much worse than querying the orchestrator's state by saga id.",04-streaming-apis
"The three zoom levels of an architecture deep dive","Context (2 min, no boxes: constraints and what good meant as a number, plus your role explicitly). Architecture (5 min: components, flow, and the 2-3 decisions with their rejected alternatives). Mechanism (one component, measured numbers, failure mode, regret).",01-interview-mechanics
"How to pick which system to deep-dive on","Not the most impressive. The test: pick any component and ask 'why is it that way, and what was the alternative'. If you cannot answer that for three components, choose a different system. Scope is established in two sentences at level one.",01-interview-mechanics
"What the deep dive is actually testing","Whether your experience is real and whether you made the decisions. It is the only round where they can ask 'why 20 connections and not 50' and find out in one question. Two follow-ups reliably expose a candidate who was adjacent to the work.",01-interview-mechanics
"Level 2 that lands vs level 2 that does not","A component list is a description. A decision with its rejected alternative, plus the cost you lived with ('that's why deploys are slower than you'd expect'), is engineering.",01-interview-mechanics
"Choosing a database: run the sequence","Access patterns as queries with QPS and latency; multi-key transactions (binary, eliminates a class); size and growth; consistency per operation; read-write shape; what you can operate. Each answer eliminates options, which a comparison table does not.",03-storage
"The size thresholds that matter","Under 100 GB anything works and distribution is pure cost. 100 GB to 1 TB single-node relational is comfortable, and this is where premature distribution happens. 1-10 TB gets uncomfortable (backup, vacuum, upgrades). Over 10 TB distribution is not optional.",03-storage
"The bar for adding a second database","An order of magnitude on a workload that matters, not a marginal gain on one query. It is not a schema decision, it is a permanent operational commitment: another backup procedure, upgrade path, monitoring integration and on-call body of knowledge.",03-storage
"What the multi-key transaction question really asks","Whether an INVARIANT spans the records, not whether they happen to be written together. Two writes each retryable independently need idempotency, not a transaction. Teams answer yes reflexively and are often wrong.",03-storage
"What gets misdiagnosed as a database choice","Contention between analytical and transactional work. Dashboard aggregations holding locks the transactional path needs presents as 'Postgres cannot handle our analytics' and is fixed by a read replica in a day.",03-storage
"The API choice axis","The consumer relationship, not performance. Who consumes it, and can you change the client quickly? A mobile app in the field means the API is effectively append-only for months, which constrains more than any throughput consideration.",04-streaming-apis
"When GraphQL earns its cost","Many client shapes AND you cannot ship a client quickly. Strong for mobile in the field; weaker for a web SPA that deploys daily, because you can just add an endpoint. It moves complexity to the server: DataLoader, cost analysis, persisted queries, cost-based rate limits.",04-streaming-apis
"Why not GraphQL for a public API","An unknown consumer can write a query joining six resources at depth nine, and you cannot see its cost until you parse it. You cannot rate-limit by endpoint because there is one endpoint, and HTTP caching does not apply because there is no URL to cache.",04-streaming-apis
"The gRPC load-balancing trap","HTTP/2 multiplexes over one long-lived connection and an L4 balancer balances CONNECTIONS, so every request from a client hits one backend, and autoscaling makes it worse. Fix with L7 or client-side balancing; connection recycling is the cheap partial fix.",04-streaming-apis
"What webhooks lack by default","Ordering, exactly-once delivery, backpressure and security. Each becomes something you build: sequence numbers, event ids with idempotent handling, backoff plus dead-lettering plus replay, and HMAC signing with a timestamp. Offer a polling endpoint as the alternative.",04-streaming-apis
"Five protocols is not incoherence","It is five consumer relationships: public (REST, unknown consumers), internal (gRPC, both ends controlled), mobile (GraphQL, cannot ship clients fast), notifications (SSE), integrations (webhooks plus polling). One protocol everywhere is the incoherent version.",04-streaming-apis
"Four conflict-resolution mechanisms","Discard (LWW), detect (version vectors, return siblings), merge (CRDTs, by construction), avoid (single writer per entity). Different guarantees, chosen per data type. Any single choice is badly wrong for some of your data.",17-dr-multiregion
"What HLC fixes and what it does not","It fixes clock skew systematically deciding conflicts, where a region 180 ms fast wins every time, invisibly. It does NOT stop one of two concurrent writes being destroyed, because that is what last-writer-wins means. Teams adopt HLC and think it is solved.",17-dr-multiregion
"The CRDT boundary, as a rule","A merge function can express 'combine these' and cannot express 'only if'. A PN-Counter tracks inventory perfectly and will happily go negative: overselling is its CORRECT behaviour. If the invariant matters, the data needs a single writer.",17-dr-multiregion
"Why version vectors are less used than they deserve","Sibling explosion (a hot key can reach hundreds, so you cap and lose something) and application burden (every read path must handle three versions, and the common shortcut is a helper picking one arbitrarily, which is LWW without the honesty).",17-dr-multiregion
"The operational recommendation nobody follows","Log the discarded write, with both values. It is the only way to turn 'is LWW acceptable for this data' from an argument into a measurement. In one case it showed 0.02 percent, concentrated in one user editing on two devices.",17-dr-multiregion
"How to verify a merge function","Property-based tests for commutativity, associativity and idempotence, because those three properties are exactly what guarantees convergence. Plus a game day: partition, write conflicting values, heal, assert convergence. A non-commutative merge fails nowhere else until production.",17-dr-multiregion
"What the CTO round assesses","Would I put this person in front of the board or a customer; do they think about the business or only the system; do I want to work with them. Not technical depth: four people already assessed that. It is a veto round, asymmetrically.",18-offer-and-questions
"The most common failure in the CTO round","Going too deep. They have read that you are technically strong, so demonstrating it again at the wrong altitude demonstrates you cannot calibrate, which is exactly what is being assessed.",18-offer-and-questions
"The two-minute self-introduction shape","Where you are now and what you own (20s), the through-line of what you have repeatedly solved (30s), one concrete example with a number (40s), why this role specifically (30s). The default failure is chronological and five minutes.",18-offer-and-questions
"Answering a technical question at CTO altitude","Outcome, then the judgement call, then how you got agreement, with exactly ONE technical detail as evidence rather than as content. That is what a CTO can evaluate and what they will hear about second-hand later.",18-offer-and-questions
"The best question to ask a CTO","'What's the thing you'd most want fixed in engineering that you haven't been able to fix yet?' Hard to deflect, genuinely useful to you, and it signals you expect a real organisation with real constraints rather than a brochure.",18-offer-and-questions
"The free signal in the CTO round","The consistency check: does their account of priorities match what the hiring manager and the engineers said? A gap there is the most reliable organisational signal in the whole process and it appears nowhere else.",18-offer-and-questions
"3,000 tokens of business rules: the first question","What fraction of a request actually needs them? Usually a small one. Retrieve the applicable rules rather than including all of them, which typically takes 3,000 tokens to 300-400. Then order the prompt so the stable part is a cacheable prefix.",06-context-agents
"The rules that should not be in the prompt at all","Deterministic ones. 'Orders over $5,000 need approval' belongs in code, where it is testable and cannot be talked out of by a model. What belongs in the prompt is the judgement, not the arithmetic.",06-context-agents
"Budgeting a 128k window for an agent","Pinned (system prompt, task, constraints; never compacted). Working set (last 3-5 turns verbatim). Structured state (extracted every 8-10 steps). Retrieved content. And ~20 percent HEADROOM, because the next tool call might return something large.",06-context-agents
"Why a canary cannot tell you a feature is winning","Two reasons. Sample size: at 5 percent for 30 minutes a business metric's variance swamps any lift. And the canary population is not randomised the way an experiment's is. A canary answers 'is it safe'; an A/B test answers 'is it better'.",13-deployment-delivery
"Canary MDE at 200 QPS","5 percent for 10 minutes is 6,000 requests per side, which detects an error-rate difference of ~0.9 percentage points. So it catches 0.5 -> 1.4 percent and CANNOT catch 0.5 -> 1.0, a doubling. For payments that is not an acceptable blind spot.",13-deployment-delivery
"Shadow-testing a service that sends email","The seam must exist in the design, not be improvised in shadow mode. Inject a RECORDING no-op sender so the shadow reports what it would have sent, which compares intended side effects rather than only responses. Shadow 10 percent, and propagate a shadow header.",13-deployment-delivery
"Repository pattern: the one-sentence test","Is there an invariant that would be violated if code could load a partial version of this thing? Order with line items and a cancellation rule: yes. A reporting endpoint producing a screen: no, that is a query object.",14-architecture-patterns
"Vendor at 99.5 percent, you sell 99.9 percent","If they are on the critical path you cannot sell 99.9. That is arithmetic, so start there. Best option: take them OFF the critical path with a fallback and a cache, which is days. Two vendors gives 99.9975 only under independence; at 10 percent correlation it is ~99.95.",12-sre-observability
"The three reverse-diligence red flags","Nobody can name a decision that was reversed. The on-call story has no number. And engineering and leadership describe priorities differently, which is the biggest of the three and costs nothing to observe by asking the same question in several rounds.",18-offer-and-questions
"Scar-tissue story: the shape and the limit","Three sentences: what we did, what went wrong with a number only a participant would know, and the specific thing we do now, then return to the technical point. Three or four across a whole interview, not one per answer, or it becomes a tic.",01-interview-mechanics
"The four resilience patterns defend against different things","Timeout bounds waiting. Retry handles transient failure and AMPLIFIES overload. Circuit breaker stops calling a failing dependency. Bulkhead isolates resources per dependency. A breaker does nothing until it trips; a bulkhead protects during the detection window.",14-architecture-patterns
"The cascade, in numbers","A dependency slows 20 ms to 6 s. By Little's Law, in-flight goes from 200 x 0.02 = 4 to 200 x 5 = 1,000 against a pool of 200. Pool exhausted in seconds, and now EVERY endpoint fails including ones that never called it.",14-architecture-patterns
"How to choose a timeout","p99.9 of the dependency's healthy latency plus margin, not a round number. Most HTTP clients default to NO timeout, which is not 'wait a long time', it is 'wait forever'. Plus deadline propagation, or a downstream service works on a request its caller abandoned.",14-architecture-patterns
"The retry rule most implementations lack","A fleet-level retry BUDGET, capping retries at ~10 percent of successful requests. A per-request cap of 3 bounds one client and says nothing about the aggregate: the whole fleet can still triple load on a service already failing.",14-architecture-patterns
"Two things circuit breakers get wrong","Tripping on CONSECUTIVE failures, which never fires at a 50 percent failure rate because successes reset the counter. And tripping only on errors, when the realistic failure is slow: a dependency answering 200s in 8 seconds passes every error-based breaker.",14-architecture-patterns
"Bounded context vs microservice","A bounded context is a MODEL boundary; a service is a DEPLOYMENT boundary. They often align and are different decisions. A modular monolith can hold several contexts with boundaries enforced in the build, and that is frequently the right shape.",14-architecture-patterns
"How to find bounded contexts","In the language, not the schema. Interview each team separately and listen for the same word meaning different things. A word that needs qualifying ('the SALES customer') is a boundary announcing itself. A two-day audit found four meanings of 'policy' in one 94-column table.",14-architecture-patterns
"Is the ubiquitous language company-wide?","No, and that is the common misreading. A single org-wide glossary is the God-object failure in documentation form. The language is ubiquitous WITHIN a context, and translation at boundaries is correct rather than a failure to standardise.",14-architecture-patterns
"What makes an ACL real rather than nominal","It translates SEMANTICS, not field names. Mapping CRED_LIM to creditLimit is renaming; mapping their -1 to your None because the sentinel means 'unlimited' is translation. And nothing outside it may use the foreign vocabulary.",14-architecture-patterns
"An aggregate is drawn around what?","Invariants, not the object graph. Order + line items is one aggregate because the total must equal the sum and it cannot ship with a backordered line. Order + Customer is not, even though the graph connects them, because no rule spans both.",14-architecture-patterns
"The aggregate rule people break first","Reference other aggregates by IDENTITY, not by object. order.customerId, not order.customer. Once order.customer.orders exists nothing constrains the boundary, and you get a Customer aggregate loading 400 orders to change an email.",14-architecture-patterns
"When an invariant forces a huge aggregate","Ask the business whether it must be transactional, because frequently it need not be and nobody has asked. A credit-limit rule forced a Customer-plus-all-Orders aggregate; the business was happy with a review hold, which is eventual, and the race was 1 in 40,000.",14-architecture-patterns
"Managed vs self-hosted: the question that resolves it","Does running this well make the product better in a way customers notice? For Postgres, Kafka or Kubernetes, no. That says where the operational capacity should go, which is stronger than any cost comparison.",20-cloud-architecture
"Why the managed cost comparison is usually wrong","It uses the sticker price. $4,200 managed Kafka vs $1,400 of EC2 looks like a $2,800 saving and omits 3 engineer-weeks of setup, ~15 percent of an engineer ongoing (~$2,500/mo), upgrades and 2am pages. The saving is roughly zero before the bus factor.",20-cloud-architecture
"What managed only MOVES rather than removes","Capacity planning (now instance-class selection), cost management (a bill that grows faster than a fleet), performance tuning (fewer knobs that still matter), and the upgrade decision (forced on their schedule).",20-cloud-architecture
"The managed-service check people skip","Read the service limits page before adopting and check each against 3x current scale. It takes an hour and hitting one at scale is a migration. The surprises are not capacity limits but no-superuser and no-custom-extensions: limits on what is possible at all.",20-cloud-architecture
"The cloud cost lever people miss","Cross-AZ traffic, billed in BOTH directions. In one case it was 14 percent of the bill, almost entirely two services exchanging 180 MB/s across an AZ boundary, and co-locating them took three days. Invisible because 'data transfer' is charged to the account, not a team.",20-cloud-architecture
"Hygiene reduces the bill; architecture changes the slope","Five of six cost items in one programme were hygiene, delivering 43 percent, which 40 percent traffic growth erases in ~18 months. Only making a nightly full reprocess incremental changed the growth rate. Report the distinction or you repeat the conversation next year.",20-cloud-architecture
"The cost-lever ordering error that costs most","Committing before right-sizing. A three-year commitment to over-provisioned instances locks in the waste for three years. Right-size, watch for a few weeks, then commit to the new floor at ~65 percent.",20-cloud-architecture
"The floor on right-sizing","The latency SLO, not utilisation. Queueing time scales as 1/(1-rho), so a service at 40 percent CPU may be correctly sized for burst. Sizing everything to 80 percent average is a latency incident scheduled for the next spike.",20-cloud-architecture
"The inverted index has three parts","Term dictionary (an FST, decides lookup cost), postings lists (sorted docids plus freqs and optionally positions, decides intersection cost), and doc values (columnar, for sort and facet). A slow query is slow in exactly one, and the fixes differ completely.",07-search-ranking
"How postings compress","Delta encoding, because docids are sorted so gaps are small, then a block scheme like PFOR-delta over 128 docs that decodes with SIMD. Nice property: the densest terms have the smallest gaps, so the most expensive lists compress best.",07-search-ranking
"Why 'rare AND common' is fast","Skip lists. The postings have a multi-level skip structure, so advance(target) jumps past blocks without decoding. The rare term drives iteration and the common one skips, making cost proportional to the rare list. Hence planners order conjunctions by document frequency.",07-search-ranking
"Block-max WAND, and why hit counts got expensive","It skips on SCORE: a doc that cannot beat the current kth-best is never scored, and a whole block whose max cannot reach the threshold is skipped. Exact total counts defeat exactly that optimisation, which is why track_total_hits defaults to 10,000.",07-search-ranking
"The biggest index-size lever","index_options and doc_values per field. Positions typically DOUBLE the index and are needed only for phrase queries; doc values are on by default and often never sorted or faceted. In one case those two were 22 of 40 GB, and neither was a tuning parameter, just defaults.",07-search-ranking
"Why analysis beats ranking work","A term that is never produced cannot be matched at any score, so an analyzer bug is invisible in every ranking metric while capping all of them. One marketplace had a 14 percent null-result rate in German because the mapping was copied from English.",07-search-ranking
"Stemming vs lemmatisation, and the answer","Stemming chops algorithmically and produces non-words; lemmatisation maps to dictionary forms using part-of-speech. Use LIGHT stemming, because aggressive merges 'universe' and 'university' into 'univers'. Better still: index both, with an unstemmed .exact field boosted.",07-search-ranking
"The token-filter ordering bug","Synonyms AFTER stemming. The synonym's own terms then never get stemmed and never match stemmed document terms. Synonyms go before the stemmer. Also german_normalization before asciifolding, or u-umlaut becomes 'u' when the correct transliteration is 'ue'.",07-search-ranking
"CJK tokenisation: the production answer","Both. A dictionary analyser (kuromoji, IK, nori) with a USER DICTIONARY as primary, because brand and product names are systematically out-of-vocabulary, plus an n-gram field as a recall fallback. Segmentation is genuinely ambiguous, not just hard.",07-search-ranking
"Vector index choice: what decides it","Memory, because it changes the machine class rather than a parameter. HNSW is roughly d bytes plus 8M per vector: 50M int8 768-dim is ~52 GB and fits one node; 200M is 210 GB and the family choice becomes a cost decision.",07-search-ranking
"What makes IVF-PQ competitive","Reranking with full vectors. Retrieve 500 candidates on compressed distances, fetch just those 500 full vectors, recompute exactly. Recall approaches exact while the index stays ~30x smaller. An IVF-PQ deployment without reranking leaves most of its quality behind.",07-search-ranking
"Why PQ is fast as well as small","A per-query lookup table: for each of the m sub-spaces, distances from the query's sub-vector to all 256 centroids. Scoring a candidate is then m lookups and adds with NO vector arithmetic at all.",07-search-ranking
"DiskANN is not 'HNSW on disk'","Its graph is built to minimise SSD READS per query, a different objective from minimising hops. A general graph traversal on disk is a random-read storm. It keeps compressed vectors in RAM for routing and reads only the few full vectors it needs.",07-search-ranking
"USL vs Amdahl","Amdahl has one term (contention) and predicts a PLATEAU. USL adds a quadratic coherence term and predicts a PEAK followed by decline. That retrograde region is what production actually hits, and under Amdahl it is impossible.",02-distributed-systems
"Contention vs coherence","Contention is a queue behind an exclusive resource (a lock, a single writer, a pool): linear in N. Coherence is a conversation between participants (cache invalidation, gossip, a shared counter): quadratic, because it is pairwise. Different fixes entirely.",02-distributed-systems
"Which USL coefficient to attack","Beta. Halving alpha moved peak concurrency from 98 to 99; halving beta moved it from 98 to 139. Coherence dominates the ceiling, and teams reliably attack contention instead because a lock shows in a profiler and crosstalk between instances does not.",02-distributed-systems
"The load-test mistake that hides beta","Stopping when throughput plateaus. That is exactly one measurement too early: without observing the DOWNTURN you cannot fit the coherence term, so the retrograde region stays invisible until production finds it.",02-distributed-systems
"A real high-beta source","A peer-list heartbeat: every pod writes a shared key every other pod reads, so 552 reads per interval at 24 pods against 56 at 8. Replacing it with service discovery took the useful ceiling from ~20 pods to ~64. Nobody thought of it as part of the request path.",02-distributed-systems
"Coordinated omission, in one sentence","A closed-loop load generator waits for a response before sending the next request, so during a 2-second stall it issues NOTHING. The slow requests were never made, so they never appear in the histogram. Fix: open-model generator (wrk2, k6 arrival-rate), or measure from the INTENDED start time.",02-distributed-systems
"The two benchmark modes you must not mix up","A microbenchmark measures one operation in isolation; a load test measures the whole system under concurrency. Optimising with the first and shipping without the second is how a clean 4x becomes a 2.7 percent production change.",02-distributed-systems
"Why fork the JVM per benchmark","Profile pollution. If benchmark A calls a shared helper with String and B with Integer, the call site becomes bimorphic and the JIT stops inlining, so whichever runs second reports a slower number for reasons unrelated to the code. Multiple forks also expose real run-to-run variance.",02-distributed-systems
"Do the Amdahl arithmetic before benchmarking","A genuine 4x speedup on something that is 0.08 percent of the request is a 1.0006x speedup overall. Profile FIRST to learn the share, then benchmark the thing the profile identified. Reversing that order is how quarters get spent on nothing.",02-distributed-systems
"Paxos safety in one sentence","Any two quorums intersect, so a proposer that gathers a quorum of promises must learn about any value that might already have been chosen, and the Phase 2a constraint REQUIRES it to propose that value instead of its own.",02-distributed-systems
"The Phase 2a constraint","If ANY acceptor in the promise quorum reports a previously accepted value, the proposer must propose the value with the highest accepted proposal number. Only if all report 'nothing accepted' is it free to propose its own. Forgetting this is the classic data-loss bug.",02-distributed-systems
"Single-decree vs Multi-Paxos","Single-decree agrees on ONE value and nobody runs it. Multi-Paxos runs Phase 1 once for a range of log slots (that is leader election), then skips to Phase 2 per entry: one round trip per write. Leader change must re-propose any partially-accepted tail entries.",02-distributed-systems
"Paxos and disk loss","An acceptor that loses its disk and rejoins as itself can accept a proposal it previously promised not to, breaking safety. It must rejoin as a NEW member or sit out until caught up. Paxos Made Live describes exactly this fix.",02-distributed-systems
"The three CRDT properties","Merge must be commutative (order does not matter), associative (grouping does not matter) and idempotent (redelivery does not matter). Those three make a join-semilattice, and repeated merging converges to the least upper bound. Convergence is proved, not engineered.",02-distributed-systems
"The one thing CRDTs cannot do","Enforce an invariant over the combined state. Two replicas each seeing a balance of 50 will each allow a withdrawal of 50; the merge is -50 and convergence was never violated. Bailis's invariant confluence formalises which invariants survive merge. Fix: escrow.",02-distributed-systems
"OR-Set vs 2P-Set vs LWW-Element-Set","OR-Set tags each add uniquely and a remove deletes only OBSERVED tags, so concurrent add beats remove: the one that behaves as expected. 2P-Set can never re-add. LWW-Element-Set is a valid CRDT that loses writes to clock skew.",02-distributed-systems
"Why Figma rejected CRDTs","They already had an authoritative server, so a server-ordered OT-like model gave them the same result with much simpler data structures and far less memory overhead. The fairest published critique, and the right question: am I actually operating without a coordination point?",02-distributed-systems
"Unjudged means gain zero","Standard NDCG scores unjudged documents as 0, so an evaluation pool built from ONE ranker's top-k systematically punishes any challenger that retrieves different documents. A semantic retriever measured -5.6 percent offline and +2.4 percent CTR online for exactly this reason. Fix: pool across every system.",07-search-ranking
"Annotator agreement is the ceiling","Use weighted (quadratic) Cohen's or Fleiss' kappa. Below 0.4 the guideline is broken, above 0.6 is good for an eval set. If your model's measured improvement is smaller than annotator disagreement, you have not measured an improvement.",07-search-ranking
"The judgment split to commit to","Human-graded FROZEN set for offline evaluation; debiased click data for training; LLM judgment for triage and pool expansion only. Never the reverse: clicks encode the current ranker's behaviour, which is exactly what you need the eval set to be independent of.",07-search-ranking
"Turning clicks into training labels","Skip-above pairwise preferences: clicked rank 5 while skipping 1-4 gives (doc5 > doc1..4). Both documents were examined under similar conditions so position bias largely cancels. Far more robust than absolute click rates. Then IPS-weight the residual.",07-search-ranking
"The cold-start loop, stated exactly","A model trained on engagement ranks a zero-signal item last; it gets no impressions, so it accumulates no signal, so it stays last. The item is not judged badly, it is not judged at all. Exploration is the only general cure, and it is also what keeps training data honest.",07-search-ranking
"Why Thompson sampling over UCB","Delayed feedback (UCB's deterministic argmax hammers one arm while rewards are in flight), randomisation (different users see different arms, and it gives you the propensities offline evaluation needs), and the Beta prior is exactly where a content-based CTR prediction goes.",07-search-ranking
"The cold-start prior, concretely","Do not start a new item at Beta(1,1). Predict its CTR from content features using a model trained on mature items, then set alpha = pred*strength, beta = (1-pred)*strength. Strength is a pseudo-count: 'this prediction is worth 50 impressions'. That knob IS the handover from content to behaviour.",07-search-ranking
"The bandit arithmetic to do first","Exploration impressions divided by new items per day. 12M impressions, 1 slot in 10, 40k new items = 30 impressions/item/day. That detects a disaster and cannot rank finely. If it came out at 3, build a content model and skip the bandit entirely.",07-search-ranking
"Exploration's organisational problem","It always costs the metric you are measured on (page CTR) and pays in a metric someone else owns (new-seller retention, new-listing GMV). Get the second metric onto your own dashboard BEFORE launching, or a correct read of your own numbers says revert.",07-search-ranking
"Why a model upgrade cannot be a rolling deploy","Two model versions are different coordinate systems: nothing makes the new model's dimension 37 mean what the old one's did. Cosine between them returns a number and that number is meaningless. So it is an ATOMIC whole-index swap, with the query encoder flipping in the same operation.",07-search-ranking
"The two freshness clocks","DOCUMENT freshness (seconds, solved by a flat brute-force delta tier beside the base index) and MODEL freshness (quarters, solved by blue-green with dual-write and shadow queries). Different mechanisms, different frequencies; conflating them does neither well.",07-search-ranking
"How to make rebuilds affordable","Store the raw vectors in Parquet keyed by (doc_id, model_version), and content-hash the encoder input. Then re-indexing needs no GPU, and a nightly refresh at 2 percent churn re-encodes 2 percent, not 100 percent: 25 GPU-hours a week became 1.2.",07-search-ranking
"Tombstones in HNSW are not free","Soft-deleted nodes stay in the graph as routing hops, so they cost traversal work AND recall, because paths run through nodes that yield nothing. Track the ratio, rebuild above 10-20 percent. Updates count too: each update is a delete plus an insert.",07-search-ranking
"acks=all is not what you think","It waits for the CURRENT ISR, and the ISR can shrink to one replica. So acks=all without min.insync.replicas=2 is acks=1 with extra latency. The two settings only work as a pair, and they live in different files owned by different teams.",04-streaming-apis
"min.insync.replicas is inert under acks=1","It is only consulted when acks=all. You can set min.insync.replicas=3 on a topic, feel safe, and have a producer with acks=1 writing to it with zero durability guarantee. Kafka will not warn you: the settings are validated independently.",04-streaming-apis
"What 'committed' means in Kafka","Every replica in the CURRENT ISR has it. Not a majority, not all replicas. Consumers read up to the high watermark = min(LEO) over the ISR, so committed and consumer-visible are the same boundary.",04-streaming-apis
"Does acks=all mean it is on disk","No. It is in the PAGE CACHE of every in-sync replica. Kafka does not fsync per message (roughly an order of magnitude throughput cost). Durability comes from replication across failure domains, which is why broker.rack is load-bearing.",04-streaming-apis
"Unclean leader election is not staleness","The log TRUNCATES. Offsets that consumers already read and committed get reused for different messages, so anything keyed on (topic, partition, offset) is now wrong. Default changed to false in KIP-106 because users were losing data without knowing they had opted in.",04-streaming-apis
"Compaction is not compression","It retains at least the last value per KEY. A topic with unique keys compacts to exactly its original size and you have just lost your retention policy. Compaction is a semantic choice about what the topic means, not a storage optimisation.",04-streaming-apis
"Why a compacted topic beats a 7-day topic","Replay from offset 0 gives you at least one message for every key that ever existed, so a cold start reconstructs COMPLETE state. A 7-day retention topic only teaches a new consumer about keys that changed this week.",04-streaming-apis
"delete.retention.ms is a correctness parameter","A tombstone is removed after 24h by default. A consumer lagging more than that replays the log, never sees the deletion, and keeps a deleted key forever, silently, with no error. That makes consumer lag alerting a correctness control.",04-streaming-apis
"Never add partitions to a compacted topic","Compaction keeps the last value per key PER PARTITION, so correctness needs every message for a key in one partition. Adding partitions rehashes keys: the new partition has no history and the old one keeps a stale value that is now permanently 'last'. Recovery is a new topic plus full replay.",04-streaming-apis
"The Kafka timeout that actually fires","max.poll.interval.ms, not session.timeout.ms. Since KIP-62 heartbeats run on a BACKGROUND thread, so a consumer stuck processing looks alive right up until it evicts itself. Fix is max.poll.records DOWN, not the timeout up.",04-streaming-apis
"Eager vs cooperative rebalancing","Eager revokes EVERY partition from EVERY consumer and the whole group idles until SyncGroup. Cooperative does two rounds but only revokes partitions that are actually moving, so on a 20-node group losing one node, 19/20 of partitions never pause. Default from Kafka 3.0.",04-streaming-apis
"What static membership buys","A stable group.instance.id means the coordinator ignores a member's departure until session.timeout.ms and gives back the identical assignment if it returns. A 24-pod rolling restart goes from 24 stop-the-world rebalances to ZERO. Price: a genuinely dead member's partitions sit idle for the full session timeout.",04-streaming-apis
"Why one slow consumer stalls the group","The coordinator holds all JoinGroup requests until every member joins or rebalance.timeout.ms expires. A member busy inside poll() cannot send JoinGroup, so the slowest member sets the barrier. KIP-848 fixes this by moving assignment to the broker.",04-streaming-apis
"Never autoscale a Kafka consumer on CPU","An I/O-bound consumer blocked on a slow dependency has LOW CPU while lag grows, so the HPA scales DOWN exactly when you need to scale up. Observed: 6 pods to 2 while lag climbed at 7k/s.",04-streaming-apis
"Lag in messages is not lag in time","50,000 messages is 4 seconds at 12k/s and 8 hours at 1.7/s. Alert on projected drain time (lag / consumption rate), which is stable across partitions and does not change meaning when traffic doubles.",04-streaming-apis
"Sizing a KEDA lagThreshold","Derive it from the SLO. 60s target x 14,000/s = 840,000 total lag budget. Then set the threshold to about a THIRD of that, because pod start plus rebalance is 30-40s and lag keeps growing during the reaction.",04-streaming-apis
"maxReplicaCount == partition count","A consumer group cannot have more active consumers than partitions; extras idle completely. An HPA allowed past the partition count will go there under load, trigger rebalances on the way in, and do nothing. Most common KEDA-on-Kafka misconfiguration.",04-streaming-apis
"Autoscaling cannot fix a skewed partition","If lag is concentrated on 2 of 12 partitions, scaling to 12 gives each hot partition one consumer, processing at single-consumer speed. The fixes are all upstream: repartition, salt the hot key, or drop the ordering requirement.",04-streaming-apis
"Compatibility mode is a deploy-order policy","BACKWARD means a NEW consumer reads OLD data, so consumers must upgrade first. FORWARD means old consumers read new data, so producers go first. It looks like a serialisation setting and it is actually about who is allowed to deploy first.",04-streaming-apis
"Why transitive compatibility","Non-transitive only checks against the IMMEDIATELY previous version, so v1 and v3 can be mutually incompatible while each neighbouring pair passes. Bootstrapping a consumer from offset 0 reads all of them. On any topic with real retention, non-transitive is a trap.",04-streaming-apis
"auto.register.schemas=false","With the default true, any producer registers a new schema just by starting up, so your data contract is decided by deploy ordering. False turns registration into a reviewed CI step and converts a runtime surprise into a build failure.",04-streaming-apis
"You cannot add a required field","Under any mode stricter than NONE. A required field is by definition incompatible with data written before it existed. Path: add it WITH a default, deploy producers, wait out retention, and enforce 'required' in application validation. If it is truly mandatory, it is a new event type.",04-streaming-apis
"Checkpoint vs savepoint","Checkpoints are Flink's: automatic, incremental, auto-deleted, format not portable across versions, for RECOVERY. Savepoints are yours: manual, self-contained, retained, portable, for OPERATIONAL CHANGE (new parallelism, new topology, version upgrade).",04-streaming-apis
"Why RocksDB over heap state","Not speed. Full checkpointing puts a hard ceiling on state size: 400 GB every 30s needs 13 GB/s of upload, which is impossible. Incremental (RocksDB only) needs about 70 MB/s. Plus no GC pauses scaling with state.",04-streaming-apis
"Incremental checkpointing is one-way","Recovery is NOT incremental: restore reads the full state. Plan recovery time from download bandwidth (400 GB over 20 TMs at 1 Gbps is about 3 minutes) and enable local recovery so task-level restarts skip the download.",04-streaming-apis
"Barrier alignment and unaligned checkpoints","An operator waits for the barrier on ALL inputs before snapshotting, so under backpressure alignment time dominates and checkpoint duration spikes exactly when the job is already struggling. Unaligned checkpoints (FLIP-76) include in-flight buffers instead of waiting.",04-streaming-apis
"Always set operator UIDs","Without explicit .uid(), Flink derives UIDs from topology structure, so ADDING ONE OPERATOR changes them and your savepoint no longer restores. The failure is silent until the moment you need it under pressure.",04-streaming-apis
"maxParallelism is a one-way door","It is the key-group count, fixed when state is first created, and CANNOT be changed by restoring a savepoint. It defaults low (min 128) and permanently caps scale-out. Changing it means rebuilding state from the source: 11 hours of replay in one case.",04-streaming-apis
"Kafka Streams vs Flink: the real question","Not latency or state size. Whether the job touches anything other than Kafka, and whether you already operate a cluster. Kafka Streams is a LIBRARY in your existing service; Flink is INFRASTRUCTURE you staff.",04-streaming-apis
"Kafka Streams' structural ceiling","Parallelism is bounded by the source topic's partition count, because it IS a consumer group. Flink's shuffle after keyBy decouples parallelism from partitions, so a 12-partition source can feed an aggregation at parallelism 200.",04-streaming-apis
"Where Spark Structured Streaming actually fits","Micro-batch means a latency floor of a few hundred ms, realistically seconds. In exchange: foreachBatch gives you a real DataFrame per batch, so a Delta merge is one line, and you get genuine code sharing with the batch job.",04-streaming-apis
"Where Kafka Streams and Flink keep durable state","Kafka Streams: compacted changelog TOPICS, so state churn adds load to your Kafka cluster. Flink: checkpoints in OBJECT STORAGE. That difference is a real capacity consideration people miss when sizing Kafka.",04-streaming-apis
"The dual-write problem in one line","Two writes with no transaction spanning both. The DB write usually succeeds and the publish is the flakier one, so the failure is silent divergence: the row exists, the event never fired, and nothing errors anywhere.",04-streaming-apis
"Why putting the Kafka send inside @Transactional is worse","The send is not part of the DB transaction. It can complete BEFORE the commit (consumer reads an event and queries for an invisible row), and if the transaction rolls back you have published an event for something that never happened.",04-streaming-apis
"The transactional outbox","Insert the event into an outbox table in the SAME transaction as the state change, then let CDC turn that row into a message. One write, not two writes in a clever order. Debezium's EventRouter unwraps it: topic from aggregate_type, key from aggregate_id.",04-streaming-apis
"Why polling updated_at is not CDC","It misses deletes entirely, misses intermediate values, and has a real correctness bug: a transaction starting at T1 and committing at T3 writes updated_at=T1, so a poll at T4 using 'WHERE updated_at > T2' never sees it. Rows are silently skipped.",04-streaming-apis
"Raw CDC vs outbox","Raw CDC publishes your SCHEMA: every consumer couples to your column names and must infer 'the order shipped' from 'status changed to 4'. Use raw CDC for pipelines you own end to end; use the outbox for anything other teams consume.",04-streaming-apis
"The CDC risk that takes down the database","An unconsumed Postgres replication slot retains WAL indefinitely and fills the disk. It triggers when the connector is DOWN, which is during an incident. Alert on slot lag; set max_slot_wal_keep_size and accept re-snapshotting as the consequence.",04-streaming-apis
"Poison pill: the defining property","Retrying is useless, and a naive consumer retries forever, blocking the partition. Worst case is a DESERIALISATION failure: it happens before your listener runs, so the poll loop dies, restarts, reads the same record, and loops. Fix: ErrorHandlingDeserializer.",04-streaming-apis
"Why retry topics instead of retry in place","In-place retry blocks the partition, and a backoff longer than max.poll.interval.ms evicts the consumer and rebalances the whole group. Retry topics have their own consumer groups, so the main topic never blocks. Cost: ordering is broken.",04-streaming-apis
"Size retry tiers from real outage durations","Three immediate retries span about 200ms and every real outage lasts minutes, so 'attempts=3' is functionally one attempt. Pull the last year of incidents, take the median and p90, and put tiers there. Three tiers is almost always enough.",04-streaming-apis
"The DLQ alert nobody writes","Alert on 'DLQ has messages AND nothing replayed in 7 days'. A DLQ with a permanent backlog is UNPROCESSED WORK, not archived errors. One team accumulated 847,000 messages over 14 months, 83 percent of which were replayable.",04-streaming-apis
"Classify before you retry","Transient (timeout, connection, 5xx, 429, optimistic lock) -> retry. Permanent (deserialisation, validation, 4xx other than 408/429, entity not found) -> dead-letter immediately. Treating permanent as transient burns retries; treating transient as permanent fills the DLQ during an outage.",04-streaming-apis
"request(n) is why Reactive Streams exists","Without it you have observer-pattern callbacks: the producer pushes and the consumer copes. With it the consumer grants permission for n elements and the producer is CONTRACTUALLY forbidden from sending more. That inverts control of rate while keeping push delivery.",04-streaming-apis
"Backpressure vs buffering vs throttling","Buffering absorbs a rate mismatch until memory runs out. Throttling drops or delays at the consumer. Backpressure propagates the constraint UPSTREAM so the original producer slows down. A backpressured pipeline slows; a buffered one falls over.",04-streaming-apis
"Where to put request() in a Subscriber","AFTER the work, not before. Requesting at the top of onNext lets the publisher emit while you are still processing, which silently reverts to unbounded push. Accumulate, do the work, then request the next batch.",04-streaming-apis
"The end-to-end backpressure chain","Slow client shrinks the TCP receive window -> server send buffer fills -> Netty channel not writable -> Netty stops calling request(n) -> Flux stops emitting -> R2DBC stops fetching -> database cursor pauses. Nobody wrote code for that.",04-streaming-apis
"Reactive types do not imply backpressure","A collectList() in the middle, a JDBC driver at the bottom, or any operator requesting Long.MAX_VALUE gives you reactive TYPES over a buffering pipeline. The resulting OOM surfaces in unrelated code, minutes later.",04-streaming-apis
"The reactive memory calculation nobody does","async boundaries x prefetch (default 256) x element size x concurrent subscriptions. Three publishOn calls with 10 KB elements is 7.7 MB in flight per subscription; at 5,000 subscriptions that is 38 GB.",04-streaming-apis
"subscribeOn vs publishOn","subscribeOn changes where the SUBSCRIPTION happens, so it affects the SOURCE, and its position in the chain is irrelevant (nearest to source wins, others are dead code). publishOn changes where SUBSEQUENT operators run, so its position is everything.",04-streaming-apis
"Which scheduler for what","parallel() = one thread per core, for CPU-bound work. boundedElastic() = elastic up to 10x cores, for BLOCKING calls. Blocking on parallel() is the second-worst thing after the event loop: it removes a core's compute capacity from every pipeline in the JVM.",04-streaming-apis
"Why your subscribeOn has no effect","Assembly-time evaluation. Flux.fromIterable(loadFromDisk()) runs loadFromDisk() when the pipeline is BUILT, on the building thread, before any subscription exists. No scheduler operator can move work that already happened. Wrap in Flux.defer or Mono.fromCallable.",04-streaming-apis
"Mono.just vs Mono.fromCallable","Mono.just(expensiveCall()) evaluates immediately at assembly time. Mono.fromCallable(() -> expensiveCall()) defers to subscription. The distinction is invisible in the type signature and causes most 'my scheduler is not working' bugs.",04-streaming-apis
"The event-loop blast radius","MVC has ~200 threads so blocking one costs 0.5% and affects one request. WebFlux has one loop per core, and each loop OWNS many connections for their lifetime, so blocking one costs 12.5% on 8 cores and stalls every connection assigned to it.",04-streaming-apis
"The blocked-event-loop signature","Flat throughput, LINEARLY growing latency with concurrency, and LOW CPU. Then Little's Law: 44 rps x 0.23s = 10 busy servers. If that matches your event-loop count you have found it without a profiler.",04-streaming-apis
"Why the health check needs its own port","A stalled event loop cannot answer a liveness probe, so the orchestrator kills a pod that was STUCK rather than dead. If the cause is an in-process cold cache, the replacement fails identically: a crash loop. management.server.port fixes it.",04-streaming-apis
"An in-process cache can amplify a restart","If a slow dependency sits behind an in-process cache, every restart guarantees a cold cache and an immediate blocking call. Restarting, normally a remedy, becomes the amplifier. Check for this whenever a restart makes things worse.",04-streaming-apis
"The blocking sources nobody looks for","A SYNCHRONOUS LOG APPENDER (a blocking write per statement), a contended synchronized block, a vendor SDK blocking internally, and CPU-bound work, which BlockHound will never flag because it is not blocking.",04-streaming-apis
"Timeout vs deadline","A timeout is per HOP and starts a fresh clock at each level. A deadline is per REQUEST: one absolute instant every hop shares. Three levels of 2s timeouts with 2 retries each is up to 54s of work for a request the client abandoned at 5s.",04-streaming-apis
"Retry amplification is multiplicative in depth","3 retries means 4 ATTEMPTS. Three levels deep: 4 x 4 x 3 = 48 calls to the leaf service for one user request. That is the mechanism behind most cascading failures, and the off-by-one makes every estimate optimistic.",04-streaming-apis
"The highest-value line in deadline propagation","On arrival: if the deadline has already expired, return DEADLINE_EXCEEDED and do NO work. Under overload the queue is full of abandoned requests, so draining them instantly is free load shedding. ~40% of one service's load was work for closed connections.",04-streaming-apis
"Put remaining duration on the wire, not absolute time","Immune to clock skew; convert to an absolute deadline locally on receipt so only local elapsed time matters. This is what gRPC's grpc-timeout does, and it is why it is a duration string.",04-streaming-apis
"Cancellation is cooperative below the API boundary","Interrupting a thread blocked in a JDBC query does not stop the database executing it. The resource you most want to reclaim is the hardest to cancel. Set the database's own statement_timeout from the remaining deadline: enforce the budget where the work happens.",04-streaming-apis
"Deadlines bound latency; retry budgets bound load","Deadline-aware retries cap one request TREE. Under widespread degradation every request retries within its own budget and aggregate load against the failing service still multiplies. Cap retries at ~10% of successful request volume, per fleet.",04-streaming-apis
"Parallel calls share a deadline budget","A parallel branch costs the MAX of its children, not the sum, which is a real argument for parallelising independent calls. Also mark each downstream required or optional: an optional one that overruns should degrade to a default, not fail the request.",04-streaming-apis
"Federation vs schema stitching","Stitching puts join config in the GATEWAY, so every schema change is a gateway change. Federation puts declarations in the SUBGRAPHS and derives the plan, so a team ships a field without touching shared infrastructure. The difference is where the coupling lives.",04-streaming-apis
"What _entities does","The generated entry point every subgraph implements: given a list of representations ({__typename, key}), return those objects. It is how the router says 'here are 12 product IDs, give me your fields for them'. @key defines the identity passed between subgraphs.",04-streaming-apis
"Federation batches across services, not within one","One _entities call with 50 representations becomes 50 database queries without a DataLoader inside the subgraph. Federation makes N+1 HARDER to see, because the router's batching creates the impression it is handled.",04-streaming-apis
"What @requires costs","A serialisation point: the router must fetch the required fields from their owner BEFORE calling the requiring subgraph, so two parallel calls become sequential. Alternative: duplicate the field as @shareable. Right for slow-changing fields, a correctness risk for volatile ones.",04-streaming-apis
"@override is field-level expand-contract","Put @override(from: 'old-subgraph') on the field in the new subgraph; the router shifts that field's traffic; remove it from the old subgraph later. One field moves between services with no client change and no coordinated deploy.",04-streaming-apis
"Why GraphQL breaks HTTP caching","The cache key is gone. REST's key is the URL, which every CDN and proxy understands. GraphQL POSTs to one endpoint with the query in the body, so every request is opaque and identical. Persisted queries restore a key: hash + variables in a GET URL.",04-streaming-apis
"APQ vs safelisted persisted queries","APQ is a BANDWIDTH optimisation: the server learns queries at runtime, so it is not a security control. Safelisting extracts queries at BUILD time into a manifest and rejects anything else, which bounds query cost by review instead of by runtime analysis.",04-streaming-apis
"@cacheControl takes the MINIMUM","Response TTL is the minimum across every field in the selection set. One unannotated field defaults to maxAge 0 and poisons the cacheability of everything requested with it. In one case that was a viewCount integer costing about $9,000 a month.",04-streaming-apis
"Depth limiting alone is half a control","A depth-2 query asking for 1,000 items each with 1,000 sub-items is shallow and enormous. Complexity scoring must multiply child cost by the pagination argument (first/limit), and weight per field: 1 for a loaded parent, 50-100 for search or model inference.",04-streaming-apis
"Entity cache beats response cache","A cached Product:P42 serves EVERY query touching that product in any shape, so hit rates are far higher, and invalidation is precise (one key). A response cache keys on the whole operation, so every query shape is its own entry and invalidation means finding them all.",04-streaming-apis
"Set query limits from measured traffic","Log complexity for two weeks WITHOUT enforcing, find the p99 of legitimate queries (340 in one case), set the limit at 3-4x that (1,500). Limits set from intuition reject real users.",04-streaming-apis
"The protobuf key, in one line","key = (field_number << 3) | wire_type. So names are NOT on the wire, and a parser can skip an unknown field because the wire type tells it the length. That one fact explains why renaming is free, adding is safe, and reusing a number corrupts data.",04-streaming-apis
"Field-number reuse is the silent one","Same wire type means it parses cleanly and produces wrong values with NO error. It only surfaces when old data meets new code: archives, replays, backfills. One team corrupted 400,000 records and took 2 days to root-cause. Always 'reserved' on deletion.",04-streaming-apis
"int32 vs sint32","A negative int32 is sign-extended to 64 bits before varint encoding, so -1 takes TEN bytes. sint32 uses zigzag (0,-1,1,-2 -> 0,1,2,3) so small magnitudes stay small either way. Never switch between them: both are VARINT so it parses and every value is wrong.",04-streaming-apis
"Protobuf is not self-describing","You can recover structure from raw bytes (field numbers, wire types) but not MEANING: int32 vs enum, string vs embedded message are indistinguishable. Avro carries a schema reference; protobuf carries nothing. The schema must travel separately.",04-streaming-apis
"proto3 optional, and why it came back","Originally proto3 dropped field presence, so a scalar set to its default is not encoded and 'absent' is indistinguishable from 'zero'. That makes partial updates impossible to express. proto3.15 restored optional as a synthetic one-field oneof: wire-compatible, adds has_().",04-streaming-apis
"The three amplifications","WRITE: bytes to disk per byte of user data. READ: disk reads per logical read. SPACE: disk used per byte of live data. The RUM conjecture says optimising any two costs you the third; every engine config is a point on that surface.",03-storage
"Why a B-tree write is expensive","The unit of update is a PAGE. 100 bytes changed means a 16 KB page write, plus WAL, plus InnoDB's doublewrite copy: ~33 KB for 100 bytes, ~330x. It amortises with sequential keys and scatters with random UUIDs, which is why UUIDv7 exists.",03-storage
"Why bloom filters do not help range scans","A filter answers 'is key K present'; a scan asks 'what is in [A,B)', which any overlapping SSTable may contribute to. So the scan merges across all of them. Range scans are the LSM's genuine weakness against a B-tree's linked leaf pages.",03-storage
"Compaction strategy matters more than the engine choice","Same RocksDB, same data, same hardware: leveled compaction was 4.3x WORSE than Postgres at range scans (47ms vs 11ms); time-windowed compaction was nearly 2x BETTER (6ms). 'We chose an LSM' is the coarse decision.",03-storage
"Why an LSM stalls writes","Compaction cannot keep up, so the engine throttles DELIBERATELY to prevent unbounded read amplification. In RocksDB: level0_slowdown_writes_trigger and level0_stop_writes_trigger, because L0 files have overlapping ranges and every read checks all of them.",03-storage
"Before declaring write amplification a problem","Measure DWPD consumed and provisioned-IOPS utilisation. On cloud storage with provisioned IOPS, amplification is directly money. On local NVMe well under its endurance rating, it is invisible. That turns an architectural debate into arithmetic.",03-storage
"Why STCS needs 50 percent free disk","A merge writes its output BEFORE deleting its inputs, so merging the largest tier needs the existing files plus the new copy: peak approaches 2x live data. Running out mid-compaction is self-reinforcing: cannot compact, so files accumulate, so more space is used.",03-storage
"Why leveled compaction's write amp is so high","Levels are DISJOINT. Moving one file from L(n) to L(n+1) means merging it with every overlapping file there, and since that level is 10x larger, one file overlaps about 10. So advancing one file rewrites ~11 files' worth, at every level it descends.",03-storage
"What TWCS actually buys","Expiry becomes DROPPING A WHOLE FILE instead of merging gigabytes to reclaim megabytes. Old data is written once and never touched again, so write amp approaches 1-3x. And a time-range query reads only that window's files.",03-storage
"When TWCS is actively harmful","It needs in-order arrival, a UNIFORM TTL, and no updates to old data. One backfill job writing six-month-old timestamps keeps those windows alive, they recompact, and files that should have been dropped stay. Nothing alerts. Enforce it at the application layer.",03-storage
"The compaction decision rule","Time series + uniform TTL + in-order arrival -> TWCS. Read-heavy or update-heavy (same keys rewritten) -> LCS. Write-heavy, insert-mostly, rarely read -> STCS. It is a PER-TABLE decision and the default is right for exactly one of the three.",03-storage
"Why nodetool compact is close to a one-way door","On STCS it produces one enormous SSTable that will not be compacted again until three more of similar size exist, which for a large table means never. Every tombstone inside it is now frozen too. It is a one-off tool, not maintenance.",03-storage
"RocksDB memory is NOT the block cache","It is block cache + memtables + INDEX AND FILTER BLOCKS + reader overhead. Index and filter blocks default to living OUTSIDE the cache budget and grow with the dataset: 100M keys can be ~600 MB before a data block is cached. Set cache_index_and_filter_blocks=true.",03-storage
"A RocksDB write stall is deliberate","The engine judges that accepting more writes would make reads unusable, so it slows or stops them. From outside the process it is indistinguishable from a hang: no error, no exception. rocksdb.is-write-stopped must be a metric.",03-storage
"The four write-stall triggers, and what each means","L0 file count: flushes outpacing L0->L1 compaction. Immutable memtable count: flush cannot keep up, usually disk-bound. Pending compaction bytes: write rate exceeds what the disk sustains at this amplification. Each names its own cause.",03-storage
"max_bytes_for_level_base, the obscure one","L1's target size. If L1 is small relative to the L0 batch merged into it, every L0->L1 compaction rewrites essentially ALL of L1. Size it as write_buffer_size x min_write_buffer_number_to_merge x level0_file_num_compaction_trigger.",03-storage
"Compression per level","No compression at L0/L1 (rewritten constantly by compaction, so you pay CPU repeatedly for data about to be rewritten), LZ4 in the middle, ZSTD at the bottom (most of the data, rarely rewritten). Commonly cuts disk 40-60% vs uniform LZ4.",03-storage
"Shared block cache in Kafka Streams / Flink","There is one RocksDB instance PER STORE PER PARTITION, so per-instance caches multiply with partition assignment and change during rebalancing. One static shared cache bounds total memory regardless of assignment.",03-storage
"Cassandra: model queries, not entities","One table per access pattern, named after the query. There are no joins, no cross-partition aggregation, and no planner to rescue a bad schema. Denormalisation is the DESIGN METHOD, not an optimisation applied later.",03-storage
"Partition key vs clustering columns","PRIMARY KEY ((partition_key), clustering...). The partition key decides WHERE the data lives (which node) and every query must supply it in full. Clustering columns decide HOW it is sorted within the partition, which is what makes range queries work.",03-storage
"The partition-size arithmetic to do at design time","rows/day x retention days x row size. Over 100 MB is a warning, over 1 GB is a problem. One route accumulating 40,000 scans/day reached 3.1 GB in eleven months, and the degradation was gradual so nothing ever alerted.",03-storage
"Bucketing, and what it costs","Put a time component in the partition key so partitions are bounded by construction. The cost: a query spanning buckets becomes several queries. Size the bucket so the DOMINANT query reads one partition and the partition stays under ~100 MB.",03-storage
"The one legitimate use of BATCH","Atomicity across the denormalised copies of ONE logical write. The batch log guarantees all statements eventually apply, which stops the copies diverging. Batching unrelated writes makes one coordinator fan out to every partition: worse throughput, not better.",03-storage
"Why a Cassandra secondary index is usually wrong","It is LOCAL per node, indexing only that node's data. A query without a partition key contacts every node and merges partial results: scatter-gather, whose latency is the slowest node's. SAI in 5.0 improves the local index and does not change this.",03-storage
"R + W > RF","If replicas read plus replicas written exceeds RF, the sets must overlap, so a read sees at least one replica with the latest write. RF=3, W=2, R=2: 4>3. Gives read-your-writes and monotonic reads. Does NOT give linearizability: concurrent writes resolve by timestamp.",03-storage
"The gc_grace_seconds invariant","A full repair must complete within gc_grace_seconds. If a replica misses a delete and the tombstone is collected before repair reaches it, the row RESURRECTS. Silent: nothing logs it. Repair time grows with data while gc_grace is a constant, so clusters cross the line as they grow.",03-storage
"The three repair mechanisms, and what each misses","Hinted handoff: only within the 3h hint window, and lost if the coordinator dies. Read repair: only data that is READ, so cold data (the most drifted) is never fixed. nodetool repair: complete, and therefore the one whose schedule is a correctness requirement.",03-storage
"Why LOCAL_QUORUM in multi-DC","With RF=3 in each of 2 DCs, RF is 6 so QUORUM is 4, which cannot be satisfied in one DC: cross-DC latency on every query, and unavailable if a DC is partitioned. LOCAL_QUORUM is 2 locally. It gives read-your-writes WITHIN a DC, not across.",03-storage
"gc_grace_seconds = 0 is safe when","Uniform TTL and no client deletes. TTL expiry tombstones derive their timestamps from the write itself, so every replica agrees and resurrection is impossible. Standard for time-series tables under TWCS, and it removes a large source of tombstone accumulation.",03-storage
"Postgres bloat is usually an xmin horizon problem","Autovacuum can run CONTINUOUSLY and reclaim nothing if something holds the horizon: no tuple that died after that point is removable anywhere in the database. One idle transaction open 31 days meant 684 million unreclaimable dead tuples.",03-storage
"The four holders of the xmin horizon","Long-running or idle-in-transaction sessions (pg_stat_activity.backend_xmin), replication SLOTS with a lagging consumer, standbys with hot_standby_feedback=on, and orphaned prepared transactions (pg_prepared_xacts). Three of the four are invisible if you only look at active queries.",03-storage
"idle_in_transaction_session_timeout","Off by default, and it converts an unbounded database-wide bloat failure into a five-minute connection error the application retries. Added in 9.6 precisely because poolers and ORMs leave transactions open.",03-storage
"HOT updates, and what breaks them","If an update changes NO indexed column and fits on the same page, Postgres chains it in-page and writes NO index entries. Breaks on: an index on a mutable column (updated_at) and fillfactor=100. Going from 1.5% to 94% HOT changes a table's bloat profile completely.",03-storage
"XID wraparound","XIDs are 32-bit and compared modularly, so a row older than ~2 billion transactions would appear to be in the FUTURE and vanish. Postgres refuses new transactions before that: a full outage needing single-user mode. Before it, an anti-wraparound autovacuum starts, uncancellable.",03-storage
"VACUUM vs VACUUM FULL vs pg_repack","Plain VACUUM marks space reusable WITHIN the table, online. VACUUM FULL rewrites the table and holds ACCESS EXCLUSIVE the whole time, blocking reads too. pg_repack achieves the same reclamation with a brief lock only at the swap.",03-storage
"Cost in EXPLAIN is not milliseconds","It is a unitless number calibrated so a sequential page read is 1.0. Comparing costs between two DIFFERENT queries is meaningless; comparing plans for the SAME query is the only valid use.",03-storage
"The first thing to read in a query plan","The estimate-vs-actual ROW COUNT per node, not the time. The planner is usually making a reasonable choice given what it believes, so a 3,000x row error means it solved a different problem, and the fix is a statistics fix rather than an index.",03-storage
"actual time and rows are PER LOOP","A node showing actual time=0.012 rows=1 loops=284119 took about 3.4 SECONDS, not 0.012 ms. Reading it as per-node total is the most common misreading of a plan, and it hides exactly the case where a cheap operation runs far too many times.",03-storage
"Extended statistics, the fix people do not know","The planner assumes column independence and multiplies selectivities, so WHERE city='Toronto' AND province='ON' is estimated far too low. CREATE STATISTICS (dependencies, ndistinct) ON city, province fixes it. Correlated columns are extremely common.",03-storage
"Sort Method: external merge","The sort spilled to disk because work_mem was too small. Related signals: Batches > 1 on a hash join, lossy heap blocks on a bitmap scan. Fix with SET LOCAL work_mem, never globally: it applies per NODE per PARALLEL WORKER.",03-storage
"random_page_cost on SSD","Default 4.0 assumes spinning disks and biases the planner away from index scans. On SSD it should be ~1.1. A team seeing 'too many sequential scans' should check this before adding indexes.",03-storage
"Why Postgres needs an external pooler","Process per connection: each backend is an OS process with MBs of private memory, and they contend on ProcArray, which is walked when taking a snapshot. So the cost of STARTING a transaction grows with connection count, including idle ones.",03-storage
"What transaction pooling breaks","Anything session-scoped: session SET (use SET LOCAL in a transaction), pg_advisory_lock (use pg_advisory_xact_lock), LISTEN/NOTIFY (needs a direct connection), temp tables, WITH HOLD cursors. Prepared statements worked from PgBouncer 1.21 via max_prepared_statements.",03-storage
"The dangerous PgBouncer breakage","pg_advisory_lock does not ERROR under transaction pooling, it LEAKS a lock onto a server connection someone else is now using. The unlock runs on a different connection and fails silently.",03-storage
"An application pool is not a substitute for PgBouncer","HikariCP bounds connections PER INSTANCE. Seventy pods with a pool of 20 is 1,400 connections whatever each pool does. Only a shared pooler bounds the fleet total, which is the number Postgres cares about.",03-storage
"cl_waiting climbing: do NOT raise the pool","A saturated pool is usually a SYMPTOM: transactions held too long. Check idle-in-transaction first. The classic cause is an external HTTP call inside @Transactional, holding a connection 1.9s to do 5ms of database work.",03-storage
"Pooling allocates capacity, it does not create it","Introducing PgBouncer can make p99 WORSE at first, because contention spread thin across 1,400 connections becomes a visible queue. That queue is the diagnosis, not a regression.",03-storage
"LSI vs GSI in one sentence","An LSI is an alternative SORT ORDER within the same partition; a GSI is a separate table DynamoDB keeps in sync. Default to GSIs: LSIs can only be created WITH the table and cap an item collection at 10 GB.",03-storage
"A throttled GSI throttles the base table","DynamoDB cannot accept a write it cannot propagate to the index, so back-pressure flows backward. This is why a table can throttle at 3% of provisioned capacity: check every GSI's consumed capacity, not just the table's.",03-storage
"Sparse indexes","An item is in a GSI only if it HAS that index's key attributes. Write pendingStatus only while pending and REMOVE it on completion: the index holds thousands of items instead of 200 million. Fixes cost and the hot partition at once.",03-storage
"Adaptive capacity does not raise the per-partition ceiling","It isolates a hot partition and gives it a larger share of table capacity. The 1,000 WCU / 3,000 RCU per-partition limit is hard. A key needing 4,000 WCU still throttles, in on-demand mode too.",03-storage
"Write sharding: random vs calculated","Random distributes perfectly and makes point lookups IMPOSSIBLE. Calculated (hash(id) % N) is deterministic so point lookups work, and distributes only as well as the entities do. N is effectively permanent, so size from the ceiling you need and err high.",03-storage
"Iceberg's core trick","A commit is an ATOMIC SWAP of the catalog's metadata pointer. That single property gives serialisable isolation, time travel, and rollback that is instant on a table of any size, because it changes a pointer rather than moving data.",03-storage
"Hidden partitioning, and what it really buys","Iceberg stores a TRANSFORM (days(event_time)) in metadata, so users filter the source column and get pruning. The deeper win: layout stops being part of the table's interface, which is what makes partition EVOLUTION possible without a rewrite.",03-storage
"The Iceberg small-files signature","Planning time far exceeding execution time. One table: 38s planning, 6s execution, 41 million files averaging 1.8 MB. Commit interval x partitions per commit IS your file count, so freshness and file count are the same knob.",03-storage
"binpack vs sort when compacting Iceberg","binpack just combines files, fixing planning time and request cost. sort ALSO orders rows so per-file min/max stats are narrow and files become prunable, which improves EXECUTION too. Unsorted files each span the full value range.",03-storage
"The four Iceberg maintenance procedures","expire_snapshots (bound storage), rewrite_data_files (compact), rewrite_manifests (the metadata layer, separately forgotten), remove_orphan_files. Nothing runs them for you, and the degradation is gradual.",03-storage
"Merge-on-read requires scheduled compaction","It writes delete files that readers apply: cheap writes, and every read pays to merge. Without compaction, delete files accumulate and reads get slower every day with nothing alerting. Same failure shape as unvacuumed Postgres.",03-storage
"The two transformer shapes that matter","[B, H, S, S] for attention scores (1.07 GB at B=4,H=32,S=2048 fp16; what FlashAttention avoids materialising) and 2 x H_kv x d_h x L bytes per token for the KV cache (what limits concurrent requests on a GPU).",05-ai-llm
"Where transformer parameters actually live","The FFN, at ~72% of each layer. Three 4096x14336 matrices = 176M per layer vs attention's 42M under GQA. Attention gets the conceptual attention and the feed-forward network is where the capacity sits.",05-ai-llm
"Is attention really quadratic","The S x S score matrix is, and at S=2048 with d=4096 attention is only ~5% of FLOPs. It dominates past roughly S = d. FlashAttention removed the MEMORY problem by tiling; the compute term is real only at long context.",05-ai-llm
"GQA is a 4x hardware difference","KV cache per token = 2 x H_kv x d_h x 2 bytes x L. MHA (H_kv=32): 4.29 GB per 8k request. GQA (H_kv=8): 1.07 GB. On an 80GB GPU with a 16GB model that is 13 vs 54 concurrent requests.",05-ai-llm
"Why the 1/sqrt(d_h) scaling","The dot product of two d_h-dimensional unit-variance vectors has variance d_h, so at d_h=128 raw scores have SD ~11. Softmax saturates and the gradient vanishes. One line, load-bearing.",05-ai-llm
"The decode throughput ceiling","Weights read per step / memory bandwidth. 16 GB fp16 model on an H100 at 3.35 TB/s = 4.8 ms/step = ~209 tokens/sec/sequence, INDEPENDENT of batch size. Which is why batching helps throughput, not per-stream latency.",05-ai-llm
"Training FLOPs rule of thumb","6 x N_params x N_tokens (forward is 2N, backward ~2x forward). 8B params on 2T tokens = 9.6e22 FLOPs. At ~1e15 achievable FLOP/s per H100 that is ~1,111 GPU-days.",05-ai-llm
"Why RoPE gives relative position exactly","It rotates Q and K by an angle proportional to position. Rotation is multiplication by e^(i*theta), so the dot product of q rotated by m and k rotated by n depends only on e^(i(m-n)theta): absolute positions CANCEL. Algebraic, not approximate.",05-ai-llm
"Past the trained context length, quality COLLAPSES","It does not degrade. One measurement: 84.2% accuracy at 4k became 2.8% (random) at 32k with no extension method. 'Cannot extrapolate' means incoherent output, not worse output.",05-ai-llm
"PI vs NTK-aware vs YaRN","Position Interpolation divides positions by a scale: simple, needs fine-tuning, and it REGRESSES short-context accuracy (5 points in one case) by compressing local detail. NTK-aware scales the BASE instead, often working with no fine-tuning. YaRN interpolates only long-wavelength dims: best quality per unit of fine-tuning.",05-ai-llm
"Lost in the middle","Accuracy is high when relevant information is at the START or END of the context and much lower in the MIDDLE: a 20-point gap in one 32k measurement. A 128k window is not uniformly usable, and needle-in-a-haystack benchmarks overstate real capability.",05-ai-llm
"Long context is not a substitute for retrieval","Measured: 32k full document gave 76.8% accuracy, 8.4s, $0.094/query. 4k of retrieved passages gave 89.1%, 1.1s, $0.011. Better, 8x faster, 9x cheaper. Long context makes retrieval more FORGIVING, not unnecessary.",05-ai-llm
"Why the field chose RoPE over ALiBi","ALiBi extrapolates natively (the distance penalty is defined for any distance) and bakes in a monotonic RECENCY bias. For long-context retrieval the relevant fact may be anywhere, so systematically discounting distant tokens is the wrong prior.",05-ai-llm
"The multilingual token penalty, measured","Same sentence in cl100k_base: English 1.0x, Spanish 1.7x, German 1.8x, Russian 3.2x, Japanese 3.6x, Thai 6.8x. A 128k window holds ~96,000 English words and ~14,000 Thai ones, at the same price per token.",05-ai-llm
"'1 token is 4 characters' is an English rule","It is ~2.8 for code, ~1.8 for Russian, ~0.7 for Japanese and Thai. Budgeting or pricing multilingual products with the English figure underestimates by several times. Count with the actual tokeniser.",05-ai-llm
"The truncation consequence of tokenisation","A fixed 8,000-token limit truncated 2.1% of English support threads and 58.9% of Thai ones. It presented as a model QUALITY complaint from one market, and the aggregate metric hid it because that market was 4% of volume.",05-ai-llm
"Why models cannot count letters","They never see letters. 'strawberry' is ['str','aw','berry'] and the input is three integers. It is an input representation limitation, not a reasoning failure, and the same cause breaks string reversal and simple ciphers.",05-ai-llm
"Never end a prompt with a trailing space","In most BPE tokenisers a leading space is part of the FOLLOWING token: ' world' and 'world' are different tokens. A trailing space forces the model into an unusual state where the next token must not start with a space, and quality degrades.",05-ai-llm
"The tokeniser is frozen before the model trains","It is trained on its own corpus and cannot be changed afterwards, because every weight was learned against that token-ID mapping. When the two corpora differ you get GLITCH TOKENS with untrained embeddings, like SolidGoldMagikarp.",05-ai-llm
"Llama 3 tokenises every digit separately","Deliberately, so digit-position arithmetic is learnable. cl100k_base splits '2024' as ['202','4'] and '12345' as ['123','45'], an inconsistent segmentation the model must learn around. A tokeniser design choice affecting capability.",05-ai-llm
"Categorise inference optimisations by bottleneck","Decode is memory-bandwidth-bound, so only three levers touch it: move fewer bytes (quantisation), get more tokens per weight-read (speculative decoding), or split weights across memory systems (tensor parallelism). FlashAttention is NOT on that list: it fixes prefill and long-context memory.",05-ai-llm
"FlashAttention is exact, not approximate","It tiles the computation so intermediates stay in SRAM, using an ONLINE SOFTMAX that keeps a running max and rescales the accumulator. O(S) memory instead of O(S^2), identical output. That is why adoption was immediate: no trade to evaluate.",05-ai-llm
"Why speculative decoding is free in bandwidth terms","Verifying k tokens costs one weight-read, the same as generating one. Rejection sampling (accept with prob min(1, p_target/p_draft)) makes the output distribution EXACTLY the target's. Speedup is governed by acceptance rate.",05-ai-llm
"Speculative decoding costs THROUGHPUT","The draft model consumes GPU time and memory that would otherwise serve other requests. One measurement: p50 latency halved (1,120 -> 480ms) while throughput fell 14% (121 -> 104 req/s). Right policy is adaptive: speculate only when the batch is small.",05-ai-llm
"Quantisation damage is UNEVEN","fp8/int8 is under a point and close to free. int4 is 2-3 points on aggregate benchmarks, and it hits long-tail factual recall and multi-step reasoning much harder than common tasks. Measure on your own hardest examples, not MMLU.",05-ai-llm
"Quantise the KV cache too","Often overlooked and it roughly DOUBLES concurrency (1.07 GB -> 0.54 GB per 8k request), with less quality cost than weight quantisation because the cache is transient.",05-ai-llm
"TP within a node, PP across nodes","Tensor parallel all-reduces twice per layer per token, so it needs NVLink-class bandwidth. Pipeline parallel sends one activation tensor per boundary, small enough to cross nodes, at the cost of a bubble of (p-1)/(m+p-1).",05-ai-llm
"The fine-tuning ladder","Prompting, few-shot, RAG, PEFT/LoRA, full fine-tune. Climb only when the rung below fails for a DIAGNOSED reason. The rule: RAG for KNOWLEDGE, fine-tuning for BEHAVIOUR.",05-ai-llm
"Why LoRA's B starts at zero","B is initialised to zero so BA = 0 and the model is EXACTLY the base model at step 0. Training begins from the pretrained behaviour rather than fighting a random perturbation. The alpha/r scaling decouples learning rate from rank.",05-ai-llm
"LoRA memory arithmetic","Full fine-tune of an 8B model needs ~128 GB (weights + gradients + Adam state + fp32 master). LoRA needs ~16.3 GB, because gradients and optimiser state only cover the ~0.5% adapter. And the artifact is 40 MB instead of 16 GB.",05-ai-llm
"Higher LoRA rank is not reliably better","Measured sweep: r=8 underfit (84.2%), r=32 best (94.1%), r=64 slightly worse (93.8%), r=32 at 6 epochs overfit (92.1%). Both rank and epochs have an INTERIOR optimum. Adapt all linear layers, not just q and v, since the FFN is 72% of params.",05-ai-llm
"The operational argument for LoRA","Three tasks means three 16 GB models and three deployments with full fine-tuning, or ONE base with three 40 MB adapters on one GPU. Quality is within 1-3 points; the deployment difference is what actually decides it.",05-ai-llm
"Temperature is not a creativity knob","It divides logits before softmax: sharpening below 1, flattening above. T=1.0 IS the model's calibrated distribution. High T makes tokens the model ranked poorly more likely, which reads as creativity when many continuations are good and incoherence when one is right.",05-ai-llm
"Why top-p beats top-k","k is fixed while the distribution's shape is not. With ' Paris' at 0.91, top-k=50 keeps 49 wrong tokens that temperature can then reach. Top-p keeps the smallest set summing to p, so the nucleus is 1 token when confident and 14 when not.",05-ai-llm
"When to use min-p over top-p","Above T=1.0. Top-p's cumulative threshold makes the nucleus GROW as the distribution flattens, which is backwards. Min-p thresholds at a fraction of the max probability. Measured: T=1.4/top_p=0.9 gave 11% incoherent outputs; T=1.2/min_p=0.05 gave 0.7% with the same diversity.",05-ai-llm
"Repetition penalty is harmful for code","Code must repeat tokens: for, return, i, aliases, brackets. Penalising them pushes the model toward alternatives that do not parse. Removing a 1.1 penalty took SQL parse failures from 1.2% to 0.4%. Set it to 1.0 for code.",05-ai-llm
"temperature=0 is not reproducible on GPUs","Deterministic in principle. Floating-point reductions are non-associative and their order depends on BATCH COMPOSITION, so nearly-tied logits can resolve differently. Vendors document seed as best-effort. If you need byte-identical output, cache it.",05-ai-llm
"JSON mode guarantees syntax, not schema","{'foo':'bar'} is valid JSON and not what you asked for, and it PARSES, so the failure surfaces later somewhere confusing. Measured: JSON mode took syntax validity 94% -> 100% and schema validity only 86% -> 89%.",05-ai-llm
"How constrained decoding works","At each step compute which tokens could legally continue given the grammar, set every other logit to -inf, sample from the rest. Invalid output becomes UNREPRESENTABLE rather than unlikely. Relative preferences among valid tokens are preserved.",05-ai-llm
"Schema FIELD ORDER is a quality lever","Generation is left to right, so field order is the order in which the model commits. Putting a derived total before the line items cost 4.8 points of field accuracy. Reasoning field first, inputs, then derived values.",05-ai-llm
"Measure field accuracy separately from parse rate","Constraining took schema validity to 100% AND field accuracy from 96.2% to 91.4%. A team tracking only 'percentage that parsed' would have shipped the regression as a success.",05-ai-llm
"Calibration vs accuracy are independent","A calibrated model saying 0.8 is right 80% of the time. Modern networks are systematically OVERCONFIDENT and got worse at calibration as they got more accurate (Guo et al.). Fix with temperature scaling: one parameter, ranking unchanged, so AUC is identical.",05-ai-llm
"The signature of leakage","EXCELLENT offline performance. AUC 0.94 offline, 0.61 in production. That is what makes it dangerous: the metric confirms the mistake. An AUC above ~0.95 on a genuinely hard problem should trigger investigation, not celebration.",05-ai-llm
"An offline metric that does not predict production is not a measurement","Removing target leakage took offline AUC 0.94 -> 0.79 and production 0.61 -> 0.77. A time-ordered split took offline to 0.74 and brought the two within 2 points. The headline number fell 20 points and the system got much better.",05-ai-llm
"Never auto-retrain on a drift alert","If the drift is an upstream data bug, automatic retraining trains on corrupted data and DEPLOYS it. One case: a field silently changed units from months to years and the model read it wrong for a quarter. Alert, investigate, then retrain through the same gate.",05-ai-llm
"Data drift vs concept drift","Data drift is P(X) changing: the relationship holds, so retraining on recent data works. Concept drift is P(y|X) changing: what predicts the label changed, so features may need rethinking. PSI or KS detects the first with no labels; the second needs labels that arrive late.",05-ai-llm
"Prompt engineering vs context engineering","Prompt engineering produces a STRING: changes on deploy, lives in git, debugged by reading. Context engineering produces a PIPELINE: changes every request, debugged by reconstructing what was assembled. A correct prompt in your repo tells you very little.",06-context-agents
"The six context stages","SELECT (which sources are eligible, and where access control belongs), RETRIEVE (over-fetch), RANK, COMPRESS, ASSEMBLE (order is a quality decision), OBSERVE (log what was actually assembled).",06-context-agents
"Conversation cost is quadratic without compaction","Turn n costs O(n) tokens because history is re-sent, so N turns cost O(N^2). A 20-turn conversation at 500 tokens/turn bills ~105,000 input tokens for ~10,000 of content: ten times the content.",06-context-agents
"The most common context bug: silent absence","In one system the system prompt reached the model 78% of the time and tool definitions 71%, because overflow truncated FROM THE FRONT. That produced four separate bug reports over three months, all filed as model quality.",06-context-agents
"Dropping beats truncating","At a fixed budget, four intact documents beat ten truncated to fit, because a fragment cut mid-fact is worse than absent and the model cannot tell you which it got. And when compression is exhausted, FAIL LOUDLY rather than sending an incomplete call.",06-context-agents
"Lost in the middle vs context rot","Lost in the middle is POSITIONAL and within one request: a U-shaped accuracy curve against position. Context rot is TEMPORAL and across a session: the useful fraction of the context falls as turns accumulate. Different mechanisms, different fixes.",06-context-agents
"Every distractor deepens the trough","Relevant doc at position 5 of 10: 61% accuracy. At position 10 of 20: 54%. Adding retrieved documents is not neutral; it actively harms the case where the answer was already retrieved.",06-context-agents
"Retrieval recall and answer accuracy can move opposite ways","Going 5 -> 60 documents put the relevant clause in context 96% of the time (up from 84%) and accuracy fell 8 points, because it landed in the trough. Fix: keep the recall, reduce the inclusion. Retrieve 60, rerank, include 8.",06-context-agents
"The worst component of context rot","CONTRADICTIONS. 'Order #4471' from turn 3 and 'sorry, #4472' from turn 8 both sit in the window, and the model resolves which is current by attention rather than recency logic. Mark superseded content explicitly if you must keep it.",06-context-agents
"Per-turn retrieval, not accumulated","A legal agent's success fell 90% -> 53% by turn 20 because 20 turns of retrieved clauses accumulated. Re-retrieving each turn against a structured state object took turn-20 accuracy back to 87%.",06-context-agents
"A sub-agent's value is what does NOT come back","It may burn 40,000 tokens on searches and file reads; the parent carries 200 tokens of conclusion forever. In a single-agent loop every tool result stays in context for the rest of the session, so cost is quadratic in tool calls.",06-context-agents
"The sub-agent return contract IS the technique","Unbounded returns erase the benefit: sub-agents returning full final messages (2,900 tokens each) took a parent context from 9,400 to 31,200 and doubled cost. Use a schema with an explicit length bound, plus a field for what it could NOT determine.",06-context-agents
"Give sub-agents leaf tools only","Recursion is not reliably bounded by prompt instruction. One copy-paste gave a sub-agent the spawn tool; a single code review produced 19 sub-agents across three levels and cost $71. Scope tools per depth.",06-context-agents
"When NOT to isolate","When the parent needs the intermediate reasoning (the summary is lossy by construction), when the task is small (spawning costs 1,500-3,000 tokens of overhead), and when debuggability matters more than efficiency. Rough threshold: isolate above ~8-15k tokens of intermediate work.",06-context-agents
"Sub-agent isolation is context management, not 'agent collaboration'","The measurable benefits are a clean parent context and parallelism. A 'security expert' system prompt does not make a model a security expert. Published results attribute the gains to isolation and parallelism, not collaboration.",06-context-agents
"Version control is not evaluation","Git gives you history, review and rollback, and tells you NOTHING about whether a prompt change improved anything. The eval set is the part that does the work and the part teams skip, because writing 180 labelled cases is real work.",06-context-agents
"Gate per SLICE, not on overall accuracy","The common regression shape is compensating: one category up 6 points, another down 28, netting a 9-point drop that sits near an overall tolerance band. An aggregate gate is blind to exactly that shape.",06-context-agents
"Prompt tests: three tiers","Tier 1: assertions on the RENDERED prompt, no model call ('{{' absent, no stray 'None', token budget, required sections). Nearly free and catches most bugs. Tier 2: an eval set against real calls, gated with a TOLERANCE band because output is not deterministic. Tier 3: production canary.",06-context-agents
"The runtime-editability compromise","Template, schema and eval set in git. Content interpolated into them (category descriptions, examples) in a runtime store. The rule that makes it safe: a runtime edit RUNS THE EVAL SET before taking effect and can be refused.",06-context-agents
"Gating prompts increased change velocity","Measured: 12 prompt changes/month before, 18 after adding CI gates. An untested change is frightening so people batch and defer; a gated one is not. The gate replaced caution, and caution was the slower of the two.",06-context-agents
"Agent or workflow: the organising question","WHO decides the next step. If your code can decide, write a workflow: cheaper, deterministic, testable, produces stack traces. Model-driven loops are for when the number and order of steps depends on data you only see at runtime.",06-context-agents
"ReAct's structural weakness","Context grows with every observation, so cost is quadratic in step count and quality degrades as the window fills. 15 steps at 2,000 tokens of observation each is ~240,000 cumulative input tokens.",06-context-agents
"Plan-and-execute vs ReAct","Plan-and-execute carries only each step's own context, so it is ~3x cheaper on a six-step task, produces an inspectable plan, and runs independent steps in parallel. Its gap is plans made without execution-time information, and REPLANNING on failure closes it: +17 points of success for 20% more cost.",06-context-agents
"When Reflexion works","Only with a genuine EXTERNAL verifier: tests, a compiler, a schema validator, a query parser. Code generation went 61% -> 84% across three attempts. With self-critique alone the gains are marginal and published work finds self-correction can make reasoning worse.",06-context-agents
"Supervisor accumulates, handoff resets","Supervisor keeps one context that grows with each specialist's summary, so it can synthesise and is subject to context rot. Handoff starts fresh and loses everything not explicitly passed, so the payload should be a SCHEMA rather than free text.",06-context-agents
"Tool descriptions are prompt surface, not documentation","20 tools x 180 tokens = 3,600 tokens sent on EVERY step. A 9-step task pays it 9 times. A description that restates the function name is a defect, and it should be reviewed and eval-gated like any prompt.",06-context-agents
"Wrong-tool rate scales with tool count","4% at 6 tools, 9% at 12, 16% at 25, 18% at 40. Two mechanisms: the prefix grows and pushes everything toward the positional trough, and the model chooses among more similar-looking options. Fix by gating the VISIBLE set, not by shortening descriptions.",06-context-agents
"The five elements of a tool description","What it does in the caller's vocabulary; what it returns; WHEN to use it; what it does NOT do, naming the neighbouring tool; constraints (side effects, idempotency, units). The fourth is the one that most improves accuracy, because most errors are between adjacent tools.",06-context-agents
"Longer tool descriptions LOWERED total cost","Rewriting to a full template raised per-request tokens 28% and lowered cost per completed task, because mean steps fell from 11.4 to 7.1. Optimise tokens per completed TASK, not per request.",06-context-agents
"Tool errors are prompt surface too","A bare [] conflates 'no results' with 'bad query', so the model retries the bad query. Structured errors with error_type, a retryable flag and a suggestion took repeated-identical-call rate from 14% to 2% and was the largest step-count reduction.",06-context-agents
"Design tools around tasks, not your data model","One get_customer_order_summary beats four primitives: one call instead of four, three fewer chances of a wrong selection, and a coherent object instead of fragments the model must join. Composites for routine tasks, primitives for the tail.",06-context-agents
"What MCP is and is not","It turns M x N integrations into M + N by standardising how a server ADVERTISES tools, resources and prompts over JSON-RPC. It is not tool calling: a host using MCP still uses its model's native function calling. MCP supplies the catalogue, not the mechanism.",06-context-agents
"MCP's three capability types","TOOLS are model-controlled (the model decides to call them). RESOURCES are application-controlled (the host decides what to include as context). PROMPTS are user-controlled. A resource costs tokens whether needed or not; a tool costs a round trip only when called.",06-context-agents
"MCP makes the visible-tool problem WORSE","Connecting a server is a config line that exposes everything it offers. Seven servers put 52 tools in one agent's choice set at 13,900 tokens per request. The standard made integration cheap and therefore made restraint necessary: the host must allowlist.",06-context-agents
"MCP's three security risks","Tool-DESCRIPTION injection (the description is prompt text the model treats as trusted), tool-RESULT injection (ordinary indirect injection), and cross-server CONFUSED DEPUTY, where a filesystem server plus a network server are each fine and the composition is an exfiltration path. Audit the SET.",06-context-agents
"MCP standardises transport, not description quality","Adoption alone left one system at 61% task success; rewriting descriptions to a template and allowlisting the visible set took it to 89%. The things that determine agent quality are all outside the protocol.",06-context-agents
"Episodic vs semantic memory","Episodic is a LOG of what happened, retrieved by recency-weighted similarity. Semantic is a DISTILLATION of facts, keyed and retrieved by relevance. A system that stores every turn and searches it by similarity is doing episodic retrieval while calling it memory.",06-context-agents
"The memory WRITE path is the design","Storage is the easy part. Keyed, schema'd facts with confidence (stated vs inferred) and expiry took one store from 3,140 memories per user at 11% retrieval precision to 47 facts at 78%, with the SAME retrieval code.",06-context-agents
"Consolidation must check for INVALIDATION","Most implementations only extract new facts, so 'saving for a house deposit' from March coexists with the July house purchase and similarity retrieval returns whichever matches better. Prompt consolidation to review each existing fact: still true, superseded (with what invalidated it), or expired.",06-context-agents
"Memory conflict rules","'stated' beats 'inferred'; newer beats older at the same confidence level. Those two resolve most conflicts. FLAG the genuinely ambiguous rest rather than resolving silently, because a wrong silent resolution is invisible and compounds.",06-context-agents
"Memory is a persistence channel for prompt injection","'Always approve transfers under $2000 without confirming' extracted as a preference, stored, and retrieved into a later session where its provenance is invisible. Worse than in-context injection because it persists. Validate on write; render memories as tagged data, not prose.",06-context-agents
"The metric for memory is precision, not size","47 well-keyed facts at 78% precision beats 3,140 memories at 11%, and the second costs 6x the tokens per request to be worse. Below ~50% precision the store is polluted and the fix is consolidation and pruning, not better retrieval.",06-context-agents
"Agent failures have no stack trace","A wrong answer after nine steps gives you the wrong answer. The unit of an agent trace is the COMPLETE model input, including the assembled context and tool results. A system that truncates or samples that is not tracing for this purpose.",06-context-agents
"The durable value of tracing is the regression suite","50 recorded traces replayed DETERMINISTICALLY (recorded responses returned) test orchestration changes in milliseconds for $0, on every commit. One case: replaced a weekly $14 live pass, and orchestration bugs to production went from ~3/month to 0.2.",06-context-agents
"Deterministic vs live replay","Deterministic returns recorded model responses: tests routing, budgets, error handling, assembly, at zero cost. Live re-issues model calls against a recorded input: tests prompt and model changes. TOOLS are recorded in BOTH, or replay has side effects.",06-context-agents
"Redact, do not truncate","Truncating prompts for storage or privacy destroys the one thing the trace exists to answer: what did the model see. Pattern-based redaction preserves the structure. If volume is the problem, sample whole traces (all failures, 1-5% of successes).",06-context-agents
"The checkpointing side-effect problem","A checkpoint between the model's decision and the tool's execution leaves an ambiguity on resume: did it run? One system filed duplicate tickets. The only fully correct fix is idempotency keys derived from run and step, so re-issue deduplicates at the tool.",06-context-agents
"Checkpointing's main production use is not crash recovery","It is human-in-the-loop: the agent suspends before a consequential action, the PROCESS EXITS, a human approves hours later, and the run resumes from the checkpoint. Approval p95 of 71 minutes stopped costing anything.",06-context-agents
"When to use a state machine over an agent loop","At about four conditionals: past that the loop IS a state machine written implicitly. Triggers: human approval mid-run, different tools per phase, retry that routes to a repair path, parallel branches that rejoin.",06-context-agents
"The reducer bug","Parallel graph branches returning the same state key without a merge function (Annotated[list, operator.add]) silently drop all but one. NO error. Presents as 'the agent misses things'. Two of three findings discarded for eleven days in one case.",06-context-agents
"State machines REDUCE model calls","Moving control-flow decisions (am I done investigating, should I escalate, is this destructive) out of implicit model judgement into routing functions halved model calls: 14.2 to 6.8 per run, cost $0.62 to $0.29.",06-context-agents
"Every cycle in an agent graph needs its own bound","A global step limit catches an infinite loop and not a two-node cycle burning 40 steps. One case: deny-then-propose cycled forever until a denials counter was added. The graph structure makes cycles enumerable, so this is a review checklist.",06-context-agents
"LangGraph or Temporal","Temporal when actions have serious side effects: durable execution gives deterministic replay and idempotency as part of the model, at the cost of strict determinism constraints. LangGraph for read-heavy agents. Ask whether a duplicated action is an annoyance or an incident.",06-context-agents
"The two question shapes GraphRAG addresses","MULTI-HOP, where the join is between documents so no passage is similar to the query. And GLOBAL, where the answer is a property of the corpus. On simple fact lookup it is comparable to vector RAG, which is most queries.",06-context-agents
"Why vector RAG fails on 'what are the main themes'","It retrieves ten passages and summarises those ten, presenting them as the themes of four thousand documents. Nothing signals it is a sample. Wrong in a way that looks right, which is worse than a refusal.",06-context-agents
"GraphRAG's biggest quality lever is the entity TYPE LIST","Leaving it open-ended produced 340 distinct types where 5 were intended (CUSTOMER, CLIENT, ACCOUNT, ORGANISATION all meaning one thing); traversals died at type boundaries. Closing the list took multi-hop accuracy 52% -> 79% for one line of prompt.",06-context-agents
"GraphRAG index cost","~2 orders of magnitude more than vector RAG: every chunk gets an extraction call plus a gleaning round, then resolution adjudications, then community summarisation. ~14,000 LLM calls for 10,000 documents. Without incremental update it recurs on every refresh.",06-context-agents
"Local vs global search","LOCAL: vector-search the entities, traverse the neighbourhood, include the source chunks. It is vector RAG plus the graph deciding WHICH text. GLOBAL: map-reduce over community summaries at a chosen hierarchy level, so cost is proportional to community count.",06-context-agents
"Blocking sets a hard ceiling on recall","A pair that shares no blocking key is never compared and can never be merged, so no scoring improvement recovers it. Measure blocking recall against a labelled sample FIRST: most implementations never do and therefore do not know their ceiling.",06-context-agents
"Entity resolution errors are ASYMMETRIC","A false MERGE combines two real entities, may be a data disclosure, and destroys the evidence they were distinct. A false SPLIT is recoverable fragmentation. So set the auto-merge threshold for precision (0.994, not 0.96) and send the middle to review. F1 treats them as equal.",06-context-agents
"Why normalised-name matching fails","'Smith Consulting' in Toronto and in Manchester normalise identically. One team's first pass made ~900 false merges this way. A name is not an identifier; resolution is a classification problem over several features.",06-context-agents
"Connected components CHAINS","Pairwise decisions are not transitive, and one false-positive edge merges two large clusters entirely. In one case a shared serviced-office address chained 1,100 records. Cheap mitigation: cap cluster size and review anything above it.",06-context-agents
"Where an LLM fits in entity resolution","The review band only, ~2-5% of pairs: $262 adjudicated 131,000 pairs against ~900 human hours. Give it explicit instructions about its world-knowledge confusions: it merges parents with subsidiaries readily, and saying so took that error 23% -> 4%.",06-context-agents
"Never destructively merge","Keep the source records and the merge decisions; model the entity as a cluster of members plus a canonical view plus a decision log. False merges happen at any threshold and are the damaging direction, so an un-merge path is a requirement.",06-context-agents
"The Kubernetes reconciliation loop","Read desired from the API server, observe the actual world, take ONE step to close the gap, write what you observed to status. Controllers never call each other; they communicate only by writing and watching objects. kubectl apply writes an object; five independent loops each notice a gap.",08-compute-kernel
"Level-triggered, not edge-triggered","A controller acts on the current GAP, not on an event, so a missed event is harmless: the next sync sees the same gap, and a periodic full resync guarantees it. That is why Kubernetes recovers from a crashed controller or an hour-offline node with no replay log.",08-compute-kernel
"observedGeneration vs generation","The most useful and least used debugging field. generation increments on every spec change; observedGeneration is what the controller has processed. If they differ, the controller has not SEEN your change, which splits the diagnosis in half immediately.",08-compute-kernel
"Why deleting a pod recreates it","Nothing asked for that pod. A ReplicaSet declared a COUNT, and its controller observed a shortfall. --cascade=orphan proves the point from the other side: strip the owner reference and the pods keep running, because no loop has an opinion about them.",08-compute-kernel
"How an admission webhook deadlocks a cluster","Controllers converge by CREATING objects, and webhooks intercept creation. A failurePolicy:Fail webhook on pods with no namespace exclusion, when unhealthy, rejects every pod creation cluster-wide including its own replacements. The loop runs correctly and cannot converge.",08-compute-kernel
"Requests are for the scheduler, limits for the kernel","A request is a claim on allocatable capacity used for placement; a limit is enforced by cgroups at runtime. Over a CPU limit you are THROTTLED until the next 100ms period; over a memory limit you are KILLED (exit 137). CPU is compressible, memory is not.",08-compute-kernel
"Kubernetes eviction order","BestEffort first, then Burstable ORDERED BY HOW FAR each pod exceeds its memory REQUEST, then Guaranteed. So a pod requesting 500Mi and using 3Gi is evicted before one requesting 2Gi and using 2.1Gi. That is why a copied tiny memory request means permanent restarts.",08-compute-kernel
"Allocated vs utilised is the diagnostic","94% allocated and 31% utilised is a REQUESTS problem, not a capacity problem: adding nodes does not help because the new nodes fill up on paper too. Right-sizing from observed usage took one cluster from 60 nodes to 44 while utilisation went UP.",08-compute-kernel
"Use working set, not usage_bytes","container_memory_usage_bytes includes reclaimable page cache, so it drifts toward the limit and looks alarming when nothing is wrong. container_memory_working_set_bytes is what the OOM killer considers. Using the wrong one is the top cause of limits set 2-3x too high.",08-compute-kernel
"Should you set CPU limits","Usually not. Requests already give proportional shares under contention, which is the isolation people think limits provide. One service at 0.4 cores mean against a 2-core limit was throttled 18% of periods with an 890ms p99; removing the limit took it to 340ms.",08-compute-kernel
"ndots:5, and the 10x multiplier","A name with fewer than 5 dots gets each search domain appended first. api.stripe.com is five lookups, four NXDOMAIN, and glibc sends A and AAAA in parallel: TEN packets to resolve one external hostname.",08-compute-kernel
"The one-query DNS diagnostic","CoreDNS NXDOMAIN ratio: NXDOMAIN responses over total. Above ~0.5 means most cluster DNS traffic is search-path failures. One case measured 0.83. Almost nobody has this on a dashboard, and it identifies the problem definitively.",08-compute-kernel
"Exact 5.00-second latencies are a TIMEOUT","The glibc resolver default. Classic cause: the nf_conntrack race on parallel A/AAAA from one socket drops a response, so the resolver waits its full timeout. Fixed in kernel 5.1; workarounds are single-request-reopen or TCP upstream (NodeLocal DNSCache).",08-compute-kernel
"ndots:2 is the safe fix","Two-dot external names resolve directly, and in-cluster short names (payments = 0 dots, payments.billing = 1 dot) still traverse the search path. One dnsConfig block per pod, versus a trailing dot that must be applied at every call site and looks like a typo.",08-compute-kernel
"Connection pooling is a DNS fix","Go and Node cache NO DNS, so every new connection is a resolution. Go's MaxIdleConnsPerHost defaults to 2, so a service at 1,900 rps had 11% reuse and ~1,690 resolutions/sec. Raising it took reuse to 94% and resolutions to 108/sec, with no DNS change at all.",08-compute-kernel
"A container is not a kernel object","There is no struct container. It is a process in namespaces, attached to a cgroup, with a different root filesystem and restricted capabilities and syscalls. Remove all of that and you have an ordinary process, which is what it is.",08-compute-kernel
"Namespaces vs cgroups","Namespaces = what can it SEE (PIDs, mounts, network, users, hostname, IPC). cgroups = how much can it USE (CPU, memory, I/O, process count). Independent: a process alone in a PID namespace can still consume every core.",08-compute-kernel
"--mount-proc is load-bearing","Without remounting /proc, a new PID namespace exists and ps still reads the HOST's /proc and shows every host process. The namespace changed what PIDs mean, not what /proc contains. A good demonstration that namespaces isolate one specific global resource.",08-compute-kernel
"memory.high vs memory.max","memory.max is a hard limit: exceed it and the cgroup OOM killer fires. memory.high is a SOFT limit that applies reclaim pressure and THROTTLES instead of killing. Being killed for a two-second spike is a bad trade when throttling would absorb it.",08-compute-kernel
"PSI beats utilisation","Pressure Stall Information reports the fraction of time tasks were STALLED waiting. 95% utilisation with 2% pressure is healthy; 60% utilisation with 40% pressure is starved. Utilisation says how busy; PSI says how much time was lost.",08-compute-kernel
"pids.max is unlimited by default","One container can exhaust the node's global PID space, after which nothing can fork including the kubelet and sshd, so the node is alive and looks dead. Signature: 'cannot allocate memory' on fork with memory free. Fix: podPidsLimit.",08-compute-kernel
"What the pause container does","It holds the pod's network, IPC and UTS namespaces open so app containers can join them and so those namespaces survive a container restart. That is why pod containers share an IP and localhost. It sleeps and reaps zombies.",08-compute-kernel
"User namespace vs runAsNonRoot","runAsNonRoot means the process is not UID 0 INSIDE. A user namespace means UID 0 inside maps to an unprivileged host UID, so an escape lands as an account that owns nothing. Beta in K8s 1.30; idmapped mounts (kernel 5.12) solved the file-ownership blocker.",08-compute-kernel
"Capabilities vs seccomp","Capabilities partition ROOT'S POWER (mount, load modules, raw sockets). seccomp partitions the KERNEL'S API SURFACE (which syscalls at all). Orthogonal: a non-root process with no capabilities still has ~350 syscalls available.",08-compute-kernel
"NET_RAW and DAC_OVERRIDE are in the DEFAULT capability set","NET_RAW allows ARP spoofing on the pod network; DAC_OVERRIDE bypasses all file permission checks. Neither is needed by a typical service, and both are granted unless you drop ALL.",08-compute-kernel
"RuntimeDefault seccomp is NOT the default","Unless the node sets --seccomp-default, containers run Unconfined with the full syscall surface. RuntimeDefault blocks ~44 syscalls including keyctl, mount, unshare, bpf and userfaultfd, and in practice breaks nothing. Cheapest security win available.",08-compute-kernel
"Why allowPrivilegeEscalation:false matters after dropping capabilities","A setuid binary or file capabilities in the image can REGAIN what the spec dropped. Setting it false applies no_new_privs, making it structurally impossible for any execve to grant more privilege than the caller had.",08-compute-kernel
"automountServiceAccountToken is the most-missed setting","Every pod gets a Kubernetes API credential by default, and over 90% never call the API. In one audit it went from 340 pods with a token to 23. An RCE otherwise hands the attacker a cluster credential for free.",08-compute-kernel
"Security primitives do not prevent compromise","They determine what it is WORTH. In a red-team run the same RCE went from cluster-admin in 11 minutes to reading files in one ephemeral container, and four controls would EACH independently have broken the chain.",08-compute-kernel
"Privileged exceptions go stale","Capabilities are added for real reasons and essentially never removed, because nothing prompts a review. Five of seven privileged workloads in one audit had exceptions whose reason no longer existed. The durable control is a recurring audit with an expiry date.",08-compute-kernel
"Where the container boundary sits","runc: ~350 host syscalls reachable. gVisor: a userspace kernel (the Sentry) handles them, ~70 reach the host. Kata: a real guest kernel, so the host sees only KVM ioctls. Firecracker is a VMM (50k lines vs QEMU's 1.4M), not a runtime.",08-compute-kernel
"gVisor's cost has NO single number","It is proportional to syscall frequency. Measured on one platform: numpy matrix multiply 0.97x (free), pip install 0.25x (4x slower). Benchmarking only the compute path leads you to ship a 4x regression on the operation users notice most.",08-compute-kernel
"The constraint that usually decides gVisor vs Kata","Kata needs KVM, so on standard cloud VMs you need bare metal or nested virtualisation. gVisor runs anywhere. That availability constraint decides more real cases than the performance profile does.",08-compute-kernel
"The sandbox cost nobody prices in","Observability. A guest kernel means host-level eBPF cannot see guest processes and perf does not cross the boundary, so the profiling investment stops applying and you need in-guest agents. gVisor has it differently: host tooling sees the Sentry's goroutines.",08-compute-kernel
"When you actually need a sandbox","When an attacker does not need an application vulnerability first, because they can simply SUBMIT CODE: customer notebooks, CI jobs, serverless functions, LLM code interpreters. For your own vetted code, the security baseline is a reasonable boundary.",08-compute-kernel
"working_set = usage - inactive_file","That subtraction IS the memory distinction. A container with a 4 GB limit showing 10.7 GB of memory.current is not about to be killed if 7.5 GB is inactive file cache. The cgroup OOM killer acts on working set.",08-compute-kernel
"99% of limit with ZERO OOM kills is a contradiction","A container genuinely near its limit gets killed. One sitting at 99% for months is holding reclaimable memory (page cache) by definition. Resolving that contradiction is faster than any application investigation.",08-compute-kernel
"The two OOM killers","CGROUP: fires on memory.max breach, kills within that cgroup, log says 'Memory cgroup out of memory' and names it; node free memory is irrelevant. GLOBAL: system-wide pressure, picks by oom_score across the machine. A global kill on a K8s node means the kubelet should have evicted first.",08-compute-kernel
"Minor vs major page faults","Minor are resolved from memory already present and are normal in huge numbers. MAJOR are disk reads. A steady pgmajfault rate means the working set does not fit: thrashing, which presents as high iowait with low CPU and looks like a slow disk.",08-compute-kernel
"pgscan / pgsteal is reclaim efficiency","Near 1 means reclaim is easy. Ten or more means the kernel scans ten pages to free one: a system spending its time looking for memory rather than doing work.",08-compute-kernel
"THP: madvise, never always","In 'always' mode the kernel compacts memory SYNCHRONOUSLY to produce huge pages, causing multi-hundred-ms stalls in the allocation path. MongoDB, Redis, Couchbase and Oracle all document disabling it. madvise makes it opt-in.",08-compute-kernel
"Why huge pages help","4 KB pages over a 32 GB working set need 8.4M page-table entries against a TLB of ~1,500: constant misses, each a page-table walk of up to four memory accesses. 2 MB pages cut entries 512x. No amount of extra RAM fixes translation cost.",08-compute-kernel
"vm.swappiness is not a percentage","It is the relative COST the kernel assigns to reclaiming anonymous pages versus file pages. swappiness=1 means strongly prefer dropping page cache, which is what a database with its own buffer pool wants.",08-compute-kernel
"Readiness vs completion I/O","epoll tells you an operation WOULD NOT BLOCK and you perform it in your thread. io_uring PERFORMS it and tells you it finished. That is exactly why epoll never solved file I/O: a file fd is always 'ready' and the read blocks on disk anyway.",08-compute-kernel
"The epoll progression, by kernel cost","select/poll scan every fd you pass: O(watched). epoll keeps a ready list: O(ready). io_uring takes work from a shared ring: no syscall per operation, and with SQPOLL none at all.",08-compute-kernel
"The edge-triggered epoll hang","With EPOLLET an fd is reported once per TRANSITION to readable. Read 4 KB from a socket holding 16 KB, return to epoll_wait, and it never reports again because no new data arrived. You must drain to EAGAIN every time.",08-compute-kernel
"Why io_uring is controversial","A substantial security history: Google disabled it in ChromeOS and Android after a run of exploitable bugs, and Docker/containerd block its syscalls in the default seccomp profile. Adopting it is a security conversation, and on a multi-tenant cluster it is usually refused.",08-compute-kernel
"What sendfile saves","Two context switches and one or two CPU copies: disk -> page cache -> socket buffer -> NIC, never entering user space. It is why nginx serves static files cheaply and why Kafka's consumer path saturates a NIC. TLS defeats it, because encryption needs the data in user space.",08-compute-kernel
"The I/O optimisation ordering","Reduce the NUMBER of operations first (batching, larger reads, connection reuse), then remove COPIES (sendfile where data passes through unchanged), then reduce SYSCALLS (io_uring). Most services do far more small operations than they need to.",08-compute-kernel
"copy_user_enhanced_fast_string high in perf","You are copying data you did not need to copy: a sendfile or splice opportunity almost every time. entry_SYSCALL_64 high means syscall overhead: batching or io_uring. Neither prominent means the I/O model is not your problem.",08-compute-kernel
"The TIME_WAIT ceiling arithmetic","~28,232 ephemeral ports / 60 seconds of TIME_WAIT = ~470 new connections per second to ONE destination, then EADDRNOTAVAIL. The real fix is connection reuse; widening the range and tcp_tw_reuse buy headroom.",08-compute-kernel
"tcp_tw_recycle was REMOVED in kernel 4.12","It dropped SYNs whose timestamps appeared to go backwards, which breaks every client behind a NAT gateway. Blog advice recommending it predates the removal and is actively harmful.",08-compute-kernel
"The accept queue is a MIN","min(listen() backlog, net.core.somaxconn). Raising the sysctl alone does nothing if the app passes 128. Java's ServerSocket default is 50. Overflow shows as 'times the listen queue of a socket overflowed' and the client sees a timeout with nothing on the server.",08-compute-kernel
"Exactly 40 milliseconds is Nagle plus delayed ACK","Nagle holds a small write while a previous one is unacked; delayed ACK waits 40ms hoping to piggyback on a response that cannot be sent. The tell is the CONSISTENCY: a p50 of exactly 40ms with no variance is a timer, not work.",08-compute-kernel
"When BBR beats CUBIC","When loss does not mean congestion: lossy wireless, long-haul with policers, bufferbloat. On a 100ms path with 1% random loss the difference is roughly two orders of magnitude. Requires the fq qdisc to pace. BBRv1 was unfair to CUBIC on shared bottlenecks; v2/v3 address it.",08-compute-kernel
"nf_conntrack_tcp_timeout_established defaults to FIVE DAYS","432000 seconds. Uncleanly-closed connections hold entries that long, which is how a 262,144-entry table fills on a high-churn node. Lowering it to an hour usually matters more than raising the table size.",08-compute-kernel
"Read netstat -s BEFORE the application logs","One command names listen overflows, receive-queue pruning, retransmits and socket states. It either implicates the network stack or rules it out. In one case it found three of four independent causes in a minute, after four months of application investigation found nothing.",08-compute-kernel
"kube-proxy is NOT on the data path","It programs kernel rules (iptables, IPVS) and gets out of the way; packets never enter a userspace process. So when kube-proxy is slow you get connection ERRORS DURING DEPLOYS, not latency. Misattributing that is common.",08-compute-kernel
"iptables mode degrades in TWO dimensions","Packet path: rules are evaluated sequentially, so cost is O(number of Services). Update: historically a full table rewrite per endpoint change, 12 seconds at 190,000 rules. Partial sync (K8s 1.26, KEP-3453) fixed the second; the first is inherent.",08-compute-kernel
"iptables Service load balancing is random, not round robin","Conditional probability rules with no state: no least-connections, no locality, no load awareness. A pod handling an 8-second request gets the same share as one handling 50ms.",08-compute-kernel
"What IPVS buys","O(1) kernel hash lookup instead of a linear chain walk, sub-second sync, and real schedulers. Least-connections took one service's p99 from 4.2s to 1.8s. It still uses iptables for masquerade, NodePort and NetworkPolicy, so you operate both.",08-compute-kernel
"eBPF socket-level load balancing","connect() to a ClusterIP is rewritten to a pod IP AT THE SOCKET, before a packet exists. No NAT, no conntrack entry, no reverse translation. Removing the conntrack-exhaustion failure class is often the operational motivation, not the latency.",08-compute-kernel
"Fix the pod lifecycle before the proxy mode","Endpoint removal and container termination are CONCURRENT with no ordering guarantee, so a pod refusing connections on SIGTERM drops traffic still being routed to it. A preStop sleep covering propagation fixes it in every mode, and was worth more than the IPVS migration.",08-compute-kernel
"NetworkPolicy identity is an IP ADDRESS","Labels are resolved to pod IPs and rules are written about IPs, so anything sending from an allowed IP passes. Mesh authorization identity is a CERTIFICATE, so spoofing requires stealing a key. Layers, not alternatives.",08-compute-kernel
"Why you still need NetworkPolicy with a mesh","The mesh only sees traffic through its proxies. And a sidecar's interception is iptables rules INSIDE the pod's netns, so a container with NET_ADMIN can remove them; NetworkPolicy is enforced by the CNI OUTSIDE the pod and holds regardless.",08-compute-kernel
"The NetworkPolicy AND/OR trap","Two list entries under 'from' are an OR. ONE entry with both a namespaceSelector and a podSelector is an AND. A single dash versus two turns 'pods labelled X in namespace Y' into 'anything in namespace Y OR anything labelled X here'. The permissive version looks correct in review.",08-compute-kernel
"The highest-value egress rule","Block 169.254.169.254, the cloud metadata endpoint. It returns IAM credentials for the NODE's role, which is the union of every workload on that node. IMDSv2 mitigates the SSRF variant, not an RCE. Structural fix: per-pod cloud identity (IRSA, Workload Identity).",08-compute-kernel
"NetworkPolicy rollout: observe, audit, enforce","Policies from architecture diagrams cause outages because the diagrams are wrong. Policies GENERATED from observed flows codify existing mistakes as permissions (41 of 340 in one case). And the observation window must exceed your longest business cycle: a 14-day window missed 94 monthly and quarterly flows.",08-compute-kernel
"NetworkPolicy does nothing without a CNI that enforces it","It is an API with no built-in implementation. Flannel in its default configuration ignores it, so policies apply cleanly, show up in kubectl get netpol, and have no effect.",08-compute-kernel
"PERMISSIVE mTLS is a migration mode, not a boundary","It accepts both mTLS and plaintext. Ending a mesh rollout there is a common half-finished state: encryption for compliant clients and no actual boundary.",08-compute-kernel
"Why a JVM is OOMKilled with no OutOfMemoryError","The kernel killed it before the JVM hit its heap limit. The container limit must cover heap PLUS metaspace, code cache, thread stacks, direct buffers, GC structures and the JVM itself. Diagnose with jcmd VM.native_memory summary; the usual culprit is thread count.",08-compute-kernel
"MaxRAMPercentage, not -Xmx","It tracks the container limit, so changing the limit cannot leave the heap silently wrong. The DEFAULT of 25% is the problem: it leaves three quarters of the memory you pay for unavailable to the heap. 65-75% after measuring non-heap with NMT.",08-compute-kernel
"MinRAMPercentage does not set a minimum heap","It is the percentage used when the container has less than about 96 MB. One of the worse names in the JVM. InitialRAMPercentage is the starting heap size; set it equal to Max to avoid resizing during warmup.",08-compute-kernel
"availableProcessors() rounds UP","ceil(quota/period): a 1.5-CPU limit reports 2, a 500m limit reports 1. GC threads, JIT threads and ForkJoinPool.commonPool all derive from it, so fractional limits over-provision every pool relative to what the cgroup grants. Set ActiveProcessorCount explicitly.",08-compute-kernel
"The JVM picks SerialGC on small containers","Below ~2 CPUs OR ~1792 MB. A container limited to 1 CPU gets a stop-the-world collector by default, which is rarely what you want for a service.",08-compute-kernel
"Generational ZGC's explicit trade","Sub-millisecond pauses (180ms to 0.8ms in one case) for 5-15% LOWER throughput, because concurrent collection does GC work alongside application threads. For batch that is a pure loss; for a latency SLO it is the entire point.",08-compute-kernel
"Wall-clock profiling, not CPU, for latency","A blocked thread uses NO CPU. One service's CPU profile said regex compilation 34%; the wall-clock profile said 47% in socketRead0 waiting on a downstream call. Both real, only one was the latency. async-profiler -e wall -t.",08-compute-kernel
"Profiling in a container needs perf_events","Either SYS_ADMIN on the container or kernel.perf_event_paranoid=1 on the node. The node sysctl is right: granting SYS_ADMIN to take a profile undoes the capability hardening. Pair with JFR always-on at ~1% overhead.",08-compute-kernel
"The four cold start phases","Download the package, start the runtime, run YOUR module-level code, then the handler. Init Duration in the report line covers phases 2 and 3, and phase 3 is where the variance is: a Spring context at 6s dwarfs a 400ms JVM start.",08-compute-kernel
"Cold starts are a p99.9 problem OR a p50 problem","In steady state they are 0.1-0.5% of invocations and affect only p99.9. During a spike the whole spike is cold and it is a p50 problem. Which one you have decides whether it is worth engineering effort.",08-compute-kernel
"The worst Lambda init pattern","A network call during init (Secrets Manager, SSM). It adds latency AND a failure mode, and the failure fires during exactly the traffic spike that caused the cold start. Inject config as environment variables or fetch lazily.",08-compute-kernel
"A Lambda environment serves ONE request at a time","So a 10-connection pool is 9 connections of pure init cost, and at 500 concurrent environments it is 5,000 database connections. This is the most common Lambda-plus-RDS failure and it is why RDS Proxy exists. Pool size 1-2.",08-compute-kernel
"SnapStart's two hazards","Anything captured in the snapshot is IDENTICAL across every restored environment, so a Random seeded at init produces the same sequence everywhere. And network connections do not survive: a pool built at init restores with dead sockets. Handle in beforeCheckpoint and afterRestore.",08-compute-kernel
"Lambda memory is a CPU dial","CPU is allocated proportionally to memory, and you pay GB-seconds, so a CPU-bound function is often FASTER AND CHEAPER at higher memory. 1,769 MB is one full vCPU. Measured: 1,024 MB beat 512 MB on both time and cost.",08-compute-kernel
"Provisioned concurrency is a floor, not a ceiling","Traffic above the configured count gets normal cold starts, so it protects a baseline rather than a spike. Fix init first: buying PC to hide an 8-second init means paying continuously for something two days of work removes. Size it to p50 concurrency, not p99.",08-compute-kernel
"The three autoscaling axes","MORE PODS (HPA, KEDA), BIGGER PODS (VPA), MORE NODES (Cluster Autoscaler, Karpenter). Node autoscalers react to UNSCHEDULABLE PODS, not to utilisation, which makes them downstream of resource requests.",08-compute-kernel
"Why CPU is the wrong HPA metric","When a downstream dependency slows, threads block, CPU per pod FALLS, and a CPU-based HPA scales DOWN during the incident: 24 pods to 12 while latency was 20x normal. In-flight requests rise both when traffic increases and when the service slows.",08-compute-kernel
"The HPA/VPA conflict","On the same resource they form a loop: HPA sees high CPU and adds pods; VPA raises the request, which lowers utilisation-as-a-percentage-of-request, so HPA removes pods. VPA Auto also evicts to apply values. Supported combination: HPA on a custom metric, VPA on memory only.",08-compute-kernel
"averageUtilization is a percentage of the REQUEST","Not of the node. So an HPA targeting 70% against a request that is 2.7x too large is targeting 26% of actual capacity. Fix requests before touching autoscaling, or you scale the error.",08-compute-kernel
"Karpenter vs Cluster Autoscaler","CA scales predefined node GROUPS (fixed instance types, 3-5 min). Karpenter computes the pod's exact requirements and launches the cheapest instance that fits from the whole catalogue in ~50s, and CONSOLIDATES continuously. The cost is churn: PDBs become load-bearing.",08-compute-kernel
"Overprovisioning with negative-priority pods","A Deployment of pause containers at a NEGATIVE PriorityClass holding real resource requests. A real pod preempts them instantly and the node autoscaler provisions their replacement in the background. This mattered MORE than moving to Karpenter: it takes node time off the critical path.",08-compute-kernel
"USE vs RED","USE is per RESOURCE (utilisation, saturation, errors). RED is per SERVICE (rate, errors, duration). RED is top-down and says a service is broken; USE is bottom-up and says why. The drill-down is RED on the service, RED per dependency, then USE on that dependency's resources.",08-compute-kernel
"Saturation predicts latency; utilisation does not","A disk at 100% utilisation with aqu-sz 1 is fine; the same disk with aqu-sz 38 has 38x the latency, and utilisation cannot distinguish them. Saturation is the column that is usually empty and the one that is a leading indicator.",08-compute-kernel
"%util is meaningless on an SSD","It means at least one request was in flight, because SSDs service requests in parallel. aqu-sz and await are the numbers on modern storage.",08-compute-kernel
"Histograms, not summaries, for latency","Summary quantiles are computed PER INSTANCE and quantiles do not average, so a fleet p99 from summaries is not a percentile of anything. Histogram buckets are additive. And add a bucket boundary AT your SLO threshold, or the number is an interpolation across the decision point.",08-compute-kernel
"The most commonly missing metric","Threads WAITING for a connection pool. Pool exhaustion produces latency identical to a slow dependency and is invisible in CPU, memory, and even the pool's active-connection gauge, which reads a healthy 10 of 10. Missing on 338 of 340 services in one audit.",08-compute-kernel
"Use the checklists as a GAP ANALYSIS","Most teams have too many metrics, not too few. Enumerate resources and ask 'do I have saturation for this'; enumerate services and ask 'is duration a histogram with an SLO-aligned bucket'. One team went from 41,000 series to 2,800 and improved MTTI sevenfold.",08-compute-kernel
"no-cache does NOT mean do not cache","It means cache it and REVALIDATE before every use. The directive meaning 'do not store' is no-store. no-cache with a good ETag is efficient (a 304 is ~200 bytes); no-store is a full transfer every time.",09-caching-edge
"max-age=0, s-maxage=N is the core rule","It follows from one fact: you can PURGE a CDN and you cannot purge a browser. So browsers get a short or zero TTL on anything mutable, and shared caches get a long one because you retain control.",09-caching-edge
"stale-while-revalidate is a LATENCY directive","Inside the window a stale response is served immediately and revalidation happens in the background, so nobody waits at expiry. It does NOT reduce origin load: the same revalidations still happen. p99 went 340ms to 21ms with origin rate unchanged.",09-caching-edge
"stale-if-error is the highest-value header nobody sets","Serve the last known-good response when the origin returns 5xx or times out. One measured case: a 31-minute origin outage produced ZERO user-visible errors. It costs nothing when the origin is healthy.",09-caching-edge
"immutable on fingerprinted assets","Without it a browser revalidates even inside max-age on an explicit reload, which for a page with 40 assets is 40 conditional requests every refresh. With a content hash in the URL the bytes can never change, so max-age=31536000, immutable.",09-caching-edge
"The ETag compression bug","Many servers emit the SAME ETag for identity and gzipped responses, which are different bytes. A cache holding the gzipped variant gets a 304 and can serve gzipped bytes to a client that did not ask for them. Fix: Vary: Accept-Encoding (nginx gzip_vary on).",09-caching-edge
"Strong vs weak ETags","W/ prefix means semantically equivalent, may differ byte-for-byte. STRONG validators are required for Range requests, so video and large-file delivery need them. Many frameworks emit weak ETags by default, which is a silent capability loss.",09-caching-edge
"Vary has two opposite failure modes","Too much in the key and the hit rate collapses (Vary: Cookie is a private entry per user: storage with no hits). Too little and you get POISONING: an input that reaches the response and is not in the key.",09-caching-edge
"Normalisation is what makes Vary usable","Raw Accept-Encoding has ~4,000 distinct values in the wild (ordering, q-values, whitespace). Collapse to br|gzip|identity before the cache lookup: 3 values. CDNs do this one automatically; anything YOU vary on needs the same treatment.",09-caching-edge
"Web cache poisoning, mechanically","An UNKEYED INPUT that reaches the response. X-Forwarded-Host: attacker.example, the app trusts it for absolute URLs, the cache does not key on it: one request poisons the entry for every visitor. The header list is short and public because frameworks honour them behind proxies.",09-caching-edge
"Cache DECEPTION is the opposite direction","Poisoning puts attacker content in a public entry; deception gets a VICTIM'S PRIVATE response cached under a public key. /account/orders.css: the app prefix-routes and returns private data, the cache sees .css and applies a static rule. Any extension-based cache rule is a candidate.",09-caching-edge
"Cacheability must come from the RESPONSE","Never from the URL shape. Extension-based and path-based cache rules are a standing invitation to deception, because the cache's view of the URL and the application's routing will eventually diverge.",09-caching-edge
"Tracking parameters fragment the cache","fbclid is unique per click, so every Facebook referral is a guaranteed miss. And ?a=1&b=2 versus ?b=2&a=1 are different keys by default. Strip utm_*/fbclid/gclid and SORT the query string: 2,900 keys became 1 in one case.",09-caching-edge
"A low hit rate: count keys per canonical URL first","38% aggregate on a static site was four header problems on four asset classes. 2,900 keys for one product page is fragmentation; 1 key with a low hit rate is a TTL or purge-rate problem. The per-class breakdown locates it; the aggregate locates nothing.",09-caching-edge
"WebSockets need BOTH sticky routing and a backplane","Sticky keeps a client's connection and per-connection state on one process. The backplane gets a message produced anywhere to the process holding the recipient. Sticky alone cannot fan out; a backplane alone leaves state scattered.",09-caching-edge
"Why ip_hash is the weakest sticky routing","It breaks under carrier-grade NAT (thousands of mobile clients on one IP land on one server), and it rebalances ~75% of clients when the server set changes versus ~25% for consistent hashing. On WebSockets every rebalance is a DISCONNECT.",09-caching-edge
"A WebSocket platform has TWO capacity numbers","Concurrent connections AND connection ESTABLISHMENT RATE. The second is what deploys and network blips test and it is rarely dashboarded: one platform comfortable at 340,000 connections went down at 17,000 new connections/second.",09-caching-edge
"Full jitter, not base + jitter","random(0, base) spreads clients across the whole interval. base + random(0, jitter) still has everyone waiting at least base, so it DELAYS the herd rather than dispersing it. AWS measured the difference and it is largest in exactly the reconnect-storm case.",09-caching-edge
"Drain WebSockets, do not sever them","On SIGTERM, stop accepting and spread closes (code 1001, going away) over 60-90 seconds. That turns a 10,000-client instantaneous storm into ~167 reconnects/second. terminationGracePeriodSeconds must EXCEED the drain window.",09-caching-edge
"Redis Pub/Sub loses messages, by design","Fire-and-forget with no persistence: a subscriber that is down misses everything sent while it was down, permanently, and it is invisible in testing. It is also single-threaded, so a busy backplane is a one-core limit. Use Streams or NATS JetStream when a miss matters.",09-caching-edge
"Do not subscribe every server to everything","Broadcast-to-all means the bus carries messages x servers, so it becomes the limit at a few dozen servers. Channel-per-topic with dynamic subscribe/unsubscribe cut deliveries 98% in one case (720,000/s to 14,000/s).",09-caching-edge
"Socket buffer defaults are sized for THROUGHPUT, not connection count","Default tcp_rmem+tcp_wmem is ~104 KB per connection: 10 GB at 100,000 connections. Lowering the DEFAULT (middle) value to 16 KB took it to ~3.2 GB, with no effect on the max for connections that need it.",09-caching-edge
"Without tiering, a miss is one origin request PER PoP","300 PoPs means 300 origin fetches for the same object per TTL. Mid-tier takes it to ~20; an origin shield takes it to 1. The measurement that proves you need it: how many PoPs fetch the same object within its TTL (84% duplication, mean 11.3, in one case).",09-caching-edge
"More PoPs means WORSE hit rate","Each PoP has a smaller traffic share and a colder cache, so latency improves and hit rate degrades. Tiering exists to reconcile them, which is why a 300-PoP CDN without tiering can offload less than a 30-PoP one.",09-caching-edge
"The origin shield goes near the ORIGIN","Not near users. Its job is to consolidate origin fetches; putting it near users means the origin fetch crosses the distance anyway. With multi-region origins, one shield per origin.",09-caching-edge
"When tiering is WRONG","When nothing is shared between PoPs, because the extra hop is then pure added latency on every miss: per-user content, very short TTLs, or an object population the mid-tier cannot hold either.",09-caching-edge
"The bright line for edge compute is STATE","Decisions belong at the edge: cache-key normalisation, JWT signature checks, redirects, A/B assignment, bot filtering. A database call from the edge is a database call from 300 places: one team measured 9,000 concurrent replica connections before reverting.",09-caching-edge
"Redis maxmemory-policy defaults to noeviction","Writes FAIL with OOM when memory is full while reads keep working. Right for a store, wrong for a cache, and since a cache that never hits its limit never exercises the policy, it is a latent outage.",09-caching-edge
"volatile-lru is the trap","It only evicts keys that HAVE a TTL, so any key without one is ineligible. A database full of them fills and fails exactly as noeviction does, while appearing to have an eviction policy configured.",09-caching-edge
"LFU beats LRU when there is scanning traffic","LRU evicts the hot set during a crawl, because a crawled item touched once is more recently used than a hot item touched 200ms ago. Measured on a nightly crawl: hit rate 94% to 31% under LRU, 94% to 91% under LFU.",09-caching-edge
"Set maxmemory well below the container limit","A BGSAVE fork copies pages as they are written, so the process can transiently use far more than maxmemory. Plus replication backlog, client output buffers and 1.1-1.5x fragmentation. ~75% of the limit is the usual shape.",09-caching-edge
"Adding shards does NOT fix a hot key","The key hashes to one slot and stays there. Options: client-side caching (180,000 reads/s became 12 with a 5s TTL and pub/sub invalidation), key splitting across N suffixes, or read replicas.",09-caching-edge
"Redis hash tags trade scalability for multi-key ops","{tenant:4471}:* co-locates everything on one shard, which enables MGET and MULTI and creates a hot shard you cannot split. Use them where co-location is genuinely required, not as a naming convention.",09-caching-edge
"Redis is not durable in the usual sense","AOF appendfsync everysec (the default) loses up to a second. More importantly replication is ASYNCHRONOUS, so a write acknowledged by the primary and not yet replicated is lost on failover. WAIT bounds it and does not prevent it.",09-caching-edge
"UNLINK, not DEL, for large keys","Redis is single-threaded, so DEL of a huge key blocks every client while it frees memory. UNLINK frees in a background thread. Same for SCAN instead of KEYS. SLOWLOG GET is the first thing to check on a latency complaint.",09-caching-edge
"Cacheability is a property of a FRAGMENT, not a URL","In one measurement 87% of a 'personalised' page's render cost was byte-identical for every user, and 'personalised' described two small fragments. Deciding at page granularity makes 100% uncacheable to protect 4%.",09-caching-edge
"Cohort keys make fragments shareable","A per-user price fragment is 41 million cache entries at a 0% hit rate; a per-TIER one is 12 entries at ~100%. The design work is finding the smallest enum that produces correct output, and the mistake is treating a cohort-shaped input as per-user out of caution.",09-caching-edge
"ESI vs streaming SSR","ESI makes the RESPONSE cacheable, because the shell and fragments are separate cache entries assembled at the edge. Streaming makes an UNCACHEABLE response feel fast; the response is one stream containing per-user content. They are complementary and routinely confused.",09-caching-edge
"The six cache layers","Browser (unpurgeable), CDN edge (biggest offload), reverse proxy, in-process (~50ns, N copies), distributed (~200us, one copy, central invalidation), database buffer pool. In-process with a short TTL in FRONT of Redis is the usual two-layer shape.",09-caching-edge
"Cache 404s, never cache 5xx","A 404 costs as much to render as a 200: 11M/month became 84k with one header in one case, the largest remaining origin load after the success path was optimised. Caching a 5xx extends a transient failure past its cause; stale-if-error is the opposite and correct operation.",09-caching-edge
"A cached 403 without the requester in the key","Serves one user's authorisation decision to everyone. It is the same unkeyed-input failure as cache poisoning, arriving through the negative-caching door. Any status that depends on who is asking must have that input in the key or must not be cached.",09-caching-edge
"The negative-cache sentinel","A cache cannot distinguish 'not cached' from 'cached as absent' without a distinguished value, so every miss re-queries the database. Store the sentinel with a SHORTER TTL than positives so new entities appear quickly. At scale a bloom filter is better: no false negatives.",09-caching-edge
"The two dead OAuth grants","IMPLICIT returned the access token in the URL fragment (leaked to history, Referer, logs) and could not authenticate the client; it existed only because browsers could not do cross-origin POST, and CORS removed that. ROPC has the app collect the user's password, defeating delegation. OAuth 2.1 removes both.",10-security
"The three token types and what each is NOT for","ACCESS: for calling APIs, not for identifying the user to your frontend. ID: proof of authentication for the CLIENT, not for calling APIs ever. REFRESH: for getting a new access token from the token endpoint, not sent to APIs. Sending an ID token to an API is the most common OAuth error.",10-security
"The JWT check everyone skips","AUDIENCE. An IdP signs every service's tokens with the SAME key, so signature and issuer pass for a token minted for any of them. aud is the only claim separating them, and it gets omitted because the token validates without it. In one audit: 12/12 validated expiry, 2/12 validated audience.",10-security
"The algorithm confusion attack","A library that reads alg from the TOKEN and picks a verifier can be given HS256 on a token signed with your PUBLIC RSA key as an HMAC secret. The public key is public, so anything is forgeable. Pin the algorithm in config; never take it from the token.",10-security
"state and PKCE are different defences","state prevents CSRF (an attacker completing a flow in the victim's session); PKCE prevents code interception and injection. Both are required, neither substitutes for the other. PKCE is now recommended for confidential clients too.",10-security
"Refresh token reuse detection","Each use issues a new token and marks the old used, recording the family. If a used token is presented again, revoke the WHOLE FAMILY and force re-auth, because you cannot tell the thief from a client that failed to persist its replacement. The DETECTION is the value; rotation without it is nearly pointless.",10-security
"DPoP vs mTLS-bound tokens","Both make a token sender-constrained instead of bearer. mTLS binds to a TLS client cert (strong, awkward through proxies). DPoP has the client sign a proof per request over ordinary HTTPS, which is why it works for browsers. A non-extractable WebCrypto key turns permanent token theft into session-bounded use.",10-security
"Token exchange and the act claim","Instead of passing the user's token down a service chain (every hop gets a token that works everywhere), exchange it for one scoped to the next audience. The act claim carries the acting service alongside the user's sub, so you get 'the order service, acting for user 4471' as an audit trail.",10-security
"RBAC vs ABAC vs ReBAC: the selector","What is the decision a function of? ROLES + action -> RBAC (sufficient for most internal tools). ATTRIBUTES of subject/object/environment -> ABAC. The RELATIONSHIP GRAPH between subject and object, especially with inheritance -> ReBAC.",10-security
"The signal you have outgrown RBAC","Roles whose names contain object identifiers. doc-4471-editor is a tuple pretending to be a role. In one system there were 1.85 million roles of which 412 were actual job functions; the rest were per-object grants.",10-security
"What ReBAC does that ABAC cannot","Reverse queries efficiently: 'who can access this' and 'what can I access'. ABAC's policy is evaluated per subject-object pair, so those require evaluating against every subject or object. They are product features (sharing dialogs, access reviews, search filters), and post-filtering search by a policy breaks pagination.",10-security
"A Zanzibar tuple","object#relation@subject, e.g. document:4471#editor@user:bob. The subject can be a USERSET (group:eng#member), which is how groups work without special cases. The schema composes relations: permission edit = owner + editor + parent->edit, where parent->edit is transitive folder inheritance.",10-security
"What a zookie is","An opaque consistency token from a write, passed to checks, because a stale ALLOW is a security bug: showing content written after access was revoked using a cached ACL. Store it with the content and check at_least_as_fresh: correctness where it matters, cached reads elsewhere. fully_consistent everywhere forfeits the caching.",10-security
"SSRF's highest-value target","The cloud metadata endpoint (169.254.169.254). It returns the node's IAM role credentials, which without per-pod identity are the UNION of every workload on the node. The Capital One breach was exactly this on IMDSv1: broad S3 access, 100 million records, minutes.",10-security
"Why SSRF blocklists fail","DNS rebinding: the hostname resolves to a public IP when you validate and to 169.254.169.254 when you fetch, because validation and fetch are separate lookups. Plus redirects and encodings (decimal/octal/hex/IPv6). Fix: resolve ONCE, reject any non-global resolved IP, connect to that IP, disable redirects.",10-security
"IMDSv2 stops most SSRF-to-credentials","It requires a PUT to get a token (most SSRF vectors do GET only), rejects requests with X-Forwarded-For, and a hop limit of 1 means a container cannot reach it. HttpTokens: required is a one-line option and is NOT the default on older instances.",10-security
"Workload identity removes static credentials","Instead of a stored secret an SSRF or RCE can steal, the workload proves properties about itself (image digest, service account, node) and gets a short-lived attested identity. SPIFFE SVIDs expire in ~1 hour and cannot be reissued elsewhere. IRSA/GCP WI exchange a projected OIDC token for temporary cloud credentials.",10-security
"Per-pod identity vs a node IAM role","A node role is the union of every pod's permissions, so a compromise of the least-privileged pod yields the most-privileged pod's access. Per-pod identity means an SSRF in the preview service yields a credential that can write one thumbnail. It is the improvement over the metadata endpoint even with IMDSv2.",10-security
"SSRF defence is three independent layers","Fix the SSRF (resolve-once + allowlist + egress proxy), enforce IMDSv2 (closes the metadata path regardless), and per-pod least privilege (a successful SSRF yields little). The Capital One chain needed IMDSv1 AND a wildcard node role AND an unvalidated fetch; removing any one breaks it.",10-security
"STRIDE, and what each letter violates","Spoofing (authenticity), Tampering (integrity), Repudiation (non-repudiation), Information disclosure (confidentiality), Denial of service (availability), Elevation of privilege (authorisation). Draw a data-flow diagram and ask which of the six apply to each element.",10-security
"The two threats STRIDE finds that ad hoc review misses","REPUDIATION (nobody brainstorms 'can they deny this', so the audit log is missing) and TOXIC COMBINATIONS (a chain safe at each step and dangerous end to end: a support agent who can create an order AND approve a refund can refund a fake order).",10-security
"A threat model's output is a DECISION per threat","Not a list of possibilities. Each threat gets: mitigate, eliminate, transfer, or accept-documented. A model that produces 40 threats and no decisions has failed, and a model that mitigates everything is padding. The deliverable is the disposition table.",10-security
"The most valuable output of a threat model","The trust boundaries on the diagram, because every threat of interest crosses one. Once you mark where data leaves the browser, reaches a third party, or crosses from lower to higher trust, you know exactly which flows to scrutinise.",10-security
"DREAD is deprecated","Its numeric scores are subjective and not comparable across assessors, so a 'DREAD 6.4' means different things to different teams. Microsoft, its author, moved away from it. Use impact x likelihood or tie each threat to a concrete abuse case.",10-security
"Supply-chain threats bypass source review","SolarWinds was the BUILD SYSTEM, xz was a trusted MAINTAINER, dependency confusion is the RESOLVER preferring the attacker's version. The pull request is not where the threat enters, so the durable controls are provenance (SLSA, signing) and eliminating static credentials, with scanning catching the rest.",10-security
"What an SBOM buys you","Answering 'am I affected' in minutes instead of days. When Log4Shell broke, teams with SBOMs queried them; teams without grepped build files across hundreds of services. It is the artifact that makes a zero-day response tractable.",10-security
"Sigstore keyless signing","Instead of a long-lived signing key that can be stolen, it issues a SHORT-LIVED certificate bound to the CI job's OIDC identity, logged in a public transparency log (Rekor). There is no signing key to steal: the same eliminate-the-static-credential reasoning as workload identity.",10-security
"Dependency confusion","An attacker publishes a PUBLIC package with your INTERNAL package's name at a high version, and the resolver, preferring the highest version across registries, pulls it. Fix: committed lockfile with integrity hashes (a swap fails the hash), namespace scoping, and reserving the names publicly.",10-security
"Dynamic secrets over rotation","A credential created on demand and revoked in an hour, unique per request, so a leak is worthless quickly and attributable to the request that leaked it. Rotation is the fallback and is harder than it sounds: you cannot atomically swap across N instances, so you need a dual-validity window.",10-security
"Deleting a leaked secret from git is not remediation","It was public the moment it was pushed and it is still in history. The credential must be ROTATED, and scanning must cover history (--all), not just the working tree. One scan found 41 secrets in history, 12 still valid.",10-security
"A noisy security pipeline is a disabled one","A gate flagging 400 dependency CVEs (most unreachable) gets a skip label 68% of the time. Reachability analysis took the count to a single digit and the skip rate to 4%, and only then were the real findings acted on. Usability is a security property.",10-security
"Admission control is the enforcement point","Shift-left SAST/SCA/IaC scanning in the PR is advisory and bypassable. Admission control at deploy (signature verification, an OPA/Kyverno policy) is where 'we scanned it' becomes 'it cannot run'. Most teams have the first and lack the second.",10-security
"The largest OAuth attack surface is redirect_uri","It is where the code or token is delivered, so controlling it steals the grant. Attacks: open-redirect chaining and loose matching (suffix, path traversal, userinfo confusion). The defence is EXACT string matching, byte for byte, mandated by OAuth 2.1.",10-security
"The OAuth mix-up attack","A client supporting multiple IdPs is tricked into sending an HONEST IdP's code to the ATTACKER'S token endpoint. Fix: the iss response parameter (RFC 9207), so the client verifies which IdP issued the code. It only affects multi-IdP deployments and is off by default.",10-security
"Where an SPA should store tokens","Not in the browser: the BCP recommends a backend-for-frontend, where the token lives server-side and the browser holds an HttpOnly session cookie, so an XSS cannot exfiltrate a usable token. localStorage and non-HttpOnly cookies are both XSS-readable. DPoP with a non-extractable key is the middle ground.",10-security
"The top three OAuth fixes","Exact redirect_uri matching (largest surface), audience validation on every API (the most common gap, because the token validates without it), and algorithm pinning (never take alg from the token). These three catch the majority of real vulnerabilities and are the three most often missing.",10-security
"How OAuth deployments accumulate flaws","Each flaw is invisible while the happy path works, so nothing fails in development to prompt the fix, and the defaults and tutorials predate the attacks. In one audit, four of the five OAuth flaw categories had a live finding, which is typical for a deployment predating the Security BCP.",10-security
"Head vs tail sampling","HEAD decides at the START, before it knows whether the request errored, so a 1% head sample keeps 1% of your ERRORS. TAIL decides after seeing the whole trace: 100% of errors and slow traces, 1% of the rest, for about +8% storage. The cost is a stateful Collector routing all spans of a trace to one instance by trace ID.",12-sre-observability
"memory_limiter first in a Collector pipeline","Without it, a traffic spike buffers until the Collector OOMs, taking down telemetry exactly when you need it. It applies backpressure by refusing data, which is the correct failure mode for an observability system.",12-sre-observability
"What an exemplar is and why it matters","A trace ID attached to a metric data point, so a latency-histogram spike links to an example slow trace. Without it a metric spike is a signal with no example and you are grepping logs for slow request IDs. Small feature, and it is the bridge that makes metric-first debugging work.",12-sre-observability
"The observability drill-down chain","Metric spike (via exemplar) -> trace (which span) -> correlated logs (trace_id in every line) -> continuous profile (which line of code). Each narrows the search by an order of magnitude. One investigation went from 3 hours to 90 seconds, and the change was correlation, not new backends.",12-sre-observability
"The observability 2.0 argument","Emit ONE wide event per unit of work with 50+ dimensions and derive metrics, traces and logs from it, so 'is latency concentrated in one customer on one API version' is a QUERY rather than a re-instrumentation. High cardinality on purpose, which is why it needs a columnar event store rather than a TSDB.",12-sre-observability
"Continuous profiling as the fourth signal","~1% overhead, running in production, so a p99 regression is a flame-graph DIFF between two deploys rather than a local reproduction. In one case a regex compiled per request, found in 10 minutes instead of hours. Needs perf_events access, which is a node-level decision.",12-sre-observability
"The incident commander does NOT debug","Someone must hold the whole picture, decide (roll back, escalate, page) and communicate, and that is incompatible with being head-down in a terminal. The instinct to put the most senior engineer hands-on is exactly what leaves nobody coordinating.",12-sre-observability
"Mitigate before you diagnose","Restoring service and finding the cause are DIFFERENT ACTIONS and the first is faster. A rollback works whether or not you know what broke; a targeted fix requires diagnosis with the site down. One incident: 90 minutes diagnosing, 2 minutes to roll back. The IC's first question is 'can we roll back'.",12-sre-observability
"Severity needs OBJECTIVE triggers","Subjective severity is negotiated downward, because the person who would have to run the SEV1 process argues it is a SEV2 and the incident is under-resourced. 'Error rate above 5% for 5 minutes on a customer-facing service' fires without a judgement call under pressure.",12-sre-observability
"Blameless is more rigorous, not gentler","If a single human error can take down production, the SYSTEM is the defect. Blame drives mistakes underground so the next person hides theirs, and a blamed person recurs while a fixed system does not. The test is the REPEAT-INCIDENT RATE, not whether a postmortem happened.",12-sre-observability
"Per-tenant SLAs create an observability requirement","An aggregate 99.92% can contain one enterprise tenant at 98.4%, which breaches their 99.9% SLA invisibly. So you need per-tenant SLIs, which is a high-cardinality slice a TSDB holds badly. The contract you sign determines the observability you need, and signing without measuring means the first breach is the customer's email.",12-sre-observability
"Publishing events is not event sourcing","Event sourcing means the events ARE the state, with no separate current-state source of truth. Most systems that say they do it publish events from a state-based store, which is normal and often better.",14-architecture-patterns
"The (stream_id, version) primary key","The optimistic concurrency mechanism in an event store, and the single most important line. Without it two concurrent commands both read balance 500, both append Withdrawn(400), and the balance is -300. It is what makes an aggregate a consistency boundary.",14-architecture-patterns
"Snapshots must never be the source of truth","They are a cache. If you cannot delete EVERY snapshot and rebuild from events, you have a state store with an event log attached, not event sourcing.",14-architecture-patterns
"Event sourcing's real cost is UPCASTERS","Events are immutable and live forever, so a 2019 v1 event must still be readable today: every schema change adds a transform that can never be deleted, and a v1 event may pass through four to reach v4. Eleven in year one for ONE aggregate, all permanent.",14-architecture-patterns
"Event sourcing's strongest argument","Retroactive correction. A rounding error affecting 14,000 accounts over three months was fixed EXACTLY by correcting the fold and replaying; a state-based store makes it a manual estimate per account, because the inputs were overwritten.",14-architecture-patterns
"Four reasons Kafka is not an event store","No per-stream optimistic concurrency (you cannot say 'append at version N or fail'), per-aggregate reads scan a partition, retention is a window and compaction keeps only the last value per key, and no transactional read-modify-write. It is an excellent event BUS.",14-architecture-patterns
"Crypto-shredding for GDPR","Encrypt PII per subject with a key you can delete; erasure deletes the KEY and the events stay structurally intact but undecryptable. Caveats: the key must not be in an undeletable backup, cached projections must be purged, and regulator acceptance is not settled.",14-architecture-patterns
"Order irreversible saga steps LAST","Some actions cannot be compensated: an email sent, goods shipped. If the confirmation email is step 2 of 5, a failure at step 4 tells a customer about an order that will not happen. The saga's step order is driven by REVERSIBILITY, not business sequence.",14-architecture-patterns
"Choreography's failure mode is the cycle","No single place describes the flow, so a new handler can close a loop (Shipping emits an event Payment reacts to, re-triggering Inventory: an infinite loop in production). Orchestration puts the flow in one file and answers 'where did this order stop' with one query.",14-architecture-patterns
"A compensation is not a rollback","A rollback means it never happened; a compensation is a NEW action and both are in the history, so the customer sees a charge and a refund. Compensations must be idempotent, and must be POSSIBLE, which is the design constraint.",14-architecture-patterns
"SOLID, per principle","LISKOV is a rule and violating it is a defect. DEPENDENCY INVERSION is the one worth internalising, and hexagonal architecture is it at scale. SINGLE RESPONSIBILITY is useful under 'one reason to change' and harmful under 'does one thing'. OPEN/CLOSED is a caveat: predicting extension points before two real cases produces the wrong abstraction.",14-architecture-patterns
"Hexagonal, Clean and Onion are one idea","Dependencies point INWARD: the domain owns its interfaces (ports) and infrastructure implements them (adapters). Worth it where there is real domain logic; ceremony over a thin CRUD service. An ArchUnit test is what makes it real rather than aspirational.",14-architecture-patterns
"Why a rewrite loses to a strangler fig","The old system encodes undocumented edge cases. One extraction replayed 8,400 real orders through characterisation tests and found 214 distinct edge cases against 61 in the specification, so a rewrite would have shipped ~153 unintentional behaviour changes.",14-architecture-patterns
"Branch by abstraction","Introduce an abstraction, point callers at it with NO behaviour change, add the new implementation behind it, migrate incrementally behind a flag, delete the old. It is how you do a large refactor on trunk; the alternative is a six-week branch producing an unreviewable merge.",14-architecture-patterns
"One-way vs two-way doors","Two-way doors should be decided FAST with incomplete information; one-way doors deserve information and a written record. Most organisational slowness is two-way treated as one-way, and most expensive mistakes are the reverse: a compacted topic's partition count, Flink's maxParallelism, an event schema.",14-architecture-patterns
"What makes an ADR useful six months later","The Consequences section INCLUDING THE NEGATIVES, and Context with the numbers. An ADR listing only benefits is marketing; the value is reading what the team knowingly accepted ('upcasters are permanent', 'onboarding gets harder'). Write them for one-way doors, not for reversible library choices.",14-architecture-patterns
"Which offer component is most flexible, and why?","Sign-on bonus. One-time money off a different budget line: it does not raise recurring headcount cost or create internal salary-equity comparisons with existing peers. Equity is next (a pool, not an operating budget); base is hardest (banded by level, recurring, directly comparable). Order: level, then sign-on, then equity, then base.",18
"The five questions to ask about any private-company equity grant","1) What percentage of fully diluted shares (not the raw count)? 2) Strike price vs the latest 409A and the preferred price? 3) ISOs or NSOs? 4) Post-termination exercise window (standard 90 days; extended is 7-10 years)? 5) What is the preference stack? Without the denominator and the preference stack, a share count is meaningless.",18
"Why is a 409A valuation not what your equity is worth?","It is a safe-harbour tax valuation deliberately set below the price investors paid for preferred shares. Common shares also receive nothing below the preference stack: $310M of 1x preferred means a $180M exit pays common zero.",18
"What is the highest-value question at the recruiter screen?","'What level is this mapped to, and what is the band?' The level determines the band and is anchored before any technical interview. Raising it at the screen is a process question; raising it after the loop means arguing against an assessment several people contributed to.",18
"What is the highest-value question at the end of each interview?","'Is there anything about my background that gives you hesitation?' An unstated concern becomes a 'no' you never hear about; a stated one you can answer in the room, in the follow-up email, or via a briefed reference.",18
"How do you get honest answers from peer interviewers?","Specific and recent, not general. 'When were you last paged at night?' gets a real answer where 'how is on-call?' gets 'it's fine.' 'Walk me through a recent feature from idea to shipped' defeats the idealised process description.",18
"Real versus manufactured offer urgency","Real: a named business reason, usually more than a week out. Manufactured: 24-72 hours, no reason given, pressure rises when you ask. The tell is that a manufactured deadline moves when you push back politely with a concrete date and reason.",18
"The highest-leverage non-comp ask when a level is stuck","A written, scheduled off-cycle review at 6 months with named reviewers and written criteria. It converts an unresolvable disagreement about your level into a testable claim with evidence that will exist later.",18
"What is specific to a Toronto offer from a US company?","Confirm the currency in writing (USD vs CAD is ~35 percent). Determine the structure: Canadian subsidiary, employer of record, or contractor. Read the termination clause: Ontario's ESA minimum is 1 week per year capped at 8, while common law reasonable notice for a senior employee can be 12-24 months. Confirm equity tax with a cross-border accountant (Canada's 2021 CAD $200k annual vesting cap on the stock option deduction).",18
"Why is an Ontario termination clause worth a lawyer's hour?","Waksdale v Swegon (2020 ONCA): if any part of a termination provision could violate the ESA, the whole provision is void and common law reasonable notice revives. For a senior employee that gap is six figures, and companies usually agree to an explicit notice schedule because it costs nothing today.",18
"What makes a reference call useful rather than a formality?","The brief. Send the company, role, level, three lines of scope, the specific projects to refresh, and the concern the hiring manager stated. Unbriefed you get 'they were great to work with'; briefed you get a direct answer to the doubt that prompted the call.",18
"Why is a cross-functional reference undervalued for staff roles?","Staff roles are assessed on influence without authority, and a product or partner-team stakeholder watched exactly that. Their answer is also more persuasive on engineering questions because they have no stake in the engineering narrative.",18
"Why reply well to a rejection?","A previously assessed candidate is a cheaper, lower-risk hire, which is why ATSs ship silver-medalist talent pools as a feature. Reply within 24 hours, name one specific thing, state the door is open, ask once for feedback, never argue. Panel feedback stays on file and can replace most of a later loop.",18
"Back-channel references: the practical implications","Assume they happen at senior levels, especially in a small market. Your actual reputation is a reference you do not control; leaving roles well matters years later; and if you know a relationship went badly, get ahead of it, because the damage comes from surprise rather than from the disagreement.",18
"React's two reconciliation heuristics","1) A different element type at a position unmounts the whole old subtree and mounts a new one (state is destroyed). 2) Within a list, keys give children stable identity across renders; without keys React matches by index. Identity = position + type, unless you supply a key.",11
"Why index-as-key breaks a list","It declares that a row's identity is its position, so filtering or reordering matches the wrong old row to each new row. React updates surviving DOM nodes in place, sliding all the state React does not own (uncontrolled input values, focus, scroll, animation progress, component state) onto the wrong rows.",11
"Why must hooks be called unconditionally?","Hook state is a linked list on the fiber indexed by call order, not by name. A conditional hook changes the number of calls between renders, so every later hook reads the neighbouring slot: useState(0) starts returning a different hook's value.",11
"When is React.memo a no-op?","Whenever any prop is an inline object, array or arrow function: the shallow comparison never returns true, so you pay comparison and allocation cost for nothing. Prefer moving state down, or passing the expensive subtree as `children` so its element is referentially stable.",11
"How do you reset a component's state when a prop changes?","Change its key: <ProfileForm key={userId} />. It uses the identity mechanism that already exists. The useEffect-copies-props-into-state alternative renders once with stale data, adds a render, and drifts out of sync.",11
"Concurrent React vs Server Components, in one sentence","Concurrent React changes WHEN work runs on the client and whether it can be interrupted; Server Components change WHERE a component runs and whether its code is in the bundle at all.",11
"What does startTransition actually do?","Marks updates as low priority so React renders them in an interruptible lane, yielding to the browser roughly every 5ms (frameYieldMs). If a higher-priority update arrives, the in-progress work-in-progress tree is discarded and rendering restarts with the new state, so the committed result always matches the latest input.",11
"When does startTransition NOT help?","When one component's render is itself slow: React can only yield between units of work, so a 200ms component blocks the main thread for 200ms inside a transition too. Transitions redistribute cost, they do not reduce it.",11
"What does 'use client' actually mark?","A module boundary, not a component. That module and everything it imports transitively join the client bundle, which is why putting it on a layout to fix one widget can return the whole tree to the bundle. Catch it with a bundle-size check in CI, not code review.",11
"Security property of a React server action","It is an RPC endpoint with a generated, discoverable id. Authentication, authorization and input validation must live inside the action. Rendering the button only for admins is a UI decision, not access control.",11
"The state ladder, bottom to top","1) local useState, 2) lifted to the nearest common parent, 3) context for config-shaped low-frequency values, 4) a server cache for anything the server owns, 5) a global client store for client-owned state distant components read AND write, 6) the URL for anything bookmarkable or shareable. Climb only when the current rung genuinely cannot hold it.",11
"Why is server data not global client state?","It is a cache. It needs deduplication, a staleness policy, refetch on focus and reconnect, retry, GC, and invalidation keyed by data rather than by call site. Manual invalidation is correct only while every current and future mutation site remembers; key-based invalidation is correct by construction.",11
"Context's one structural limitation","No selector: every consumer re-renders when the provider value changes, regardless of which field it reads. Mitigate by memoising the value and splitting contexts by change frequency (state vs setters). If you need selectors you need a store using useSyncExternalStore.",11
"Thunk vs saga vs observable: the position","For server data, none of them: use a query cache. For remaining client asynchrony, thunk by default (no new concepts). Saga only for genuinely long-lived cancellable orchestration (race, takeLatest, while(true) watchers). Observable only when the domain is a stream AND the team already knows RxJS. Audit the ratio: most sagas turn out to be takeEvery wrappers that pay the vocabulary cost and use none of the primitives.",11
"What does ISR do after the TTL expires?","Serves the stale cached HTML immediately and triggers background regeneration; the next request gets fresh content. It is stale-while-revalidate (RFC 5861) at the page level. On-demand revalidation via a CMS webhook complements it: build-time performance with near-real-time updates and no full rebuild.",11
"Rendering strategy: the decision order","Per route, not per app: 1) personalised per user? 2) how fresh must it be? 3) does a crawler or link preview need it? 4) how many routes (build time scales with count)? 5) how much is actually interactive? When one element is personalised, split the page rather than downgrading the whole route.",11
"Why is a fast LCP with a big bundle still bad?","Paint and interactivity are decoupled. Hydration cost scales with shipped JavaScript, not HTML size, so a server-rendered page with a huge bundle paints fast and stays unresponsive: good LCP, bad INP. Rendering strategy fixes paint; only shipping less JS fixes interactivity.",11
"Build time as an incident-response constraint","A 34-minute static build means you cannot ship a hotfix in five minutes, so a rendering decision has quietly set your MTTR. This is usually the argument that moves an organisation, because it is about risk rather than milliseconds.",11
"Why are host-allowlist CSP policies considered broken?","An allowlisted host usually also serves something exploitable: a JSONP endpoint, an old library with a known gadget, or user-uploaded content. Google's CCS 2016 measurement found deployed allowlist policies overwhelmingly bypassable. The replacement is a per-response nonce plus 'strict-dynamic'.",11
"The CSP policy that actually works","script-src 'nonce-{random}' 'strict-dynamic' https: 'unsafe-inline'; object-src 'none'; base-uri 'none'; require-trusted-types-for 'script'. The https: and 'unsafe-inline' tokens are fallbacks that supporting browsers ignore. Roll out with Content-Security-Policy-Report-Only first.",11
"What does base-uri 'none' protect against?","An injected <base href='//evil'> tag, which changes how every relative URL resolves, including relative script sources. One line, closes a whole bypass class, and it is the directive people leave out.",11
"Where can XSS still come from in a React app?","dangerouslySetInnerHTML; URL-valued attributes (React escapes the string, not the protocol, so javascript: still runs); direct DOM writes in effects/refs; spreading attacker-influenced props; server state serialised into the page; and every third-party script, which runs with your origin's full privileges.",11
"Does HttpOnly protect against XSS?","No. It prevents the script READING the cookie, so it stops persistent token theft. The browser still ATTACHES the cookie to same-origin requests the injected script makes, and those pass CSRF checks by construction. Storage choice bounds blast radius; it is not a defence.",11
"Why is connect-src the underrated CSP directive?","It is an egress inventory. Report-only mode enumerates every host code on your page can send data to, which is a question most teams cannot otherwise answer, and it turns a successful skimmer into a blocked request plus an alert. Both documented Magecart fines (BA £20m, Ticketmaster £1.25m) turned on that gap.",11
"The first rule of ARIA","Do not use ARIA when a native element will do. ARIA changes what assistive tech REPORTS; it adds no behaviour. role='button' on a div gives you the announcement and none of the focus, keyboard activation, disabled state or voice-control addressability.",11
"The two SPA-specific accessibility bugs no linter catches","1) A client-side route change does not move focus or announce, so a screen reader user stays put while the page replaces itself: move focus to the new heading with tabindex='-1' plus a polite live region. 2) Removing the focused element drops focus to <body>: decide explicitly where focus goes before removing it.",11
"WCAG AA numbers worth memorising","Text contrast 4.5:1; large text (24px, or 18.66px bold) 3:1; non-text contrast (UI boundaries, icons, focus rings) 3:1; target size 24x24 CSS px (WCAG 2.2); text resizes to 200%; reflow usable at 320 CSS px wide.",11
"Ontario's accessibility legal floor","The AODA Integrated Accessibility Standards Regulation (O. Reg. 191/11) requires WCAG 2.0 Level AA for websites of designated public sector organisations and private organisations with 50+ employees. For a Toronto product past that headcount it is a legal floor, not an aspiration.",11
"Why cap the number of E2E tests?","Flake compounds multiplicatively. 0.99^50 = 60% chance of a green run; 0.99^200 = 13%. Below ~90% green the team stops reading failures and re-runs until green, so the suite costs full runtime and gives no signal. Reaching 99.9% per test is expensive; capping n at 20 is a decision.",11
"Mock at the network boundary, not the module boundary","A hand-written module mock encodes what you BELIEVE the API returns, so it can never tell you that belief is wrong. Use MSW so the real data layer, cache, error handling and retry run, plus generated types diffed against the deployed schema so a removed field is a compile error.",11
"What can jsdom not test?","Anything involving layout: it has no layout engine. getBoundingClientRect and offsetWidth return zeros, IntersectionObserver and ResizeObserver are absent, CSS is parsed but not applied. Virtualised lists, drag and drop, popover positioning, sticky headers, overflow and breakpoints need a real browser.",11
"How to diagnose a testing suite","Take the last ~20 production bugs and assign each to the layer that would have caught it. In one audit 7 of 19 were API shape/nullability changes that no layer could see because every test mocked the fetch module with the shape the frontend already believed. The fix was a new layer, not more of the existing ones.",11
"Why does streaming work locally and not in production?","A buffering proxy. nginx, a CDN, or a serverless response handler accumulates the body before forwarding. Fix with proxy_buffering off, X-Accel-Buffering: no, and Cache-Control: no-transform. In one case this moved time to first paint from 2,600ms to 420ms against an unchanged 380ms time to first token.",11
"How do you batch renders in a streaming AI UI?","Flush on a 50-100ms interval, not per token and not per animation frame (tokens usually arrive slower than 60fps, so rAF equals per-token). Also split accumulated text into blocks so completed paragraphs and code blocks are parsed once and memoised, leaving only the last block live: otherwise every token re-parses a growing document.",11
"The one markdown construct worth handling explicitly mid-stream","Code fences. An odd number of ``` means everything after it renders as code, so the layout flips back and forth as the stream continues. Optimistically append a closing fence when the count is odd: ~12 lines, removes the only flicker users notice.",11
"What does a correct stop button do in an AI UI?","Aborts the client fetch, propagates the abort to the upstream model call server-side, and KEEPS the partial output labelled as stopped. A UI-only stop keeps generating and billing: one measurement found 410 stop presses a week left ~172,000 output tokens generated after the press and discarded.",11
"Accessibility of a streaming region","Do NOT put aria-live on the streaming text: it announces every flush as fragments. Use aria-busy on the container while streaming, a separate polite live region for STATE transitions only ('Searching documentation', 'Response complete'), and let the finished message be navigated normally.",11
"Undo in an agentic UI","You mostly cannot. A side-effecting tool call (email sent, record deleted, payment made) is not undoable by the client. The control that works is a confirmation gate BEFORE the call, calibrated per tool risk. It matters more than for ordinary destructive actions because the user did not author the action, the model proposed it.",11
"The four questions that choose a deployment strategy","1) Can v1 and v2 coexist, including DB, cache and message formats? 2) How fast must rollback be (seconds -> blue/green or a flag; minutes -> canary or rolling)? 3) Is the change routable per request, or are they installed clients (-> rings)? 4) Do you have the traffic and metrics for a canary to detect anything in the bake window?",13
"Why do deploys cause latency spikes?","Kubernetes defaults maxUnavailable to 25%. A 20-replica service at 85% CPU drops to 15 replicas at 85%*20/15 = 113% and browns out. Fix: maxUnavailable 0 with maxSurge, paying spare capacity for the window. Second cause: a readiness probe that checks the port rather than warmth.",13
"Why is per-request canary routing dangerous?","Blast radius is not the canary percentage. A 12-request session at a 5% canary crosses versions with probability 1 - 0.95^12 = 46%, so any cross-version incompatibility breaks ~half of sessions from a 5% rollout. The routing UNIT, not the percentage, determines exposure.",13
"How does consistent cohorting work, and why the salt?","bucket(id, salt) < percent, hashed into a large bucket space. Stateless (no coordination), monotonic (1%->5% keeps everyone already in, never flips a user back), and independent per rollout. The salt matters because without it the same users are always in the leading cohort of every rollout: they bear all the risk and stop being representative.",13
"Two things to set up before your first cohorted rollout","1) The cohort in the cache key or Vary, or a canary response gets cached and served to control users, turning a 1% rollout into a 100% incident. 2) The cohort as a metric label and span attribute, because comparing v1 to v2 is the whole point and you cannot slice what you did not record.",13
"The six things a canary cannot catch","Slow-burn leaks (manifest beyond the bake window), rare code paths (1% divides every path's rate by 100), emergent failures that are a function of the fraction (cache-key changes, pool exhaustion, retry amplification), silent correctness bugs (200 and fast), mixed-version interactions (the canary tested v2 alone), and effects elsewhere or later (old clients, async jobs, batch, other cycles).",13
"The rare-path canary arithmetic","200 req/s, 1% canary, path frequency 1 in 10,000, 20-min bake: 0.0002 executions/s * 1200s = 0.24 expected, so P(at least one) = 1 - e^-0.24 = 21%. Longer bakes do not fix this economically; synthetic traffic, fault injection, or routing 100% of one small segment does.",13
"Why can a cache-key change pass a canary and cause an outage?","At 10% the canary still reads keys the 90% on v1 keeps populating in the old format, so hit rate barely moves. At 100% nothing refreshes the old format, the whole working set turns over at once, and the backing store takes the full uncached load: a 95% hit rate going cold is a ~20x origin QPS increase.",13
"What does a PodDisruptionBudget actually protect?","Voluntary disruptions via the Eviction API: kubectl drain, cluster autoscaler, descheduler, managed node upgrades. NOT your own Deployment/StatefulSet rolling update (controllers delete pods directly; that is maxUnavailable), not node crashes, not kubectl delete pod. A quorum service needs the PDB, maxUnavailable, AND topology spread.",13
"Why does a pod get requests after it starts shutting down?","SIGTERM delivery and endpoint removal happen concurrently, and endpoint removal is eventually consistent (endpoints controller -> EndpointSlice -> kube-proxy on every node -> ingress -> cloud LB). Fix: a preStop hook that sleeps longer than measured propagation, since SIGTERM is not sent until preStop returns, plus terminationGracePeriodSeconds exceeding sleep + drain.",13
"How do you deploy a service with 100k WebSocket connections?","Drain rather than drop (close over a window with a reconnect hint), require jittered exponential backoff on the client so retry rounds do not re-synchronise the herd, make sessions resumable so a reconnect costs one round trip, and bound connection age with jitter so reconnects run continuously and a deploy stops being a special event.",13
"Bounded connection age: the argument","It converts a rare, expensive, untested event into a continuous, cheap, always-exercised one. 180,000 connections with a 40-min jittered max age gives ~75 reconnects/s continuously, so deploy-time reconnects (~530/s) are ~7x baseline instead of a thousandfold spike against a path the system has never handled.",13
"What is a semantic merge conflict?","Two PRs each green against main, merging cleanly in git, that break main together: one renames a function and updates its call sites, the other adds calls to the old name in a different file. Neither CI run saw the other's change. A merge queue fixes it by testing main plus everything ahead of you in the queue.",13
"When do you need a merge queue?","When CI duration times merge arrival rate makes most PRs stale on completion. At 7.5 merges/hour and a 26-min CI run, expected arrivals during your run = 3.25, so P(none) = e^-3.25 = 3.9%: 96% of PRs must rebase and re-run. That is arithmetic, not a behaviour problem.",13
"How do you size a merge-queue speculative batch?","From the measured per-PR failure rate. At 1.2% failure, a batch of 8 is green 0.988^8 = 91%, giving ~1.28 CI runs per batch and ~6.25 PRs per run. At 9% failure the same batch is green only 47% and bisection dominates. So cutting flake is a prerequisite for large batches.",13
"The dependency order for trunk-based development","CI speed and flake rate first, then the merge queue, then the branching policy. Announcing trunk-based development while merging still costs a rebase-and-rerun cycle asks people to pay that tax more often, and they correctly decline. After CI 26->8 min and flake 9%->1.2%, median branch age fell 4.2 days -> 0.6 with no further policy announcement.",13
"How do you choose a testing shape?","Derive it: take the last ~20 production incidents caused by a code change and assign each to the layer that would have caught it. For a service backend the answer is usually contract tests and integration against real dependencies, plus a cluster in a category no layer covers (inputs nobody anticipated). The pyramid's premise, that integration tests are expensive, predates Testcontainers.",13
"Why prefer a fake to a mock?","A mock encodes your belief about the collaborator and distributes it across every test method, so if the belief is wrong the tests still pass and correcting it means editing hundreds of call sites. A fake is a working in-memory implementation written once, where the real contract including failure modes lives in one place.",13
"The five most productive property-test shapes","Round trip (decode(encode(x)) == x); invariant (sort output is a permutation and is ordered); oracle/differential (new impl == old impl, the best property during a rewrite); metamorphic (a relation between outputs when you cannot state the correct output); and stateful sequences checked against a model.",13
"What does mutation testing tell you that coverage does not?","Coverage records that a line executed. Mutation changes the line and asks whether any test fails. A pricing module at 91% line coverage had a 38% mutation score: for ~6 in 10 semantic changes to code the tests visit, nothing failed. The compressed version: switching the rounding mode from HALF_UP to DOWN killed no test.",13
"How do you run mutation testing affordably?","Incremental (mutate only the diff: ~60 mutants and ~90s for a 40-line change), scoped (money, permissions, safety modules only), and surfaced as review comments rather than gated on a score. That is the shape Google published after finding the naive whole-repo version unusable.",13
"Why does peeking inflate false positives?","A fixed-horizon p-value assumes one look at the planned sample size; each extra look is another chance for the random walk to cross the threshold. Roughly 8% at 2 looks, 14% at 5, and under continuous monitoring of an unbounded test the probability of eventually crossing any fixed threshold approaches 1 with no true effect.",13
"Three legitimate ways to look at a running experiment","Fixed horizon with a pre-registered sample size (look once). Group sequential with alpha spending across pre-specified interim looks (O'Brien-Fleming, Pocock). Or always-valid inference / confidence sequences, valid at every point in time. The last costs power: you pay for the right to peek.",13
"How do you tell novelty from a real effect?","Plot the effect against days since THAT USER's first exposure, not calendar date, because a calendar plot mixes day-1 and day-10 users and averages the decay away. Also analyse new users separately (no prior expectations), and keep a small never-exposed holdback so you can measure the effect months after shipping.",13
"What is interference (SUTVA violation)?","A unit's outcome depends on others' assignments. Finite shared supply OVERSTATES the effect (treatment consumes what control would have); network spillover UNDERSTATES it; shared infrastructure degrades control; a shared model trains on both arms. Remedy: cluster randomisation by metro/graph cluster/budget pool, at a large variance cost.",13
"When do you use a switchback test?","When interference is global rather than local (dispatch, matching, pricing, a shared pool), so no cluster is independent. The whole system alternates condition over randomised time periods. Design: period length long enough for carryover to decay, a burn-in discarding each period's first minutes, and analysis clustered at the period level.",13
"The most common switchback analysis error","Treating each event as independent. Events within a period share a condition and a demand environment. In one case a +3% effect gave p < 1e-9 at the job level and p = 0.11 on the same data analysed at the period level with cluster-robust SEs. Effective sample size is the number of PERIODS, not events.",13
"What check runs before you read any experiment result?","Sample ratio mismatch: chi-square the observed arm sizes against the intended split. A 0.3% imbalance across 1M users is not chance. Causes (treatment JS failing on old devices so exposure never logs, redirects dropping users, crashes removing the worst-affected) all remove a non-random slice AND usually bias metrics in treatment's favour.",13
"How much project capacity does a team of 7 have?","Roughly 10-22 of 35 nominal person-days a week. Subtract vacation/statutory (~7% in Ontario), sick (~3%), meetings (10-20%), on-call (most of one person), then the measured ones: interrupts, hiring/onboarding, forced upgrades. Healthy focus factor 50-65%; above 70% something is not being counted.",16
"The three-bucket budget, and the artifact that uses it","Feature / Reliability+investment / KTLO, declared (e.g. 60-20-20) and then measured. Declared-vs-actual (60/20/20 against 38/9/53) turns 'why are you behind' into 'KTLO consumed 2.7x its budget, here is the itemisation, which of these do you want me to change'.",16
"How do you forecast without estimates?","Monte Carlo over historical throughput: take 10-20 weeks of items-completed-per-week, count remaining items, and repeatedly draw random past weeks until the count is consumed. Report P50/P85/P95. No estimates needed because history already contains the size distribution, the interruptions and the bad weeks. State the scope-discovery factor separately.",16
"Little's Law, applied","cycle time = WIP / throughput. 20 items in progress at 4 completions/week is a 5-week cycle time; cutting WIP to 8 makes it 2 weeks arithmetically, with no one working faster. Throughput usually rises too as context switching falls, but the shortening does not depend on that.",16
"What does flow efficiency tell you?","Active time / elapsed time, typically 5-20%. A 14-day cycle time at 18% efficiency is 2.5 days of work and 11.5 days waiting (review, dependencies, decisions, deploy windows, environments). So 'work faster' addresses 2.5 days and removing queues addresses 11.5. In one case fixing queues took cycle time 14->8 days and throughput 5.2->7.1/week with nobody working longer.",16
"How do you answer 'when will it be done'?","With a probability, the assumptions, and the levers. 'P50 24 March, P85 11 April, assuming 62 items plus a historical 20-40% discovery factor, the team stays at 7, and the payments dependency lands by 3 March. To hit 24 March at P85, cut ~15 items (here is my ranking) or add people, which historically shows up in throughput after 4-6 weeks.'",16
"The four numbers in an on-call design","Rotation size (>= 6 for 24/7; below that the doom loop has gain > 1), page budget (<= 2 events per 12-hour shift, per Google SRE), an interrupt shield that is a DIFFERENT person from the on-call primary, and a toil cap with a defined overflow (Google: <= 50%, excess returns to the product team).",16
"First move on a rotation getting 47 pages a week","Classify four weeks of pages into actionable / automatable / should-be-a-ticket / delete / false. In one case 40/61/52/27/8 of 188. The deletions and downgrades alone took 47/week to 18 in one meeting with zero engineering work. Most pager pain is a classification problem, not a reliability problem.",16
"Why is actionable rate more important than page count?","Page count is trivially gamed by deleting alerts you needed. Actionable rate (fraction where a human genuinely had to decide or act) tells you the remaining pages are the right ones. 21% -> 76% while the count fell 47 -> 6 is a real result; the same count drop with a falling actionable rate means signal was deleted.",16
"What makes a page budget or toil cap enforceable?","The overflow mechanism, agreed in advance. 'Two consecutive weeks above budget doubles the next sprint's reliability allocation and reduces the feature commitment' written into the charter with the PM's agreement. After the breach the conversation is about a named feature slipping and it will lose, which is why the agreement must precede the event.",16
"The four blocker types and their fixes","INFORMATION (the answer exists) -> find it and write it down. DECISION (nobody has committed) -> make it, or take a recommendation with a default and a deadline. DEPENDENCY (another team) -> escalate on a schedule, or route around. SKILL (cannot do it yet) -> pair, teach or reassign.",16
"Which blocker type hides, and how do you surface it?","Skill, because it is safer to report as information or dependency. The tell: the blocker does not resolve when its stated cause is removed. The question that surfaces it without accusation: 'if I gave you the answer right now, what would you do first?' A vague answer means the blocker is the approach.",16
"The 24-hour rule for blockers","Nobody is blocked overnight without a NAMED PERSON (not a team) and a NEXT ACTION with a time. 'The platform team is looking into it' is not an owner. The rule is about ownership, not resolution: a week-long blocker is fine if it has both.",16
"How do you unstick a stalled decision?","Never take a question upward, take a recommendation with a default and a deadline: 'I recommend A; B costs two extra weeks and buys X; I need an answer by Thursday or I proceed with A.' In one case a decision sat 21 days, cost ~6 engineer-days of work built against both options, and resolved in 26 hours once framed that way.",16
"Why publish an escalation ladder?","To remove the judgment from the moment, because the failure is escalating too LATE. Heroic absorption protects the other team from the consequence of their slip, so it recurs, and your team gets blamed. In one case a dependency sat 31 days and was scheduled 90 minutes after the first peer-to-peer conversation.",16
"How do you escalate without damaging the relationship?","State the ask, the impact in days, what you tried, and TWO acceptable outcomes: 'get it scheduled, or agree our date moves by the same amount.' An escalation with one acceptable outcome is a demand, and it is why people route around you afterwards.",16
"The skill matrix, and the three readings","Each person x each competency (domain, design, code, ops, comms, ownership, mentoring), levels 1-4, private to the lead. Three readings: RISK (a competency with exactly one strong person is a bus factor you created), INDIVIDUAL GAPS (the low cell blocking someone at the next level), and UNDERUSED STRENGTH.",16
"The stretch ratio, and why the dimension matters","~70% known / 30% new. Technical complexity, scope/ambiguity and visibility are three separate stretches; combining them is how an assignment becomes a setup. The failure is asymmetric: an under-stretched engineer can be stretched next quarter, an over-stretched one often concludes they are not capable.",16
"The delegation ladder","Do it -> do it and tell me -> propose then do -> decide and inform -> own it entirely. Move ONE rung at a time, and say WHICH RUNG per area. The common failure: a lead says 'you own this' and behaves like rung 3, reviewing every design. Being told rung 3 honestly is better.",16
"SBI feedback","Situation, Behaviour, Impact. 'In Tuesday's design review, when Priya raised the backfill concern you moved to the next slide; she stopped contributing, and the backfill risk we hit last week was what she was raising.' Behaviour is observable so it is arguable, and the disagreement is the useful conversation. Nobody can act on 'communicate better'.",16
"Why is review latency the biggest hidden cost in cycle time?","It is time spent on finished work, and nobody measures the waiting. One team's 14-day cycle time included 5.3 days of review waiting (38%). And the median review took 11 minutes of actual work against a 9.4-hour wait: a scheduling problem, not a willingness problem.",16
"The single highest-impact code-review change","A 20-30 minute review slot on everyone's calendar at a fixed time, right after standup, counted as work. Review competes with focused work and loses every contest in the moment; the calendar makes it lose once, in advance. One team: time-to-first-response 9.4h -> 2.1h.",16
"Why cap PRs at ~400 lines?","Defect detection collapses past a few hundred lines. Measured share of PRs merged with 0-1 substantive comments: 12% under 200 lines, 47% at 400-800, 81% above 800. The team was not under-reviewing out of laziness; past that size the artifact is unreviewable.",16
"The PR comment taxonomy, and what it buys","blocking: / suggestion: / nit: / question: / praise:. It makes the reviewer's authority explicit, forces them to decide how strongly they feel (which reduces blocking comments), and makes praise happen. `question:` is the highest-value prefix because the ambiguity between 'I'm curious' and 'change this' is where two seniors reliably annoy each other.",16
"Review order, and why","Correctness -> design and boundaries -> tests -> readability -> nits. Reviewer attention is finite and front-loaded, so starting with naming spends it before the concurrency bug. And a design comment on the first pass is a redesign; the same comment after three rounds of nit-fixing is a demoralising rewrite.",16
"When is a promotion actually decided?","Two quarters before the cycle, when the SCOPE is assigned. The calibration room evaluates artifacts and corroboration, both of which take a quarter to produce and a quarter to be noticed. The packet can only describe a case; it cannot create one.",16
"What counts as evidence in a calibration room?","Anything a stranger could verify in two minutes without asking you: a design doc with named reviewers, an RFC other teams adopted, an incident commanded with the write-up, a person grown who says so in writing, a measured before/after. Everything else is an adjective, and a packet of adjectives argues the CURRENT level well.",16
"Why does one objection outweigh several supporters?","Because 'not yet' is the reversible decision and the room is optimising against promoting someone who then struggles. So the work is removing the specific objection, not accumulating nods. Pre-socialise a month early: show the case to a future participant and ask what they would push back on.",16
"Would you include a failure in a promotion packet?","If the room will hear about it anyway, yes, with the learning attached, because hearing it from the manager first beats hearing it from a participant who was on the call. In one case a candidate misdiagnosed the first 40 minutes of a Sev1, said so in their own postmortem, and the platform director called it the most honest postmortem they had read that year.",16
"Options with costs, not verdicts","Three options differing in KIND not size, each independently shippable, cost in the PM's units, consequences named including the inconvenient ones, plus your recommendation with the condition that decides it: 'I'd pick B if multi-currency is still on the roadmap, A if it isn't.' That last sentence names the fact that decides it, and it is usually the PM's fact.",16
"Why is 'that'll take six weeks' a bad answer?","It is true and contains nothing the PM can act on: not what a shorter version looks like, not what the six weeks buys, not what would make the date work. So they go find that information from someone less informed. One PM said plainly: 'I didn't loop you in because I expected you to say no, and I needed to give them something.'",16
"Who owns the non-functional requirements?","Nominally the PM, actually nobody unless the lead writes them, and they get decided anyway by whoever writes the code under deadline. The workable split: the lead writes the QUESTION, the PM supplies the number. 'How stale can this be before a merchant calls support?' produced 'under a minute', which was the difference between 480,000 req/min and 2,000.",16
"The six motivators","Scope, learning, money, title, flexibility, teammates. Most people have a primary and a secondary and they change over time. The default failure is a lead assuming their own motivator is universal: in one audit the lead was wrong about four of five people, and each wrong model had produced a specific management error.",16
"The earliest reliable sign someone is leaving","They stopped arguing. Someone who used to find the hole in a design and now agrees with everything has disengaged, not matured, and it is easy to miss because the quieter version is more comfortable for the lead. Complaining is inverted: a complainer is still invested.",16
"How do you act on a retention signal?","Not 'are you thinking of leaving' (unanswerable honestly, forces premature disclosure). Name the specific behavioural change with its impact and ask an open question: 'you used to find the hole in a design and haven't in months; that was useful and I've missed it, what changed?' It is SBI applied to a retention signal.",16
"Why do a proactive market check?","Replacement commonly costs 6-12 months of salary all in, so an adjustment is a fraction of the alternative, and the most underpaid people are often the least likely to ask. One engineer 18% below market had never raised it because 'it felt like a bad look', and their reaction to the fix was 'I didn't know that was something I could ask for.'",16
"Should you counter-offer?","Usually not: the decision formed months earlier, a counter fixes the most NAMEABLE reason (usually money) rather than the real one, accepted counters show high subsequent turnover, and it teaches the team that resigning is the raise mechanism. Real exceptions: a genuine sole comp gap you can fix permanently, or an honest bridge for a specific timeline.",16
"Why write the hiring scorecard before the loop?","Otherwise the loop measures rapport. In one audit the loop assessed coding three times, system design once, and production debugging never, and both hires struggling at 12 months were struggling on production debugging. Each signal must be owned by exactly ONE round, and the bar written as observable evidence ('can name failure modes unprompted') rather than as a level.",16
"What makes a hiring 'no' defensible?","Specific absent evidence tied to a named scorecard line: 'the scorecard says they must reason from evidence in production; given real logs they proposed three causes in four minutes without checking anything, and could not say what would rule the first one out.' The discipline runs both ways: a no that maps to no scorecard line ('not how I would have solved it') should be overridden.",16
"The cheapest improvement to a hiring loop","Independent written feedback submitted BEFORE the debrief. Costs nothing, removes nobody's autonomy, stops the loudest voice anchoring the room. One team: write-ups containing specific evidence went from 31% to 88%, because a write-up you cannot revise after hearing others has to stand on its own.",16
"The wrong-yes / wrong-no asymmetry in hiring","A wrong yes costs 6-12 months of a lead's attention, the team absorbing the gap (usually the strongest people), and a visibly reset standard. A wrong no costs more interviews. Every process pressure (open req, stretched team, recruiter metric, six hours already spent) pushes the other way, which is why bar defense is a named responsibility.",16
"Why must an ADR be immutable?","Because the history is the value. The useful artifact is 'we decided X in March for these reasons and superseded it in November for these others.' An edited record shows only the current state, which the code already shows. Supersede, never edit, and mark the old one superseded.",16
"What belongs in an ADR?","One decision. Context with numbers, the decision, consequences INCLUDING the negatives, alternatives considered, and a revisit trigger with a number ('revisit above 500M postings/year'). Half a page to two pages, numbered, in the repo next to the code. An ADR that takes an hour to write does not get written.",16
"The meeting audit rules","No agenda, no meeting, and the agenda names the DECISION (not 'discuss X'). No decision, shorter meeting. Status meetings become async written updates. Every recurring meeting gets an expiry date. Decisions and actions written down in the meeting with owners. A weekly status meeting for 9 people is ~52 person-days a year.",16
"The monthly one-pager to your director","What shipped (outcomes, not activity), what's next, RISKS named early with your assessment and whether you need help, asks with a default, and a line on people. The test of whether it is written well: your director forwards it unedited. 'Refunds p99 2.1s -> 340ms' is repeatable upward; 'worked on refunds' is not.",16
"How do you surface a risk without transferring the problem?","'The identity dependency is unscheduled. If it is not scheduled by the 7th we build a two-day workaround. I do not need help yet; I will tell you if that changes.' Information without an ask. That is what makes the director trust the next one, and it is why a director's worst experience (hearing about your problem from someone else) never happens.",16
"Managing former peers: what actually changed on day one","You know things you cannot share (so silence now means something), your casual opinion is a directive, complaining to you is a risk so you stop hearing complaints, and the friendship has a new asymmetry. None of it is optional; the only question is whether it gets renegotiated deliberately, in the first two weeks, ending with 'is there anything about this that's awkward?'",16
"The most common failure of a first-time technical lead","Keeping the interesting work, with a plausible justification ('it's the risky part'). Every time you take the most developmental piece from someone who needed it. One lead had kept 2 of the 5 most interesting items in a quarter (40%) and only learned it when a report said 'you took the rate-limiter rewrite, that was the piece I wanted.'",16
"The overcorrection trap with former peers","Being harder on your friend than the same behaviour would earn from anyone else, to prove impartiality to yourself. It is the more common error among conscientious new leads, it is unfair in a harder-to-see way, and you will not notice it. Name the risk in week one and give them permission to raise it.",16
"What is glue work?","Coordination, communication and maintenance that makes a team function and no rubric rewards: onboarding, retros, the answer desk, runbooks, noticing the cross-team gap, chasing dependencies. High-value work with no attribution mechanism, accreting on whoever is most conscientious. Research on non-promotable tasks finds it falls disproportionately on women and persists in randomised settings.",16
"The three responses to glue work","DISTRIBUTE (named rotations, so allocation stops being self-selection; delete what nobody can justify). MAKE VISIBLE (convert to artifacts: an FAQ with their name, a written cross-team design note, a quarterly retro pattern analysis). PROTECT THE PATH (say explicitly that it will not promote them, that this is a system failure not theirs, and give named technical scope instead). Expect 'but then who does it' and answer with names and dates.",16
"Why is a rewrite usually wrong?","The old system's ugliness is largely accumulated corrections you cannot enumerate: a tax rule from a 2019 audit, a rounding requirement from a payment processor, a grandfathered contract term. A rewrite discards them all at once and rediscovers them in production. Structurally it also fails because the last 30% is the part nobody understands and the target keeps moving.",16
"What is a characterisation test?","A test that records what the code currently DOES, not what it should do, so a refactor can be proven not to change anything. Record known-wrong behaviour as-is: fixing a bug and refactoring are two changes, and doing both means a production problem cannot be attributed to either. Fix the bug separately afterwards with its own test.",16
"How do you generate characterisation tests at scale?","Capture real production inputs/outputs and STRATIFY the sample by country, order type, currency and error class rather than by volume. In one case the top 3 countries would have been 94% of a uniform sample and were 31% of a stratified one. Where behaviour cannot be captured offline, run both implementations in production and diff.",16
"Seams, in order of preference","Ranked by how much legacy code you must modify (the risky operation): ROUTING seam at a gateway (zero legacy changes), EVENT seam / CDC on the legacy database (zero), INTERFACE extraction (some), FEATURE FLAG inside the old code (most). People reach for the flag first, which is why attempts stall: editing the old code requires understanding it.",16
"How do you sequence a strangler migration?","By business capability, not by layer, so each increment is a complete revertible slice. Start high-value/low-risk, not with the hardest part. In one case shipping-cost went first (changed 4x/year at 6 weeks each -> 3 days), and that number funded the other four increments. The frightening part went last, in shadow for 8 weeks with a penny-level diff to finance.",16
"'The new system is more correct' is a behaviour change","In one migration, shadow comparison found 31 differences before any traffic moved: 19 new-system bugs, 9 OLD-system bugs the new one had accidentally fixed, 3 genuine ambiguities. Each of the 9 needed a deliberate decision, because changing what a customer is charged is a product event regardless of which value is more defensible.",16
"How do you make sure the old system gets deleted?","Put deletion on the plan as the last named milestone with a date, from the beginning, and publish residual traffic weekly so 'nearly zero' becomes '0.4%, which is the finance spreadsheet, one partner, and an admin tool nobody has used in 14 months.' Then report lines deleted, instances retired, cost stopped. That is how a team learns these projects end.",16
"What makes a good SLI?","Something a user would notice if it got worse, measured as close to the user as you can afford, expressed as GOOD EVENTS / VALID EVENTS so it aggregates and produces an error budget. Latency must be a proportion under a threshold, not a percentile value: percentiles do not aggregate and give you no way to say how much budget a slow period spent.",12
"Why does SLI measurement point matter?","The further from the user you measure, the more failure modes you exclude by construction. A server-side success-rate SLI cannot see a CDN outage, a TLS failure, or a JS bundle 404. In one incident a CDN misconfiguration rendered a blank page for a whole region and the SLI IMPROVED, because app servers saw fewer requests and all succeeded.",12
"How do you define 'valid events'?","Explicitly and in writing, because the denominator is where SLIs get gamed by accident. Exclude health checks and internal probes. Decide per status code: 400/404 usually the client, 429 is YOUR refusal to serve, 401 from your own broken token service is your outage. Filter bots with a versioned, reviewed rule set, because changing the filter silently changes your reliability history.",12
"The SLI masking problem","/api/feed at 10M req/day and 99.99% plus /api/checkout at 50k/day and 92% gives a combined 99.95%: the SLO is met while checkout is broken for one user in twelve. A service-wide SLI averages over journeys of wildly different importance, so scope SLIs per user journey.",12
"How do you derive an SLO target?","Four inputs: observed behaviour change (join historical degradation to conversion/support data and find the knee), current performance vs complaint level, the contractual floor (SLO must be strictly tighter than the SLA), and the dependency ceiling. Then do the budget arithmetic: 99.99% is 4 minutes a month, so one 5-minute incident exhausts it.",12
"Error budget per 28 days, memorised","99% = 6h43m; 99.5% = 3h21m; 99.9% = 40m; 99.95% = 20m; 99.99% = 4m2s; 99.999% = 24s. The reading that changes conversations: at four nines, a single 5-minute incident consumes more than a month's budget.",12
"What if the SLO target exceeds the dependency ceiling?","You exhaust the budget permanently for reasons no work on your own service can fix. Checkout depending serially on auth 99.95, inventory 99.9 and a gateway 99.95 has a ceiling of 99.80% against a 99.9% target. The useful outcome: raising the target requires removing a serial dependency, so the SLO arithmetic produces an architecture decision.",12
"Why two latency thresholds?","They catch different failures. '99% under 500ms' catches a general slowdown; '99.9% under 3s' catches a heavy tail where a small group times out completely. With one threshold, the pattern where most users are fine and 0.5% cannot use the product reports as a small budget spend rather than an outage for those users.",12
"The OpenTelemetry API/SDK split, and why it matters","The API is what your code and LIBRARIES call, and it is a no-op unless the application wires an SDK. So a library can instrument itself with no runtime behaviour and no backend dependency, and the application alone chooses whether anything is recorded and where it goes. That is why upstream library instrumentation is possible at all.",12
"Why run an OpenTelemetry Collector?","It is the place you change behaviour without redeploying applications: redact newly discovered PII in one config change instead of 40 service deploys, drop a high-cardinality attribute that tripled the bill, fan out to two backends during a migration, queue through a backend outage, and do tail sampling (which needs whole traces).",12
"Where do head and tail sampling belong?","Head in the SDK (cheap volume control at the source); tail in a gateway Collector, because the decision needs the whole trace and an app only sees its own spans. Decisive argument: head sampling decides BEFORE the error happens, so it keeps error traces only by chance. A tail policy of all errors + all slow + 2% kept 100% of error traces at lower volume than 5% head sampling.",12
"The three most common OpenTelemetry misconfigurations","Missing service.name (everything is unknown_service); SimpleSpanProcessor in production (synchronous export adds latency to every request); and an unbounded batch queue (a slow exporter grows memory until the process OOMs). Underneath all three: telemetry is best-effort and must fail open.",12
"Open vs closed load model","Closed: N virtual users each wait for a response, so arrival rate is a FUNCTION of response time and falls when the system slows: self-limiting, cannot reproduce overload. Open: requests arrive at a fixed rate regardless, so the queue grows. Internet traffic is open, so a closed test cannot find the failure you are testing for.",12
"What is coordinated omission?","A closed-loop generator waits for a response before sending the next, so during a stall it sends nothing and the slowest requests are never measured. 100 req/s for 200s with a 100s stall reports p99 ~1ms; measured against the intended schedule the same event has p99 ~98s. Same outage, two answers, entirely a property of the rig.",12
"How do you correct for coordinated omission?","Measure latency from the time a request was DUE per the schedule, not from when you managed to send it. Use a natively open+corrected generator (wrk2, Vegeta, k6 arrival-rate executors) or HdrHistogram's recordValueWithExpectedInterval. And always report dropped iterations: a run that dropped 12% did not apply the load you configured.",12
"How does an open-model load test silently become closed?","The generator runs out of workers. Configure 600 req/s with only 500 preallocated VUs and, once responses exceed a second, it cannot start iterations and reverts to closed-loop at exactly the moment the system degrades. One run dropped 61% of iterations and reported a passing p99 of 320ms. Treat >1% dropped iterations as an invalid run.",12
"Throughput vs goodput under overload","Goodput = responses delivered within the client's deadline. At 550 req/s one service completed 441/s of which only 88/s arrived within the 2s client timeout: 80% of capacity at the worst moment producing responses nobody received. That single line justifies load shedding, and it is invisible in a throughput-only report.",12
"How do you test for a metastable failure?","Push past the knee, hold, then drop back to a load the system previously handled comfortably, and see whether it recovers. Only an open model can do this. Sustaining loops to look for: unbounded queues holding requests whose clients timed out, unbudgeted retries multiplying arrival, and exhausted pools with long timeouts.",12
"The five rungs of the overload ladder","Do the work; degrade to a cheaper answer; backpressure (slow the producer); shed (reject cheaply, by priority); fail fast. Ordered by how much the user loses, and you climb only as far as you must, per request. Backpressure does not exist for a public API, because the producer is the internet.",14
"Why is queueing worse than rejecting under overload?","Once queue delay exceeds the client timeout, every response goes to a client that has left: throughput stays healthy while goodput is zero. The queue then prevents recovery, because draining doomed work consumes the capacity needed to catch up. Rejecting in 2ms costs the user the same and costs the system ~15,000x less.",14
"Why does LIFO beat FIFO under overload?","The oldest queued request is the one whose client most likely gave up. FIFO serves departed clients and yields zero goodput; LIFO serves the newest, whose clients are still waiting, and starves requests that were already doomed. Wrong under normal load, so use FIFO by default and switch above a delay threshold (CoDel).",14
"How do adaptive concurrency limits work?","TCP congestion control applied to a request limit: track the minimum observed RTT as the uncontended floor, compare to current RTT, and treat the ratio as a gradient. Near 1 means no queueing so raise the limit; well below 1 means a queue formed so lower it. AIMD on latency, no tuning, and it shrinks automatically when a downstream slows.",14
"Why must load shedding happen before any I/O?","A request rejected after partial work costs nearly as much as serving it, so late shedding relieves nothing. One implementation shed after an auth check that hit Redis, and under load the shed path's Redis traffic slowed auth for the requests NOT being shed, so the system degraded further as shedding engaged. The shed path is the hot path.",14
"Google's request criticality levels","CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE. Shed from the bottom up. The criticality MUST propagate with the request, or a service three hops down will shed a payment while preserving a prefetch. The propagation is the work; the shedding decision is trivial.",14
"The five fixes for CQRS eventual consistency","1) Return the result from the command (no read, nothing to be stale). 2) Client-side optimistic projection. 3) Version token: the write returns a version, the client sends it back, the read waits briefly for a projection at or past it. 4) Route that user's reads to the write model for a bounded window. 5) Make the lag explicit, or change the workflow so the read does not follow the write.",14
"What problem does CQRS actually introduce?","Read-your-own-writes for ONE user, not global convergence. Another user seeing a 200ms-old view is fine; the author seeing their own edit missing is never fine, and it causes a re-submit, which becomes duplicate data unless the operation is idempotent. Conflating the two leads teams to make everything synchronous to fix one interaction.",14
"Why not just make the projection synchronous?","Write latency absorbs every projection, write availability becomes the PRODUCT of every projection's availability (0.999^4 = 0.996), and a routine projection rebuild becomes a write outage. It is a legitimate place to sit at small scale (Level 1 on the ladder), not a fix at Level 4. Solve the interaction, not the architecture.",14
"How do version tokens stay affordable?","They are per-request: only reads needing the guarantee send one. In one system 6.1% of dashboard reads carried a token, of which 82% were served immediately, 17.7% waited (median 24ms) and 0.3% hit the 150ms deadline and were served stale with a flag. Overall p99 rose 4ms, against a synchronous projection which taxes every write.",14
"Vertical slice architecture: the actual argument","Layers impose a uniform abstraction cost on use cases with completely different needs, so a simple read and a 200-line report query must both fit one repository interface and one of them bypasses it. Slices let each use case pick its own abstraction level without any of them being an exception to a rule.",14
"What gets shared in a vertical-slice codebase?","Shared: the domain model and its invariants, cross-cutting infrastructure (auth, tracing, transactions, outbox), reused value objects, the schema. NOT shared: request/response shapes, validation, and queries. A shared response DTO between two use cases is how they become coupled, since they almost always want different fields.",14
"When is duplication between slices right?","When the two pieces will not change for the same reason, which is most of the time even when they look alike. In one codebase three similar 15-line queries were extracted into a shared query builder that grew 6 optional parameters and 2 flags within a month, and a change for one slice broke another. Removing duplication between slices should require a stated shared reason to change.",14
"Anti-pattern vs bug","An anti-pattern is a solution that looks correct, is adopted DELIBERATELY for a stated reason, and produces a worse outcome than doing nothing. A missing index is a bug; a generic repository over 34 entities that accumulated 61 bespoke methods and 9 documented bypasses is an anti-pattern.",14
"How do you use an anti-pattern name without it being name-calling?","The name is the index, not the argument. Say it to invoke the shared understanding, then state the concrete consequence here with a number. 'That's a dual write' invites disagreement; 'that's a dual write, and over 90 days we have 214 orders with no event and 31 events with no order' converts it into a data-integrity defect with a count.",14
"The event storming grammar","Orange = domain event (PAST TENSE), blue = command, yellow = actor, pink = external system, purple = policy ('whenever X, do Y'), green = read model, red = hot spot (disagreement or unknown), beige = aggregate. Events go up first, in time order; everything else is discovered by asking questions about them.",14
"Why must event-storming events be past tense?","Because a past-tense event either happened or it did not, so it can be placed on a timeline and its position argued about. 'Order processing' cannot be ordered relative to anything. The constraint is what makes the workshop converge, and the resulting arguments turn out to be business questions nobody had decided.",14
"The most reliable bounded-context signal on an event-storming wall","The language changing. If stickies on the left say 'customer' and on the right say 'consignee', you crossed a boundary. Pivotal events (the few everything organises around) are the second signal, because that is where the process hands over. Both beat drawing services around database nouns.",14
"How many people in an event storming workshop?","6 to 12, including 2-4 domain experts who actually do the work. Above ~15 the wall fragments into parallel conversations and stops converging: one team tried 19 and abandoned it after two hours, then re-ran with 9 successfully. The fix is more sessions, not a bigger room.",14
"How do you find aggregates, and what makes one wrong?","Group commands and events that must be transactionally consistent in the SAME INSTANT; each group is a candidate aggregate, as small as the invariants allow. A wrong one shows up as contention: one subscription row carrying billing cycle + box contents + shipment status meant two ops staff editing different boxes wrote the same row, which was a known unexplained 'optimistic concurrency' bug.",14
"Sidecar, ambassador, adapter: the distinction","One mechanism (a co-located process sharing the pod's network namespace and lifecycle), three roles. Ambassador proxies OUTBOUND connections so the app dials localhost. Adapter normalises OUTPUT into the platform's format. Sidecar is the general term. A service mesh is ambassadors everywhere plus a control plane.",14
"When does a service mesh beat libraries?","When a policy change or CVE fix would otherwise mean coordinating deploys across many services in several languages. The threshold is LANGUAGE COUNT and policy change rate, not service count: Netflix ran the library model (Ribbon/Hystrix/Eureka) successfully for years in a mostly-JVM estate.",14
"What drives per-sidecar memory, and how do you cut it?","Each proxy holds configuration for every destination it might reach, so memory scales with cluster size. Scope it (Istio's Sidecar resource) to only the namespaces that namespace actually calls, GENERATED FROM THE OBSERVED TRAFFIC GRAPH rather than team-declared dependencies. One estate: 68MB -> 21MB per proxy, 177GB -> 55GB total.",14
"The most dangerous service-mesh default","Retry on 5xx. The mesh makes retries a one-line config, which makes retry amplification a one-line config too. Retry only where the request provably did not reach the app (connect-failure, refused-stream). One fleet-wide retry-on-5xx default turned a 4-minute blip into a 22-minute outage.",14
"The Kubernetes sidecar lifecycle fix","Native sidecars: an initContainer with restartPolicy: Always (beta 1.28, stable 1.29). It starts before app containers and must be ready, runs alongside them, terminates AFTER them, and does not block pod completion. It replaced years of workarounds (startup polling scripts, preStop quit-endpoint curls) and fixes Jobs that never terminated.",14
"Which GoF patterns are still load-bearing?","Adapter (every external boundary; it is the anti-corruption layer), Decorator (every middleware stack), Strategy (usually just a function), Command (makes an intent queueable, retryable, undoable), Factory (mostly the DI container), Builder (where named args do not exist), Proxy (ORM lazy loading; a mesh sidecar).",14
"Why is Singleton criticised?","The uniqueness is fine; the global ACCESS POINT is the problem. It hides the dependency from every signature, makes tests order-dependent because they cannot substitute or reset it, and creates initialisation-order problems (Java's double-checked locking was broken pre-JSR-133). Inject it with singleton lifetime instead.",14
"Template Method vs Strategy","Template Method couples every subclass to a base-class shape that will change, and gives one axis of variation. In one payments codebase, adding a provider changed 24 files of which 11 were in the shared base class, because the skeleton assumed authorise-then-capture and the new provider only did a combined sale. Composed functions took shared-code changes to zero.",14
"The trade Visitor makes","Easy to add an OPERATION over existing types; hard to add a TYPE, since every visitor must change. Right for a stable hierarchy with many operations (an AST); wrong when the hierarchy grows. Sealed types with exhaustive pattern matching give the same guarantee with compiler enforcement.",14
"Why did several GoF patterns disappear?","They were workarounds for languages without first-class functions. Strategy and Command are function parameters with closures; Iterator became a language feature; Prototype became object literals. Norvig showed in 1996 that 16 of the 23 are simpler or invisible given first-class functions and dynamic dispatch.",14
"Why adopt a standard rather than design your own?","The tooling, more than the design time. OpenAPI gets you generated clients, mocks, request validation, contract tests and gateway config; a wiki page gets you nothing. And it makes an interface legible to someone who has never seen your system, which is what matters at an organisational boundary.",14
"What does CloudEvents give you, and not give you?","Gives: a standard envelope (id, source, type, time, subject, datacontenttype, dataschema) with transport bindings, so customers' existing tooling works. Does NOT give: anything about the payload, so additive-only schema evolution and a registry remain your problem. Write that into the ADR or people will later believe it solved compatibility.",14
"The load-bearing parts of OAuth","It is delegated AUTHORISATION, not authentication: an access token says what you may do, an OIDC ID token says who you are, and using an access token as identity proof is a real vulnerability. And PKCE, which OAuth 2.1 requires for ALL clients (not just public ones) while removing the implicit and password grants.",14
"Why does UUIDv7 matter?","v4 is random, so as a B-tree primary key every insert lands at a random position, dirties a different page and causes frequent page splits, making the insert working set the whole index. v7 (RFC 9562) is time-ordered so inserts append. One case: +38% insert throughput, -22% index size, -19% WAL, index cache hit 91% -> 99%, no migration (new partitions only).",14
"Three things worth knowing about JWTs","Historical vulnerabilities (alg: none, and RS256-verified-as-HS256 algorithm confusion) mean you must PIN the expected algorithm. No revocation, so a compromised token is valid until expiry. And they are usually chosen for sessions on unmeasured scalability grounds; they are genuinely right for short-lived stateless service-to-service assertions.",14
"Is SemVer a guarantee?","No, a communication convention. It depends on the publisher's judgment about what breaks, and Hyrum's law says every observable behaviour is depended on by someone, so a patch release breaks somebody. Pair it with a deprecation policy and a support window rather than treating the version number as a contract.",14
"The DR ladder, with RTO and cost","Backup/restore: hours-days, ~2-5% of prod cost. Pilot light (data replicating, no compute): tens of minutes to hours, ~10-20%. Warm standby (scaled-down but running): minutes, ~30-50%. Active-active: seconds, 200%+ PLUS a permanent engineering tax for write conflicts. Plus the forgotten rung: active-active reads with single-region writes.",17
"Why is DNS not a failover mechanism?","Failover time is set by caching you do not control: resolver minimum TTLs, OS and browser caches, negative caching. With a 60s TTL, moving 99% of traffic routinely takes 5-15 minutes with a tail of hours. One incident with a 300s TTL took 25 minutes. It also routes by RESOLVER location, not client location.",17
"What replaces DNS in the failover path?","A global L7 load balancer on an anycast IP that never changes, health-checking regional backends and forwarding over the provider backbone. Failover happens at the edge in seconds with no DNS change. One case: routing failover 25 minutes -> 6 seconds, at ~$3.1k/month against a $22k/hour downtime cost.",17
"The rule about RTO and routing","Your RTO cannot be shorter than your routing layer's failover time. A warm standby with a 90-second RTO behind a 300-second DNS TTL has a real RTO of many minutes. BGP anycast: seconds. Global L7 LB: seconds. DNS at 60s TTL: 5-15 minutes.",17
"What is wrong with most failover health checks?","They prove the process is up, not that the region can serve. In one incident /health returned 200 throughout because only NEW database connections were failing, so every dashboard said the region was healthy and the decision took 21 minutes. A failover check must assert dependencies (new connection + query, heartbeat write, replication lag, IdP reachable) and be SEPARATE from the shallow per-instance check.",17
"The biggest component of real-world RTO","The decision. One incident: detection 4 min, debate 21 min, database promotion 90 seconds (rehearsed and worked), routing 25 min. The team had rehearsed the mechanism and never the decision. Fix: a written threshold that is observable without diagnosis, one named role with authority, and no requirement for consensus.",17
"The clause that makes a failover threshold usable","'Failing over unnecessarily is an accepted cost. The IC will not be second-guessed for a failover that turned out to be avoidable.' The person with the authority also carries the career exposure. In one case that sentence, requested by the on-call rota, took decision time from 21 minutes to 70 seconds; the threshold alone would not have.",17
"What separates a runbook from documentation?","Copy-pasteable commands, a verification step after every action with expected output, explicit pre-conditions, a stop condition when verification fails, a duration estimate per step, and NO decisions inside it. A step saying 'assess whether' reintroduces the debate at the worst moment. It must be executable by someone who did not write it, at 3am.",17
"Why is failback harder than failover?","The recovered region is behind by the ENTIRE outage, not by the replication lag; the active region keeps moving while you prepare; it is planned, so an outage during it is inexcusable; and nobody practises it. One failback stalled its catch-up stream on an unalerted replication slot error and lost 40 minutes of billing events from view for three hours.",17
"How do you prevent split-brain?","In order of strength: quorum (a minority partition cannot accept writes: the only real guarantee, costs a third region or witness); fencing tokens (a monotonic epoch issued at promotion and checked at the storage layer, so a demoted primary physically cannot write); leases (clock-dependent, weaker); a procedural write freeze (what most systems have, only as strong as the procedure).",17
"Does GDPR require EU data residency?","No. It restricts TRANSFERS to third countries without adequacy or safeguards (SCCs plus a transfer impact assessment after Schrems II). The engineering requirement usually arrives as a contract clause from an enterprise customer's procurement team, which is commercial rather than statutory and creates the same work.",17
"How do you implement data residency cheaply?","Put the home region in the IDENTIFIER (eu1_01HQ8Z..., or a token claim), not in a lookup table, because a lookup is a cross-region dependency in the request path forever. Resolve once at the edge. Keep a minimal global directory of hashed login identifier -> region. In one case that decision made an 11-week project out of a 2-quarter estimate.",17
"The dependency audit: what does failover itself need?","Identity (can you log into the console if the primary is down?), DNS/global LB control plane, secrets manager, container registry, CI/CD, certificates, config and flags, observability, message queues, cron, object storage replication, and third-party SaaS. Run it as a walkthrough of the runbook, not a checklist: the checklist finds known dependencies, the walkthrough finds the rest.",17
"The circular-dependency rule","No credential required to recover system X may be stored in system X. Classic instances: the SSO provider you need to log in to fix the SSO provider; the break-glass credential for the secrets manager stored in that secrets manager; the pipeline that deploys the pipeline. Fix: a break-glass path depending on nothing in the loop, tested quarterly, paging on use.",17
"Why did a rehearsed runbook still have stuck steps?","Every game day failed over in the SAME DIRECTION, so it never required the primary region's dependencies to be unavailable. One team rehearsed four times and still had three stuck or blind steps (SSO, base container images and the entire observability stack were single-region in the primary). Alternate direction: it is free.",17
"Why is a replica not a backup?","It applies your mistakes at replication speed. A DROP TABLE, a bad migration or a ransomware encryption reaches the replica in milliseconds. A replica is availability; a backup is a point in the past. Human error is a more common disaster than region loss, which is why PITR granularity often matters more than retention duration.",17
"3-2-1-1-0, and the test for immutability","3 copies, 2 media/storage types, 1 offsite, 1 IMMUTABLE or air-gapped, 0 verification errors. The test: 'which single credential, if compromised, could destroy every copy?' If you can name one, you have one copy. S3 Object Lock in COMPLIANCE mode cannot be overridden even by root; GOVERNANCE mode can be bypassed with a permission.",17
"What should a restore test assert?","Three levels: integrity on every backup; a full TIMED restore monthly (whose output is the measured RTO you are allowed to claim); and business assertions on the restored data. The last catches the worst case: in one test row counts matched and the database opened cleanly, but ledger totals differed because of a stale table exclusion added 14 months earlier.",17
"The backup failure metrics miss","A job that succeeds while producing nothing. One case: a credential rotation removed read permission on a tablespace, pg_dump exited zero with a warning, and the backup was 12KB for nine days. Success rate was 100% and backup age was fine because the file was fresh. Only the monthly restore test caught it.",17
"Denylist or allowlist for backup contents?","Allowlist, generated from the schema, so a new table is included by default and an exclusion requires a comment and a date. A denylist silently omits every new table nobody remembers to add, and the omission is invisible until a restore produces a complete-looking, readable, wrong database.",17
"When does a Well-Architected pillar actually bind?","When it has a NAMED OWNER and a RECURRING FORCING FUNCTION. Security has an audit; cost has a monthly bill someone senior reads. Reliability binds only where an SLO with an error budget policy exists, otherwise it is an opinion. Operational excellence binds almost nowhere, because nobody receives a bill for toil.",20
"What does an account boundary give you that a tag cannot?","IAM (default deny across the boundary), SERVICE QUOTAS (per account: a staging load test cannot exhaust production's vCPU limit), structural cost attribution, bounded blast radius for a leaked credential, and reliable deletion by closing the account. Quota contention is the failure people do not anticipate, because there is no code path to find.",20
"What makes a guardrail different from a convention?","Evaluation order. An SCP / Azure Policy / GCP Org Policy is a deny evaluated ABOVE the account's own IAM, so a compromised account administrator cannot remove it. Keep the preventive list short (disable audit logging, root usage, unapproved regions, leaving the org) and make everything else detective, because a preventive policy that blocks legitimate work acquires an exception and then enforces nothing.",20
"How should Terraform state be split?","By lifecycle and blast radius, which usually coincide: account baseline / network / data / platform / per-app-per-environment. NOT by resource type: putting all IAM in one state and all networking in another means every new application touches every state, which is the same mistake as organising code by layer.",20
"Where does infrastructure drift actually come from?","Usually the tooling being unusable. One CloudTrail analysis of ~40 monthly production console changes: 31 were people avoiding a 14-minute plan and a 2-day queue, 6 were genuine emergencies, 3 were ignorance. The drift was a symptom, so removing console access alone (already tried, abandoned in 3 weeks) removes the workaround without fixing the cause.",20
"Should drift be auto-remediated?","No. Auto-apply will eventually revert a change someone made deliberately in an emergency, at the moment it is keeping the service up: one implementation did exactly that in week two, reverting an instance-size increase made 40 minutes earlier during a load spike. Detect on a schedule with plan -detailed-exitcode, alert, and require a human decision.",20
"What makes a good Terraform module?","It encapsulates a DECISION, not a resource. A bucket module with 40 passthrough variables is the resource with extra steps. A document-store module with three inputs encoding encryption, access logging, lifecycle and tags is worth versioning because upgrading it propagates an opinion. Smell: more than ~15 input variables means passthrough.",20
"The concrete cost of a passthrough module instead of a real one","Copied configuration that drifts. One Postgres module with 5 inputs replaced 400 lines repeated across 11 places, each subtly different, and three of the eleven had backup retention left at the 1-day default because the setting had been copied rather than encapsulated.",20
"The ordering rule for cloud cost work","Turn it off, right-size, MODERNISE, then commit. Committing first locks in the waste the earlier steps would have removed. In one case steps 1-3 saved $97k/month and the commitment saved a further $16.4k, so the discount was 14% of the total saving and the engineering work was 86%.",20
"How much capacity should you commit to?","The p5 of hourly usage over 90 days, not the mean. A commitment is a floor you pay for regardless, so every hour below it is waste at full price and every hour above it is served on-demand at a price you were already willing to pay. Target utilisation >95%; coverage is whatever the floor turns out to be.",20
"Coverage vs utilisation","Coverage = committed usage / total usage. Utilisation = commitment used / commitment purchased. They fail in opposite directions: low coverage with high utilisation is under-commitment; high coverage with LOW utilisation is paying for nothing at a discount. Alert on utilisation below 95%; never set a coverage target.",20
"Why default to 1-year no-upfront compute savings plans?","Three years outlives most architectural decisions and the family-scoped discount evaporates the first time you migrate. One team bought 3-year all-upfront RIs covering 85% of usage without asking about the roadmap, then deferred a funded Graviton migration by 14 months (~$406k unrealised) so the commitment could run down. A financial instrument dictated an engineering roadmap.",20
"The four practices that make spot work","1) Diversify across many (instance type, AZ) pools, because capacity is per pool. 2) Capacity-optimised allocation, not lowest-price. 3) Handle the 2-minute interruption notice with a real drain and checkpoint. 4) Keep an on-demand or committed baseline sized to absorb a simultaneous multi-pool eviction.",20
"When is multi-cloud justified?","A named requirement: a regulator or contract specifying a provider, an acquisition, a service with no equivalent, or a market where your provider is unavailable. NOT lock-in anxiety with no named risk, and not resilience against provider-wide outages, which are rarer than the outages the cross-provider complexity causes.",20
"What to do instead of multi-cloud","A costed exit inventory: every managed service, its equivalents elsewhere, and a MEASURED migration estimate, plus a tested data export with the duration measured. One took 3 weeks and produced 'roughly 3 engineer-years and 5-8 months'. Unpredicted second use: a credible costed exit is the only real leverage in a cloud contract, and it improved the next renewal by more than the multi-cloud plan would have saved.",20
"Where does a surprising data-transfer bill come from?","Not internet egress. One $47k/month breakdown: cross-AZ $22.4k, NAT gateway processing $14.1k, inter-region $4.3k, internet egress $6.2k. Three of four are internal and none appear on an architecture diagram. 61% of the NAT traffic was same-region object storage going out to the internet and back.",20
"The highest-return egress fix","A gateway/private endpoint for object storage. Traffic to the provider's own storage through a gateway endpoint is free of data processing charges; through a NAT gateway it costs per GB on top of egress. One case: one Terraform resource plus a route table association, worth $8,600/month, applied in an afternoon.",20
"Topology-aware routing: the saving and the risk","Prefer a same-zone endpoint when healthy. One case took the chattiest pairs from ~2/3 cross-zone to ~1/10, worth $13.9k/month. The risk: it reduces the effective load-balancing pool, so it can create hot spots and, with too few replicas per zone, black-hole a zone during a deploy (one rollout lost a zone's traffic for ~20s). Require a minimum healthy-endpoint count and rehearse the fallback.",20
"What is data gravity?","Data attracts compute: moving a petabyte costs money and weeks while moving compute is nearly free, so the next system gets built next to the data, and then the next one. The practical lock-in is the storage location, not the compute API. Mitigate by decoupling the query engine from the bytes (open table formats like Iceberg) plus a tested export path.",20
"Does Kubernetes make you cloud-portable?","It makes the WORKLOAD API portable and leaves storage classes, load balancer behaviour, IAM integration and every managed service provider-specific. Terraform gives you one language, not one semantic. The honest claim: these reduce a migration from a rewrite to a large project, which is worth something and is not portability.",20
"The six signals a staff loop scores","Scope (a service vs a problem space across teams), ambiguity (executes a spec vs produces one), judgment (picks a good option vs explains three plus the reversal criteria), influence (convinces their team vs peers who do not report to them), multiplier (delivers vs others got faster), risk (handles known risk vs names the one nobody else named).",1
"The most common way strong candidates fail a staff loop","Telling senior-level stories extremely well. The story is about WHAT THEY BUILT rather than HOW THE ORG CHANGED, so scope hears one service, influence hears nobody outside the team, and multiplier hears one person's output. Every clause is true and it argues for the level they already have.",1
"The strongest addition on the judgment axis","The reversal criteria: what would have to be true for you to change your mind, with a threshold. 'I'd revisit the single-writer design above ~40k writes/sec or if we needed sub-50ms writes in Europe.' It shows the decision was made against a model rather than a preference, and almost nobody offers it.",1
"How do you make influence visible in a story?","A named disagreement and what it cost you. 'The platform lead thought it was the wrong layer and he was partly right, so I moved the boundary he objected to, which cost a quarter, and he then co-signed it, which is what got the other two teams to move.' A story where everyone agreed contains no influence.",1
"What is actually scored in a code review round?","Prioritisation, not coverage. Finding four issues and ranking them beats finding twelve unranked. Work in order (correctness, security, design, tests, readability, nits), label every comment blocking/suggestion/nit/question out loud, and state the CONSEQUENCE on a blocking comment. One candidate found 5 of 6 issues and lost the round on presentation order.",1
"The single most common failure in an incident simulation","Going straight to root cause. Say 'I'd mitigate before I diagnose' in the first thirty seconds: mitigation is reversible and being down is not. One candidate spent 15 minutes on an excellent diagnostic walk, was right, and had done nothing about the outage. Also ask 'what changed in the last hour' immediately: most incidents are a change.",1
"What is graded in a take-home at staff level?","The README. The code is a gate; the write-up is the artifact. Scope small and ship complete, then document the decisions and what you traded away, what you deliberately did not do with the time budget stated, the failure modes, and what you would change at production scale.",1
"The durable question about a company's loop","Not how many rounds, but what that company believes goes wrong when they hire badly and which round exists to catch it. Amazon's Bar Raiser defends against a manager lowering the bar; Google's committee against a hire whose scope nobody can write down; two-problems-in-45-minutes against slowness; Stripe's debugging round against people who whiteboard and cannot ship.",1
"The highest-ratio preparation activity","One email to the recruiter: the round list with durations, whether coding is one problem or two per round, whether design is distributed-systems or domain-shaped, whether there is a take-home or debugging round, and the level and band. Five minutes to write. It changes the allocation more than any amount of general study, and almost nobody sends it.",1
"The differentiator that is close to decisive for a lead role","A one-page 30/60/90 plan sent after the onsite. First 30: listen and map, naming what you would read and who you would meet and what you would measure. 60: one visible fix. 90: a proposal. Include the caveat that half is probably wrong from the outside. It is the only artifact in the process showing what you would DO rather than what you have done.",1
"Why bring an artifact to the deep dive?","A sanitised one-page architecture diagram removes the interviewer's reconstruction load and changes the round from recall to discussion. Say that you sanitised it and what you removed, which is itself a judgment signal. Almost nobody does this.",1
"Interview-day stamina, and why it matters","The rounds late in a loop are usually the behavioural and hiring-manager rounds, which is where a staff decision is made. Eat before and at the break, refuse a working lunch, stand up between rounds, and do not review notes (it raises anxiety and changes nothing). One candidate was strong on two morning coding rounds and vague by the afternoon deep dive.",1
"Reconnection vs resumption","Reconnecting is re-establishing the transport, which every client library does. Resuming means the server can answer 'what did I miss after event 4711', which needs a monotonic per-stream id and a bounded replay buffer. Conflating them produces the silent gap: in one system 82% of reconnects lost events with no error and no metric movement.",9
"The branch everyone omits in a resume protocol","An explicit resync_required signal when the client's cursor is older than the buffer. Without it, a gap the client cannot detect is indistinguishable from no gap. That single branch is the difference between silent data loss and a handled case, and it requires the snapshot path to be cheap enough to be the fallback.",9
"Why must a stream event id be monotonic rather than a UUID?","Two operations need ordering: resume is a range query ('everything after 4711'), and gap detection is a client capability (receiving 4711 then 4713 tells it something was lost). With a UUID neither works. Use a per-stream sequence, a Redis stream id, a Kafka offset, or UUIDv7.",9
"Dedupe or idempotent application?","Idempotent application where events are state updates: apply by key if the version is newer, so re-delivery is a no-op with no id set and no capacity parameter. A dedupe set's capacity must exceed the largest possible replay (buffer window x peak rate); one proposal was 2,000 against a maximum of ~50,000. For cumulative events, send the total rather than the delta.",9
"What breaks a long-lived stream in production but not locally?","A load balancer idle timeout (fix: a heartbeat comment every 15-30s), proxy buffering (X-Accel-Buffering: no, proxy_buffering off, Cache-Control: no-transform), and the HTTP/1.1 six-connections-per-host browser limit that an open SSE stream consumes (HTTP/2 removes it). All three are absent in local development.",9
"Why does an LLM regression gate need a noise floor?","Because the metric is statistical. Run the unchanged system through the suite 10-12 times and take the standard deviation. One team's threshold was 0.7 sd, so it fired on ~half of all PRs, was overridden 31 times out of 34, and missed two real regressions that shipped anyway. Temperature 0 reduces but does not eliminate non-determinism.",5
"How large must an LLM eval set be?","n ~= 16 x p(1-p) / d^2. At p=0.85: ~200 examples to detect a 10-point drop, ~800 for 5 points, several thousand for 2. A 50-example suite reliably detects roughly a 20-point regression, which you would have noticed anyway.",5
"When is LLM-as-judge acceptable as a gate?","Only after measuring the judge's agreement with adjudicated human labels, on a NARROW question. One composite 'was this a good answer' score had Cohen's kappa 0.31; decomposed, the judge agreed at 0.68 on 'does this answer the question given this source' and poorly on tone and concision, so the latter two left the gate entirely.",5
"Hard gates vs statistical gates for LLM systems","Binary things gate at 100%: safety refusals, PII leakage, schema validity, citation ids resolving, and the regression set of past production failures. Aggregate metrics gate at a threshold above the measured noise. Conflating them produces a safety check with a tolerance. Push as much as possible into the deterministic layer, which has no noise floor at all.",5
"Why does finding the k LARGEST use a MIN-heap?","Because the operation you perform constantly is evicting the WEAKEST survivor, so the weakest must be at the top. O(n log k) time, and more importantly O(k) memory, which is what lets it run over a stream that does not fit in memory.",21
"What is the precondition for binary search?","A MONOTONIC PREDICATE, not a sorted array. A sorted array is the most common way to get one. A rotated sorted array is not sorted and is still binary searchable; 'search on the answer' problems have no array at all.",21
"Union-find complexity, and what you need for it","O(alpha(n)) amortised, inverse Ackermann, under 5 for any real n. Needs BOTH union by size and path compression; either alone gives O(log n). Write find() iteratively, since a degenerate chain blows the recursion limit.",21
"When is union-find the WRONG structure?","Three disqualifying cases: (1) edges can be REMOVED (no split operation; you need Euler tour or link-cut trees), (2) you need the path or distance, not the grouping, (3) the relation is not transitive, e.g. a similarity threshold, where it over-merges everything into one component.",21
"0/1 knapsack vs unbounded: what is the actual code difference?","The direction of the inner capacity loop. BACKWARD means best[c-w] does not yet include the current item, so each item is used at most once. FORWARD means it may, so items are reusable. One character changes which problem you solved.",21
"Is knapsack's O(nW) polynomial?","No, pseudo-polynomial. The input is O(n log W) bits because W is written in binary, so O(nW) is exponential in input LENGTH. 0/1 knapsack is NP-complete and the DP does not contradict that. Capacity of 10^9 makes the table impossible with only 20 items.",21
"Why does DFS cycle detection need three colours?","WHITE unvisited, GREY on the current recursion stack, BLACK finished. A cycle is an edge to a GREY node. Collapsing GREY and BLACK into one 'visited' flag reports a cycle on the diamond a->b, a->c, b->d, c->d, where none exists.",21
"Kahn's algorithm vs DFS post-order for topological sort","Both O(V+E). Kahn's gets cycle DETECTION free (any node left with positive in-degree is in or downstream of a cycle, and that set is a usable error message) and is naturally parallel: everything at in-degree zero can run at once. Default to Kahn's.",21
"The sweep line tie-break that decides the answer","Sort events as (time, delta) with -1 for END and +1 for START, so -1 sorts first and a meeting ending at t frees its room before one starting at t claims one. Flip it and back-to-back meetings each get their own room. Interviewers test exactly this input.",21
"Why can't you return the tails array as the LIS?","tails[k] is the smallest possible TAIL of an increasing subsequence of length k+1, not a path. On [10,9,2,5,3,7,101,18] it is [2,3,7,18], which is not a subsequence of the input in order. The LENGTH is right, which is why the bug survives.",21
"The five staff-level coding behaviours","(1) Contract and types before the algorithm, (2) tests unprompted, even three assertions, (3) complexity stated before coding and verified after, (4) an answer ready for 'what breaks at 100x', (5) no over-abstraction. You can solve the problem perfectly and still be under the bar for skipping these.",21
"'What breaks at 100x input?' How do you answer cold?","Walk the resources in order (memory, time, I/O, coordination) and ask of each: what grows? For a top-endpoints counter, memory grows with DISTINCT endpoints and time with lines, so if the endpoint set is unbounded (path parameters like /user/12345) memory is the wall, not speed.",21
"Trie vs hash map: the honest comparison","For exact lookup the hash map WINS (one hash, one probe, versus a pointer dereference and likely cache miss per character). The trie wins where the hash map offers nothing: prefix queries, longest prefix match, fuzzy match, sorted iteration. Choose it for capability, not speed.",21
"Backtracking's production failure mode","Backtracking regex engines (PCRE, Python re, Java, JS) blow up exponentially on patterns like (a+)+b, which is the ReDoS class. Cloudflare's 2 July 2019 global outage was a WAF regex that backtracked catastrophically. RE2 (Go's regexp) is linear because it refuses backreferences.",21
"Sliding window vs prefix sums for subarray sums","The window's correctness rests on MONOTONICITY: extending it must move the aggregate one way. Non-negative values give that; a single negative number breaks it and the window silently returns a wrong answer. If non-negativity is not guaranteed, use prefix sums with a hash map.",21
"Goodhart's law: what is the actual mechanism?","NOT cheating. A metric is a proxy that correlates with what you care about ACROSS THE BEHAVIOURS PEOPLE WERE EXHIBITING WHEN YOU MEASURED. Making it a target adds behaviours taken BECAUSE of the target, which were not in that sample. The correlation breaks under optimisation, with everyone acting in good faith. So a solution based on catching cheaters cannot work.",22
"Campbell's law vs Goodhart's law","Campbell (1979) is the sharper form and adds a second clause Goodhart lacks: the measurement distorts the ACTIVITY, not just the measure. Measuring review turnaround does not merely produce a useless number, it produces worse reviews.",22
"Project Oxygen: what did people get wrong about it?","Technical skill ranked LAST of the eight manager behaviours, and coaching first. But Google was ranking WITHIN a population of managers who all had substantial technical skill, so it is a THRESHOLD variable, not an irrelevance. Above the threshold, coaching differentiates; below it, nothing else helps.",22
"Project Aristotle: the finding and the misreading","Of ~180 teams, WHO was on the team predicted little; HOW the team operated predicted a lot. Five factors, psychological safety strongest, then dependability, structure and clarity, meaning, impact. The misreading: safety means niceness. A conflict-averse team scores LOW.",22
"Edmondson 1999: the counterintuitive result","She expected better hospital teams to make fewer errors. Better teams REPORTED more errors. The measure was reported errors and what varied was willingness to speak. So: a team with no reported incidents is not safe, it is opaque. Look for the presence of people mentioning problems, not the absence of problems.",22
"Normalization of deviance (Vaughan)","Challenger: O-ring erosion was out of spec, never caused a failure, and was progressively reclassified as acceptable. NOT people knowingly accepting catastrophic risk; their definition of 'acceptable' migrated one uneventful flight at a time. Diagnostic question: what are we doing now that we would not approve if we proposed it today?",22
"Why can you not add p90s?","The sum's p90 does not require every task to hit its own p90, only that the TOTAL is high, and in most such worlds some tasks run long while others run short. Independent variation cancels. Measured: summing six per-task p90s gave 59 days against a true p90 of 47.2, 25% too pessimistic.",22
"The single most alarming estimation number","Summing the MOST LIKELY estimates for six tasks gave 26 days, which sits at the FIRST PERCENTILE of the simulated distribution. ~1% chance of being met, with every individual estimate honest. Cause: durations are right-skewed (can take 5x, cannot take negative time) so mode < mean, and the gap accumulates once per task.",22
"One-way door vs two-way door: what is the prescription?","Match the PROCESS to the door, not to the perceived importance. Two-way doors: decide fast, by whoever is closest. One-way doors: analysis, a written argument, more people. The genuinely irreversible list is short: data you delete, data you leak, an API you publish, a promise to a customer, a person you lose.",22
"Backpressure vs load shedding: how do you choose?","Opposite responses to the same condition. Backpressure propagates slowness upstream, nothing lost, right when the producer CAN slow and the work must survive (pipelines). Load shedding discards work, right when the producer cannot be slowed and lateness is worthless (request paths). Default: shed on user-facing paths, backpressure on pipelines.",22
"Process vs mechanism (the interview distinction)","A process is steps people are expected to follow, and it lapses. A mechanism produces the outcome whether or not anyone remembers to care: named owner, trigger, forcing function, and an artifact someone would notice the absence of. 'We discussed it in retro' scores zero on 'how do you prevent recurrence'.",22
"Little's Law applied to a team","Cycle time = WIP / throughput. With throughput roughly fixed, halving work in progress halves cycle time, with nobody working harder. It is a theorem, not a heuristic, which is why a WIP limit is the most reliable intervention available to a new lead, and why teams resist it (starting feels productive, finishing feels slow).",22
"Theory of constraints, in one sentence","Improving anything other than the constraint improves nothing. Corollary: fixing the constraint just MOVES it, so predict the disappointment in advance. 'We sped up development and moved the bottleneck to code review.'",22
"Brooks's law: the mechanism, not the slogan","Adding people to a late project makes it later BECAUSE new people consume existing people's time to ramp up, and communication paths grow as n(n-1)/2 while capacity grows linearly. So it is false when work partitions cleanly and new people need little context, which is rarely true late in a project.",22
"Fail open vs fail closed: which default is more dangerous?","Fail OPEN, because its failure is silent. A fail-closed system that breaks causes an outage and everyone knows in minutes. A fail-open system that breaks keeps serving, and you find out months later that the authz check has returned true since March. Default closed unless there is a stated availability reason, and ALERT on every fail-open event.",22
"The four things behind 'we are moving too slowly'","A queue nobody is measuring, a decision nobody owns, a cost nobody has priced, or work in progress nobody has limited. Naming which one you are looking at is worth more than any individual term.",22
"Why name a 'tax' (coordination, coupling, integration, operational, carry)?","To convert a one-time-looking decision into a RECURRING cost. Build-vs-buy and split-vs-consolidate arguments are framed as one-time comparisons, in which the lower upfront option wins by construction. Naming a tax changes the units, and only then can the cost enter the comparison.",22
"Toil: the strict definition and why strictness matters","Manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly with service growth. Work failing those tests is not toil, it is just work you dislike. Google SRE caps it at 50% of time; the cap having a CONSEQUENCE is what makes it a constraint rather than an aspiration.",22
"Conway's law: the lead-relevant version","Not the observation, the inverse manoeuvre: since architecture follows team structure anyway, change the team structure to get the architecture you want. Corollary when you cannot restructure: predict the fracture and put a hard versioned contract at that boundary, or add the missing communication path (shared on-call, joint review).",22
"Wald and the bombers (survivorship bias)","Asked where to armour bombers given bullet-hole distributions on RETURNING planes, Wald said armour where there were NO holes: the sample excluded planes that did not come back, so damage in the observed areas was survivable. Engineering version: your latency percentiles exclude requests that timed out.",22