Senior 5–8 years

SR-E-O01 · Fire-and-forget submit() in an SMS notification batch

Easy · Output · Senior (5–8 years) · concurrent.futures, ThreadPoolExecutor, exception propagation, observability · 4 min

Question: An OTP/notification service sends SMS through a gateway using a thread pool. On-call says "the batch job reports success but some customers never got their SMS, and there's nothing in the logs." What does this print, and where did the error go?

from concurrent.futures import ThreadPoolExecutor


def send_sms(phone: str) -> str:
    if phone.endswith("0"):
        raise ConnectionError(f"gateway refused {phone}")
    return f"sent {phone}"


phones = ["9845012340", "9845012341", "9845012342"]
with ThreadPoolExecutor(max_workers=2) as pool:
    futures = [pool.submit(send_sms, p) for p in phones]
print("batch finished")

print([f.done() for f in futures])
print([type(f.exception()).__name__ for f in futures])
print(futures[1].result())
try:
    futures[0].result()
except ConnectionError as exc:
    print("caught:", exc)

Expected output:

batch finished
[True, True, True]
['ConnectionError', 'NoneType', 'NoneType']
sent 9845012341
caught: gateway refused 9845012340

Explanation:

pool.submit() returns a Future. If the callable raises, the worker thread catches the exception and stores it on the future; nothing is printed, logged or re-raised in the submitting thread. The exception surfaces only when someone calls future.result() (which re-raises it) or future.exception() (which returns it).

Leaving the with block calls shutdown(wait=True), so all three futures are done() by the time batch finished prints, and the batch "finishes" without a trace of the failure. exception() returns None for the successful futures, hence NoneType. Only the explicit result() call at the end brings the ConnectionError back into the caller's control flow.

This is the usual cause of "silent" failures in fire-and-forget pools: the code submits work, never inspects the futures, and the errors are garbage-collected along with them. executor.map() behaves differently: it re-raises the first failure when the iterator reaches that item, and you lose the results after it.

Takeaway: a submitted task's failure is data on its future. Every submit() needs a matching place that calls result() (or an add_done_callback that logs and counts failures); otherwise you have no error handling at all.

Follow-ups:

1. "How would you change this so every failure is logged and counted, without blocking the submitting code on each result?" Attach future.add_done_callback(on_done) where on_done calls fut.exception() and logs with the phone (masked) and a metric such as sms_send_failures_total{reason=...}. Or collect futures and iterate as_completed(futures), calling result() inside try/except per future. A callback runs in the worker thread (or immediately in the caller if the future is already done), so it must be cheap and must not raise.
2. "Why did the team pick threads rather than processes or asyncio here, and when would you revisit it?" Sending an SMS is I/O-bound: the thread spends its time waiting on the network with the GIL released, so threads give real concurrency with the existing blocking HTTP client. Processes add pickling and startup cost for no gain. asyncio would win at high fan-out (thousands of concurrent sends) with an async HTTP client, because threads cost memory and context switches; revisit when concurrency, not CPU, is the limit.
3. "The gateway is flaky and you add retries inside send_sms. What else must change?" Retries on a non-idempotent send can deliver duplicate OTPs. Send an idempotency/dedup key the gateway honours (or check delivery status before resending), retry only on retryable errors with backoff and jitter, and cap total time so the pool's threads aren't held for minutes.

Scoring: Strong: Exact output, explains that exceptions are stored on the future and only re-raised by result(), and names a concrete fix (done-callback or as_completed with per-future handling, plus a failure metric). Contrasts with map() re-raising. Acceptable: Gets the output after a moment's thought and knows result() re-raises; needs a prompt for how to make failures observable in production. Red flag: Expects the traceback to print or the with block to raise; or proposes wrapping the with block in try/except as the fix.

SR-E-O02 · Duplicate and missing log lines from logger propagation

Easy · Output · Senior (5–8 years) · logging, logger hierarchy, propagation, observability · 5 min

Question: The platform team configures the root logger at WARNING with one handler. The payments team adds its own handler to payments.refunds so they see INFO in their dashboard. Support now complains of duplicated lines in the central log and "missing" settlement logs. What does this print?

import logging
import sys

root = logging.getLogger()
root.setLevel(logging.WARNING)
root_handler = logging.StreamHandler(sys.stdout)
root_handler.setFormatter(logging.Formatter("ROOT  %(name)s: %(message)s"))
root.addHandler(root_handler)

log = logging.getLogger("payments.refunds")
log.setLevel(logging.INFO)
team_handler = logging.StreamHandler(sys.stdout)
team_handler.setFormatter(logging.Formatter("TEAM  %(levelname)s %(name)s: %(message)s"))
log.addHandler(team_handler)

log.info("refund queued id=R-101")
log.warning("refund retry id=R-101")

log.propagate = False
log.warning("refund failed id=R-101")

upi = logging.getLogger("payments.refunds.upi")
upi.debug("raw bank response")
upi.info("VPA verified")
logging.getLogger("payments").info("settlement batch started")
logging.getLogger("payments").warning("settlement late")

Expected output:

TEAM  INFO payments.refunds: refund queued id=R-101
ROOT  payments.refunds: refund queued id=R-101
TEAM  WARNING payments.refunds: refund retry id=R-101
ROOT  payments.refunds: refund retry id=R-101
TEAM  WARNING payments.refunds: refund failed id=R-101
TEAM  INFO payments.refunds.upi: VPA verified
ROOT  payments: settlement late

Explanation:

Two rules explain every line:

1. The level check happens once, on the logger you call. A record is created only if it passes the effective level of the logger it was logged on (its own level, or the nearest ancestor's that is set). After that, the record is handed to that logger's handlers and then, while propagate is true, to each ancestor's handlers. Ancestor logger levels are not re-checked; only handler levels are, and these handlers are at NOTSET. That is why the INFO "refund queued" reaches the root handler even though root is at WARNING: two lines per call, and the duplicates in the central log.
2. propagate = False stops the walk up the tree at that logger. "refund failed" goes only to the team handler and disappears from the central log.

payments.refunds.upi has no level, so its effective level is INFO (inherited from payments.refunds): DEBUG is dropped, INFO is created and handled by the parent's team handler, and the walk stops there because the parent has propagate = False. payments has no level either, so it inherits WARNING from root: "settlement batch started" is dropped at creation (the "missing" log), and "settlement late" reaches the root handler.

Takeaway: in a service, attach handlers in one place (the root, configured at startup via dictConfig) and control verbosity with logger levels; libraries and team modules should only call getLogger(__name__) and never add handlers.

Follow-ups:

1. "How would you let the payments team see INFO for their loggers in the central sink without duplicates?" Remove their handler and set logging.getLogger('payments').setLevel(logging.INFO) in the central config. The records are created at INFO and propagate once to the root handler. If they need a separate destination, put a second handler on root with a Filter on record.name.startswith('payments'), or route by logger name in the log pipeline instead of in-process.
2. "What should a library you publish for other teams do about logging?" Use logging.getLogger(__name__), never configure handlers or levels, and add a NullHandler to the package's top-level logger to avoid the 'last resort' handler printing warnings. Configuration belongs to the application.
3. "Your service logs ~20k lines/s and p99 latency rises when the log disk is slow. What do you change?" Logging handlers run synchronously in the calling thread. Use a QueueHandler on the hot path with a QueueListener writing to the real handler in a background thread (bounded queue, decide drop vs block), log structured JSON to stdout for the collector, and cut volume: sample or drop DEBUG/INFO in hot loops.

Scoring: Strong: All seven lines exact. States that ancestor levels are not consulted during propagation (only handler levels), explains the effective-level inheritance for upi and payments, and recommends handlers-at-root-only as the standard. Acceptable: Spots the duplication and the effect of propagate = False, but expects the root's WARNING level to block the propagated INFO; corrects with a hint about where the level check happens. Red flag: Treats loggers as independent (no hierarchy) or 'fixes' duplicates by setting propagate = False everywhere.

SR-E-O03 · Request IDs in contextvars across asyncio tasks

Easy · Output · Senior (5–8 years) · contextvars, asyncio, tasks, observability, request tracing · 5 min

Question: An async API tags every log line with a request ID stored in a ContextVar, so a single request can be followed through the logs. What does this print?

import asyncio
import contextvars

request_id = contextvars.ContextVar("request_id", default="-")


def log(msg: str) -> None:
    print(f"[{request_id.get()}] {msg}")


async def handle(rid: str, delay: float) -> None:
    request_id.set(rid)
    log("start")
    await asyncio.sleep(delay)
    log("done")


async def main() -> None:
    request_id.set("boot")
    await asyncio.gather(handle("req-A", 0.05), handle("req-B", 0.01))
    log("after gather")
    await handle("req-C", 0)
    log("after direct await")


asyncio.run(main())
log("outside the loop")

Expected output:

[req-A] start
[req-B] start
[req-B] done
[req-A] done
[boot] after gather
[req-C] start
[req-C] done
[req-C] after direct await
[-] outside the loop

Explanation:

Every asyncio.Task runs in its own copy of the context, taken when the task is created. gather() wraps each coroutine in a task, so handle("req-A") and handle("req-B") each set the variable in their own copy. While they interleave at await asyncio.sleep(...), each still sees its own ID ("req-B" finishes first because its sleep is shorter). Their set() calls never touch main's context, so main still sees boot.

Awaiting a coroutine directly does not create a task. await handle("req-C", 0) runs in main's own context, so its set("req-C") is still visible in main afterwards: the ID "leaks" into whatever main logs next.

asyncio.run() itself runs main() as a task in a copied context, so boot never reaches the module level, which still sees the default -.

This is why ContextVar (not a global, not threading.local) is the right tool for request IDs and tenant IDs in async code: a thread-local would be shared by every coroutine on the loop's single thread. Takeaway: a task gets a snapshot of the context at creation; a directly awaited coroutine shares its caller's.

Follow-ups:

1. "A handler offloads work with loop.run_in_executor(None, fn) and the logs from fn show [-]. Why, and what's the fix?" run_in_executor runs fn in a pool thread with that thread's context, not the caller's. asyncio.to_thread(fn) copies the current context for you (it uses contextvars.copy_context().run). With a raw executor, submit ctx.run, fn where ctx = contextvars.copy_context().
2. "How would you make middleware set the request ID so it can't leak between requests?" Set it at the start of each request and reset it in finally with the token: token = request_id.set(rid) ... request_id.reset(token). Frameworks usually run each request in its own task, but resetting is cheap insurance for code paths (tests, background loops) that reuse a context. Then a logging.Filter copies request_id.get() onto every record so the formatter or JSON encoder emits it.
3. "The request fans out to three downstream services. How does the ID reach their logs?" Propagate it on the wire: a header such as X-Request-ID or, better, W3C traceparent via OpenTelemetry, whose context API is itself built on contextvars. The outgoing HTTP client reads the context var and injects the header; the downstream service's middleware extracts it and sets its own context var.

Scoring: Strong: All nine lines exact. Explains task context copying, why the direct await leaks req-C into main, and why asyncio.run keeps boot from the module level. Contrasts with threading.local and mentions to_thread vs run_in_executor. Acceptable: Gets the gather part right, misses the leak from the direct await or the final [-]; understands once tasks vs plain awaits are pointed out. Red flag: Expects the IDs to clobber each other like a global, or suggests a module-level dict keyed by task as the design.

SR-E-O04 · State and pickling across a ProcessPoolExecutor boundary

Easy · Output · Senior (5–8 years) · multiprocessing, ProcessPoolExecutor, pickling, per-process state, CPU-bound work · 5 min

Question: A scoring job moves CPU-heavy feature computation into a process pool. A developer keeps an audit list of everything scored in a module global. What does this print?

import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor

seen: list[int] = []


def score(batch: list[int]) -> tuple[int, int]:
    seen.extend(batch)
    return sum(batch), len(seen)


if __name__ == "__main__":
    ctx = mp.get_context("spawn")
    with ProcessPoolExecutor(max_workers=1, mp_context=ctx) as pool:
        print(list(pool.map(score, [[1, 2], [3, 4], [5]])))
    print(seen)
    print(score([10]))

    with ProcessPoolExecutor(max_workers=1, mp_context=ctx) as pool:
        future = pool.submit(lambda x: x * 2, 21)
        try:
            future.result()
        except Exception as exc:
            print(type(exc).__name__)

Expected output:

[(3, 2), (7, 4), (5, 5)]
[]
(10, 1)
PicklingError

Explanation:

Processes don't share memory; everything crossing the boundary is pickled. The pool sends each batch to a worker process by pickling it, and pickles the return value back. The worker has its own copy of the module, including its own seen list.

• With max_workers=1, all three batches go to the same worker, which keeps its state between tasks: seen grows to 2, 4, then 5 items inside the worker. map returns results in input order.
• The parent's seen is untouched: []. Calling score([10]) in the parent appends to the parent's list, so it reports length 1.
• A lambda can't be pickled (pickle stores functions by importable name, and <lambda> isn't one), so the submit fails with PicklingError, raised from future.result().

The spawn start method (the default on macOS and Windows; Linux defaults to fork up to 3.13 and forkserver from 3.14) starts a fresh interpreter that imports the main module, which is why the if __name__ == "__main__": guard is required. Spawn is used here so the behaviour is the same on every OS.

Takeaway: in a process pool, task functions must be importable top-level functions, arguments and results must be picklable (and cheap to pickle), and worker globals are per-process caches, not shared state. Results come back as return values.

Follow-ups:

1. "Each task needs a 400 MB model. How do you avoid loading or sending it per task?" Load it once per worker with ProcessPoolExecutor(initializer=load_model, initargs=(path,)) storing it in a worker global, and send only small inputs (IDs, file paths, or array chunks) per task. With fork on Linux you can also load before creating the pool and rely on copy-on-write, but refcount updates touch pages and gradually copy them, and fork with threads is unsafe; prefer the initializer, or shared memory or memory-mapped arrays for large read-only data.
2. "When is a process pool the wrong choice even for CPU-heavy work?" When per-task data is large relative to compute (pickling dominates), when tasks are tiny (IPC overhead per task; batch with chunksize), or when the heavy work is already in NumPy/pandas/C code that releases the GIL, where threads or vectorisation get the speedup without copying. Measure first: profile to confirm it's Python-level CPU, then compare wall time.
3. "A worker process is killed by the OOM killer mid-task. What does the caller see?" The pool becomes broken: pending and future result() calls raise concurrent.futures.process.BrokenProcessPool, and the executor can't be reused. Production code catches it, recreates the pool, and resubmits the affected tasks, which must therefore be idempotent. max_tasks_per_child (3.11+) can recycle workers that leak memory.

Scoring: Strong: Exact output, including that the single worker accumulates state across tasks. Explains pickling of arguments and results, per-process globals, why lambdas fail, and the main guard under spawn. Mentions initializer for per-worker setup. Acceptable: Knows the parent's list stays empty and the lambda fails, but predicts (3, 2), (7, 2), (5, 1) (a fresh worker per task); corrects once told the pool reuses one worker. Red flag: Expects the parent's seen to contain the items, or thinks processes share globals like threads.

SR-E-C01 · A latency-metric context manager that never breaks the request

Easy · Coding · Senior (5–8 years) · observability, metrics, context managers, contextlib, monotonic clocks, testability · 12 min

Problem:

Your team is standardising latency metrics across services. Write timed, used as a context manager or a decorator, that measures how long a block takes and calls emit(metric, duration_ms, tags) exactly once per block.

Requirements: - tags is the caller's keyword tags plus outcome="ok", or outcome="error" and error_type=<exception class name> when the block raises. - The block's exception must propagate unchanged; timing must never swallow it. - Telemetry must never break business logic: if emit itself raises, the block's result or exception is unaffected. - Use a monotonic clock by default (time.perf_counter), injectable for tests. - As a decorator, each call is timed separately.

Signature: @contextmanager def timed(metric: str, emit: Callable[[str, float, dict[str, str]], None], *, clock: Callable[[], float] = time.perf_counter, **tags: str) -> Iterator[None]:

Constraints: Standard library only. emit may be slow or failing (a StatsD/OTel exporter). Must work for sync code; async support is a follow-up.

Examples:

• with timed('db.query', emit, table='orders'): run_query() # took 250 ms → emit('db.query', 250.0, {'table': 'orders', 'outcome': 'ok'})
• with timed('db.query', emit): raise TimeoutError() # after 500 ms → emit('db.query', 500.0, {'outcome': 'error', 'error_type': 'TimeoutError'}), then TimeoutError propagates

Hints:

1. Which construct runs on both the normal and the exceptional exit of a yield inside @contextmanager?
2. Why is time.time() the wrong clock for durations?
3. What should happen if emit raises while an exception from the block is already propagating?
4. @contextmanager objects are also ContextDecorators, and they create a fresh generator per decorated call.

Reference solution:

import logging
import time
from collections.abc import Callable, Iterator
from contextlib import contextmanager

logger = logging.getLogger(__name__)

Emit = Callable[[str, float, dict[str, str]], None]


@contextmanager
def timed(
    metric: str,
    emit: Emit,
    *,
    clock: Callable[[], float] = time.perf_counter,
    **tags: str,
) -> Iterator[None]:
    """Time the enclosed block and emit one latency sample, in milliseconds.

    Works as a context manager or a decorator. The block's exception always
    propagates; a failing `emit` is logged and ignored so telemetry can never
    change business behaviour.
    """
    outcome: dict[str, str] = {"outcome": "ok"}
    start = clock()
    try:
        yield
    except BaseException as exc:          # includes CancelledError, KeyboardInterrupt
        outcome = {"outcome": "error", "error_type": type(exc).__name__}
        raise
    finally:
        duration_ms = (clock() - start) * 1000.0
        try:
            emit(metric, duration_ms, {**tags, **outcome})
        except Exception:
            logger.warning("metric emit failed for %s", metric, exc_info=True)

Complexity: O(1) time and space per timed block, plus whatever emit costs. The overhead that matters in practice is emit: it runs inline, so a slow exporter adds directly to request latency (see follow-ups).

Tests:

import inspect
import time


class FakeClock:
    def __init__(self, *ticks: float) -> None:
        self.ticks = list(ticks)

    def __call__(self) -> float:
        return self.ticks.pop(0)


events = []


def emit(metric, ms, tags):
    events.append((metric, round(ms, 3), tags))


# success path, caller tags kept
with timed("db.query", emit, clock=FakeClock(10.0, 10.25), table="orders"):
    pass
assert events == [("db.query", 250.0, {"table": "orders", "outcome": "ok"})]

# error path: still emitted once, exception propagates unchanged
events.clear()
err = TimeoutError("db slow")
try:
    with timed("db.query", emit, clock=FakeClock(1.0, 1.5)):
        raise err
except TimeoutError as caught:
    assert caught is err
else:
    raise AssertionError("exception was swallowed")
assert events == [("db.query", 500.0, {"outcome": "error", "error_type": "TimeoutError"})]


# a failing emitter must not break the block or mask its exception
def broken_emit(*args):
    raise RuntimeError("statsd down")


with timed("x", broken_emit, clock=FakeClock(0.0, 1.0)):
    result = 42
assert result == 42
try:
    with timed("x", broken_emit, clock=FakeClock(0.0, 1.0)):
        raise KeyError("sku")
except KeyError:
    pass
else:
    raise AssertionError("KeyError lost")

# decorator use: each call timed separately
events.clear()
clock = FakeClock(0.0, 0.1, 5.0, 5.3)


@timed("pricing.quote", emit, clock=clock)
def quote() -> str:
    return "₹499"


assert quote() == "₹499" and quote() == "₹499"
assert [e[1] for e in events] == [100.0, 300.0]
assert all(e[2] == {"outcome": "ok"} for e in events)

# default clock is monotonic
assert inspect.signature(timed).parameters["clock"].default in (time.perf_counter, time.monotonic)

A common wrong solution (the tests catch it):

import time
from collections.abc import Callable, Iterator
from contextlib import contextmanager


@contextmanager
def timed(metric, emit, *, clock: Callable[[], float] = time.perf_counter, **tags) -> Iterator[None]:
    start = clock()
    yield                                   # no try/finally: errors skip the emit
    emit(metric, (clock() - start) * 1000.0, {**tags, "outcome": "ok"})

Alternatives and tradeoffs:

• A class with __enter__/__exit__ (subclassing ContextDecorator): more explicit, and lets you expose elapsed to the caller after the block. Slightly more code; same semantics. Note that a single class instance used as a decorator is shared across calls, so per-call state must not live on the instance; @contextmanager avoids this by creating a fresh generator for every decorated call.
• OpenTelemetry spans / prometheus_client.Histogram.time(): in a real service, use the standard library of your metrics stack rather than a home-grown emitter; this exercise is the core of what those do.
• time.time() is wrong for durations: wall-clock time can jump backwards or forwards (NTP corrections), producing negative or huge latencies. perf_counter is monotonic and high-resolution.

Follow-ups:

1. "Make it work on async def functions as a decorator." @contextmanager used as a decorator on a coroutine function only times the creation of the coroutine object, not its execution. Write an @asynccontextmanager twin, or a decorator that checks inspect.iscoroutinefunction(fn) and wraps with an async def that does with timed(...): return await fn(...). Count CancelledError as its own outcome (cancelled), since client disconnects shouldn't page anyone as errors.
2. "The exporter is slow (5 ms per emit). How do you keep it off the request path?" Emit into an in-process aggregator (counters and histogram buckets updated under a lock, or per-thread) and flush in a background thread every few seconds; or use UDP StatsD, which is fire-and-forget. Never do a synchronous network call per sample on the hot path.
3. "A teammate tags each sample with user_id and order_id. What do you say in review?" That's unbounded label cardinality: every distinct value creates a new time series and will blow up the metrics backend's memory and cost. Keep tags low-cardinality (endpoint, status class, region); put IDs in traces and logs, which are built for high cardinality.

Scoring: Strong: Uses try/except/finally so exactly one sample is emitted on both paths, re-raises the original exception, isolates emit failures, defaults to a monotonic clock, and tests with an injected clock. Raises the async-decorator pitfall and cardinality without prompting. Acceptable: Correct success and error paths with finally, but lets an emit failure escape or uses time.time(); fixes both when asked. Red flag: Emits only on success, swallows the block's exception, or tests by sleeping and asserting on real elapsed time.

SR-E-C02 · Idempotent ledger apply for at-least-once wallet events

Easy · Coding · Senior (5–8 years) · idempotency, at-least-once delivery, data integrity, atomicity, API design · 15 min

Problem:

A wallet service consumes credit/debit events from a queue with at-least-once delivery: the same event can arrive again in a later batch (redelivery after a consumer crash) or twice in the same batch. Implement Ledger.apply(events) so that:

• Each event_id affects balances at most once, across all calls. Re-applying a batch that was already applied is a no-op.
• A replayed event_id whose payload differs from the one already applied (or staged in this batch) is a contract violation: raise ConflictingEventError.
• A batch is all-or-nothing: if it raises, no balance changes and no event is marked as applied, so the consumer can safely retry or dead-letter it.
• apply returns how many events were newly applied.

Amounts are integer paise (₹1 = 100 paise); debits are negative.

Signature: def apply(self, events: Iterable[Event]) -> int:

Constraints: In-memory state is fine for the exercise; say how it maps to a database in production. Batches can be up to 10,000 events. Standard library only.

Examples:

• ledger.apply([Event('e1', 'acct-asha', 50_000), Event('e2', 'acct-asha', -12_500)]) → 2; balances == {'acct-asha': 37_500}
• ledger.apply([Event('e2', 'acct-asha', -12_500), Event('e3', 'acct-ravi', 1_000)]) # e2 redelivered → 1; balances == {'acct-asha': 37_500, 'acct-ravi': 1_000}
• ledger.apply([Event('e3', 'acct-ravi', 9_999)]) → ConflictingEventError; balances unchanged

Hints:

1. Where must the record of applied IDs live so a later batch can see it?
2. What should a redelivered event be compared against to detect a conflict?
3. To get all-or-nothing, validate the whole batch before mutating anything.

Reference solution:

from collections.abc import Iterable
from dataclasses import dataclass


@dataclass(frozen=True)
class Event:
    event_id: str
    account: str
    amount_paise: int


class ConflictingEventError(ValueError):
    """Same event_id seen with a different payload: a producer bug, not a retry."""


class Ledger:
    def __init__(self) -> None:
        self.balances: dict[str, int] = {}
        self._applied: dict[str, Event] = {}

    def apply(self, events: Iterable[Event]) -> int:
        """Apply a batch exactly once per event_id; all-or-nothing per batch."""
        staged: dict[str, Event] = {}
        for event in events:
            previous = self._applied.get(event.event_id) or staged.get(event.event_id)
            if previous is None:
                staged[event.event_id] = event
            elif previous != event:
                raise ConflictingEventError(
                    f"{event.event_id}: {previous} already recorded, got {event}"
                )
        # Validation passed: commit balances and the applied-ID record together.
        for event in staged.values():
            self.balances[event.account] = self.balances.get(event.account, 0) + event.amount_paise
        self._applied.update(staged)
        return len(staged)

Complexity: O(n) time per batch with dict lookups; O(n) extra space for the staged batch, plus O(total distinct events) for the applied-ID record, which grows forever unless bounded (see follow-ups).

Tests:

ledger = Ledger()
batch1 = [Event("e1", "acct-asha", 50_000), Event("e2", "acct-asha", -12_500)]
assert ledger.apply(batch1) == 2
assert ledger.balances == {"acct-asha": 37_500}

# full redelivery of a previous batch is a no-op
assert ledger.apply(batch1) == 0
assert ledger.balances == {"acct-asha": 37_500}

# partial redelivery across batches, plus a duplicate inside the batch
e3 = Event("e3", "acct-ravi", 1_000)
assert ledger.apply([Event("e2", "acct-asha", -12_500), e3, e3]) == 1
assert ledger.balances == {"acct-asha": 37_500, "acct-ravi": 1_000}

# a generator works (single pass)
assert ledger.apply(Event(f"g{i}", "acct-meera", 100) for i in range(3)) == 3
assert ledger.balances["acct-meera"] == 300

# conflicting replay: error, and the whole batch is rejected
before = dict(ledger.balances)
try:
    ledger.apply([Event("e4", "acct-ravi", 500), Event("e3", "acct-ravi", 9_999)])
except ConflictingEventError:
    pass
else:
    raise AssertionError("conflict not detected")
assert ledger.balances == before
# e4 was not marked applied, so a corrected retry applies it once
assert ledger.apply([Event("e4", "acct-ravi", 500)]) == 1
assert ledger.balances["acct-ravi"] == 1_500

# conflict inside a single batch
try:
    Ledger().apply([Event("x", "a", 1), Event("x", "a", 2)])
except ConflictingEventError:
    pass
else:
    raise AssertionError("in-batch conflict not detected")

assert Ledger().apply([]) == 0

A common wrong solution (the tests catch it):

from collections.abc import Iterable
from dataclasses import dataclass


@dataclass(frozen=True)
class Event:
    event_id: str
    account: str
    amount_paise: int


class ConflictingEventError(ValueError):
    pass


class Ledger:
    def __init__(self) -> None:
        self.balances: dict[str, int] = {}

    def apply(self, events: Iterable[Event]) -> int:
        seen: dict[str, Event] = {}           # dedupes within the batch only
        for event in events:
            if event.event_id in seen:
                if seen[event.event_id] != event:
                    raise ConflictingEventError(event.event_id)
                continue
            seen[event.event_id] = event
            self.balances[event.account] = self.balances.get(event.account, 0) + event.amount_paise
        return len(seen)

Alternatives and tradeoffs:

• Database (the production answer): a processed_events(event_id PRIMARY KEY, payload_hash) table written in the same transaction as the balance updates. INSERT ... ON CONFLICT DO NOTHING (or catch the unique violation) gives dedup, the transaction gives all-or-nothing, and it works across consumer replicas. Storing a payload hash lets you detect conflicts without keeping full payloads.
• Per-account sequence numbers instead of an ID set: apply only if seq == last_seq + 1. Constant state per account and detects gaps, but requires the producer to assign ordered sequences.
• Transactional outbox / exactly-once in the broker (e.g. Kafka transactions) moves the problem but doesn't remove it: side effects outside the broker still need idempotency.

Follow-ups:

1. "Two consumer replicas process overlapping batches concurrently. Does this design still hold?" Not in memory: each replica has its own _applied. With the DB version, the unique key on event_id makes the second insert fail or no-op inside its transaction, so only one replica applies the event. Lock ordering on account rows (or SELECT ... FOR UPDATE in a consistent order) avoids deadlocks when two batches touch the same accounts.
2. "The applied-ID record grows by 50 million events a day. How do you bound it?" Keep IDs only for the redelivery window the broker can produce (e.g. 7 days, partitioned by date and dropped), and reject or alert on events older than the window. Or use per-account sequence numbers. Document the window as part of the data contract with producers.
3. "Why raise on a conflicting replay instead of just ignoring it like other duplicates?" Same ID with different content is not a retry; it's a producer bug or an ID collision, and silently keeping the first version hides a money discrepancy. Failing loudly sends the batch to a dead-letter queue with an alert, so someone reconciles it.

Scoring: Strong: Persists applied IDs across calls, detects conflicting payloads, validates before mutating so a failed batch leaves no trace, handles in-batch duplicates and one-shot iterables. Maps it to a unique key in the same DB transaction without being asked. Acceptable: Cross-batch dedup works but the batch is partially applied on conflict; fixes the ordering once the all-or-nothing requirement is pointed at. Red flag: Dedupes only within a batch, uses floats for money, or proposes 'just make delivery exactly-once' instead of idempotent consumption.

SR-E-C03 · Bounded thread fan-out that keeps every result and every error

Easy · Coding · Senior (5–8 years) · concurrency, ThreadPoolExecutor, I/O-bound work, error isolation, API design · 15 min

Problem:

A KYC service must fetch documents for a list of customer IDs from a slow internal HTTP service (I/O-bound, a blocking client). Write fetch_all that calls fetch(key) for every key concurrently with at most max_workers calls in flight, and returns one Result per key in input order. A failure for one key must not lose the results of the others: record the exception in that key's Result instead.

max_workers < 1 raises ValueError. An empty input returns [].

Signature: def fetch_all(keys: Sequence[K], fetch: Callable[[K], V], *, max_workers: int = 8) -> list[Result[K, V]]:

Constraints: Up to a few thousand keys per call. Standard library only. fetch is blocking and thread-safe. Catch Exception, not BaseException.

Examples:

• fetch_all(['c1', 'c2', 'c3'], fetch) # fetch('c2') raises TimeoutError → [Result('c1', doc1, None), Result('c2', None, TimeoutError(...)), Result('c3', doc3, None)]
• fetch_all([], fetch) → []

Hints:

1. Is this CPU-bound or I/O-bound, and what does that say about threads vs processes?
2. What does executor.map do when one call raises?
3. submit returns futures; you can keep them in a list aligned with the keys.

Reference solution:

from collections.abc import Callable, Sequence
from concurrent.futures import ThreadPoolExecutor
from typing import Generic, NamedTuple, TypeVar

K = TypeVar("K")
V = TypeVar("V")


class Result(NamedTuple, Generic[K, V]):
    key: K
    value: V | None
    error: Exception | None

    @property
    def ok(self) -> bool:
        return self.error is None


def fetch_all(
    keys: Sequence[K],
    fetch: Callable[[K], V],
    *,
    max_workers: int = 8,
) -> list[Result[K, V]]:
    """Call `fetch` for every key on a bounded thread pool.

    Results are returned in input order. A failing key produces a Result with
    `error` set; it never hides the other keys' results.
    """
    if max_workers < 1:
        raise ValueError(f"max_workers must be >= 1, got {max_workers}")
    if not keys:
        return []
    with ThreadPoolExecutor(max_workers=min(max_workers, len(keys)),
                            thread_name_prefix="fetch_all") as pool:
        futures = [pool.submit(fetch, key) for key in keys]
    results: list[Result[K, V]] = []
    for key, future in zip(keys, futures):
        exc = future.exception()
        if exc is None:
            results.append(Result(key, future.result(), None))
        elif isinstance(exc, Exception):
            results.append(Result(key, None, exc))
        else:                                   # KeyboardInterrupt and friends
            raise exc
    return results

Complexity: Wall time is about ceil(n / max_workers) times the per-call latency for uniform calls; CPU work is O(n). Memory is O(n) for the futures and results. Threads cost roughly a stack each (reserved, mostly not committed), which is why max_workers stays in the tens, not thousands.

Tests:

import threading
import time

def fake_fetch(key: str) -> str:
    if key.startswith("bad"):
        raise TimeoutError(f"upstream timeout for {key}")
    return f"doc:{key}"

out = fetch_all(["c1", "bad-2", "c3", "bad-4", "c5"], fake_fetch, max_workers=3)
assert [r.key for r in out] == ["c1", "bad-2", "c3", "bad-4", "c5"]
assert [r.value for r in out] == ["doc:c1", None, "doc:c3", None, "doc:c5"]
assert [type(r.error).__name__ if r.error else None for r in out] == [
    None, "TimeoutError", None, "TimeoutError", None]
assert [r.ok for r in out] == [True, False, True, False, True]

# order is input order even when completion order differs
def slow_first(key: int) -> int:
    time.sleep(0.05 if key == 0 else 0.0)
    return key * 10

assert [r.value for r in fetch_all([0, 1, 2], slow_first, max_workers=3)] == [0, 10, 20]

# concurrency is real but bounded
lock = threading.Lock()
active = peak = 0

def tracked(key: int) -> int:
    global active, peak
    with lock:
        active += 1
        peak = max(peak, active)
    time.sleep(0.03)
    with lock:
        active -= 1
    return key

assert [r.value for r in fetch_all(list(range(12)), tracked, max_workers=4)] == list(range(12))
assert 1 < peak <= 4, peak

assert fetch_all([], fake_fetch) == []
try:
    fetch_all(["c1"], fake_fetch, max_workers=0)
except ValueError:
    pass
else:
    raise AssertionError("max_workers=0 accepted")

A common wrong solution (the tests catch it):

from collections.abc import Callable, Sequence
from concurrent.futures import ThreadPoolExecutor
from typing import Generic, NamedTuple, TypeVar

K = TypeVar("K")
V = TypeVar("V")


class Result(NamedTuple, Generic[K, V]):
    key: K
    value: V | None
    error: Exception | None

    @property
    def ok(self) -> bool:
        return self.error is None


def fetch_all(keys, fetch, *, max_workers=8):
    if max_workers < 1:
        raise ValueError("max_workers must be >= 1")
    results = []
    with ThreadPoolExecutor(max_workers=max_workers) as pool:
        try:
            for key, value in zip(keys, pool.map(fetch, keys)):
                results.append(Result(key, value, None))
        except Exception as exc:            # map re-raises the first failure...
            results.append(Result(keys[len(results)], None, exc))   # ...and the rest are lost
    return results

Alternatives and tradeoffs:

• executor.map: simplest, ordered, but re-raises the first exception when iteration reaches it and you lose everything after; only use it when any failure should fail the whole batch.
• as_completed: process results as they finish (useful for streaming progress or early abort); reorder at the end if callers need input order.
• asyncio + httpx.AsyncClient + Semaphore: better for thousands of concurrent calls with an async-capable client; a bigger change if the client is blocking.
• Processes: wrong here; the work is waiting on I/O, which releases the GIL, so processes only add pickling and startup cost.

Follow-ups:

1. "The KYC service starts returning 429s under your fan-out. What do you add?" Treat max_workers as a concurrency limit agreed with the owning team, add a client-side rate limiter (token bucket) and honour Retry-After, retry only idempotent reads with exponential backoff plus jitter, and stop early with a circuit breaker if the error rate crosses a threshold instead of hammering a struggling service.
2. "One fetch hangs forever. What happens to fetch_all, and how do you fix it?" The with block waits for every future, so the whole call hangs. Set timeouts in the HTTP client itself (connect and read), since you can't kill a thread. Waiting with wait(futures, timeout=...) bounds the caller's wait, but the stuck thread stays alive and holds a slot.
3. "Why return Result objects instead of raising an ExceptionGroup?" Partial success is the normal case for a batch: callers want the 4,990 documents that worked and a list of the 10 failures to retry. An ExceptionGroup is better when any failure means the batch failed and you want structured error handling with except*. Either way, the API should make the choice explicit rather than half-raising.

Scoring: Strong: Chooses threads with a reason, preserves input order, isolates failures per key, bounds concurrency, validates inputs, and avoids map's fail-fast behaviour deliberately. Tests prove ordering and the concurrency bound without flaky sleeps-as-assertions. Mentions client timeouts and rate limits. Acceptable: Working submit-based version with errors captured; needs prompting on ordering or on why map loses results. Red flag: Uses map and loses results after the first failure, spawns one thread per key without a bound, or picks processes for an I/O-bound call.

SR-E-D01 · Re-runnable daily load of UPI settlement files, with backfills

Easy · Design · Senior (5–8 years) · data pipelines, idempotency, partitioning, backfills, data contracts, observability · 35 min

Scenario:

A fintech in Bengaluru receives settlement files from three partner banks every night: CSVs dropped into an SFTP folder between 01:00 and 04:00 IST, together about 6 GB/day (~40 million UPI transactions). A Python job loads them into the warehouse table settlements, which finance uses at 09:00 IST to reconcile against the internal ledger.

Today the job is a cron script that appends whatever files it finds. Last month a bank re-sent a corrected file, the job was re-run by hand, and finance saw ₹3.2 crore of duplicated settlements. Separately, a schema change at one bank (a new column in the middle) silently shifted values into the wrong columns for two days.

Design the replacement job so it can be re-run any number of times for any date, supports backfills of the last 18 months, and fails loudly on bad input.

Clarifying questions:

• Does each file cover exactly one settlement date per bank, and can a bank re-send a corrected file for a past date? (Yes, one bank-date per file; corrections arrive up to 7 days later and replace the whole file.)
• Is there a stable transaction ID, and is it unique per bank or globally? (UTR is unique per bank; use (bank, utr).)
• What does the warehouse support? Partition overwrite, MERGE, transactions? (Partitioned by settlement_date; supports atomic partition overwrite and MERGE.)
• Who consumes the table, and what freshness do they need? (Finance at 09:00 IST; a dashboard refreshing hourly.)
• Is there a schema agreement with each bank, and who do we call when it breaks?
• What volume of rows is normal per bank per day, so we can alert on anomalies?
• What's the retention for raw files? (Keep 2 years for audit.)

Reference design:

Unit of work = (bank, settlement_date). Every run, scheduled or backfill, processes a set of these units, and processing a unit is idempotent: the result depends only on the input file, not on how many times or in what order it ran.

1. Land raw, immutably. A sensor copies each new file from SFTP to object storage at raw/bank=<b>/date=<d>/<sha256>.csv and records it in a file_manifest table (bank, date, sha256, size, row count from the trailer, received_at). The checksum makes a re-sent identical file a no-op; a different checksum for the same bank-date marks it a correction and supersedes the older one in the manifest.

2. Validate against a data contract before anything touches settlements: expected header names (by name, not position, which would have caught the shifted column), types, amount as integer paise or Decimal (never float), non-null utr, row count equal to the trailer count, amount totals within tolerance of the bank's control total. Implemented as a Pydantic model or pandas schema checks in Python; failures send the file to quarantine/ and page the owner with the reason. Nothing partial is loaded.

3. Load by overwriting the unit, not appending. Write the validated rows to a staging table, then in one transaction replace the target rows for that (bank, settlement_date): either DELETE WHERE bank=? AND date=? + INSERT, or an atomic partition overwrite if the table is partitioned by bank and date. Re-running yesterday's unit replaces it with identical data; a corrected file replaces the old data. UNIQUE(bank, utr) as a backstop.

4. Track state. A load_runs table (unit, file sha, status, rows, started/finished, code version) gives an audit trail and lets the scheduler skip units already loaded from the same file sha.

5. Orchestrate. An orchestrator (Airflow/Dagster/Prefect, or even cron plus the state table) runs a daily DAG per bank; a backfill is the same task invoked over a date range with bounded parallelism (e.g. 8 dates at a time), so the warehouse and the SFTP server aren't overwhelmed. No separate "backfill code path".

6. Observe. Per unit: rows loaded vs trailer count, amount total vs control total, lateness of the file vs its SLA (04:00 IST), duration. Alert on missing files by 04:30, on row counts outside ±30% of the trailing 4-week median, and on any quarantined file. A small "data ready" marker per date tells finance the 09:00 report is safe to run.

Python-specific: stream the CSV in chunks (pandas.read_csv(chunksize=...) with explicit dtype and usecols, or csv + batched inserts), keep memory flat regardless of file size, and make the job a package with a CLI (load --bank X --date 2025-03-31) that the orchestrator calls.

Trade-offs:

• Overwrite per partition vs row-level MERGE. Overwrite is simpler and naturally handles deletions in corrected files; MERGE touches fewer rows but needs a rule for rows missing from the corrected file. At 40M rows/day split across banks, overwriting a bank-day is cheap, so prefer it.
• Fail the whole file vs load good rows and quarantine bad rows. Finance reconciliation needs completeness, so fail the file; a product analytics feed might prefer partial loads plus a reject table.
• Orchestrator vs cron. An orchestrator gives retries, backfill UI and lineage, at the cost of running it. With one job, cron plus the state table is defensible; with ten pipelines, standardise on one orchestrator.
• Strict contract vs tolerant parsing. Strict (fail on unknown or missing columns) catches silent shifts but pages someone when a bank adds a harmless column; read by header name and fail only on missing or renamed required columns.

Deep-dive probes:

• "A bank re-sends 2025-03-12 with 200 rows removed (they were reversals)." Look for: the unit is overwritten, so removed rows disappear; an append or upsert-only design would keep them. The manifest marks the old file superseded, and the change is visible in an audit report.
• "Backfill 18 months after a bug fix in fee calculation." Look for: the same idempotent task over a date range, bounded parallelism, running against raw files (not the already-transformed table), code version recorded per run, a dry run comparing totals before overwriting, and telling finance which periods will change.
• "The 09:00 report ran while the 2025-04-01 load was halfway through." Look for: atomic swap per unit (staging + one transaction, or partition overwrite) so readers never see half a day; a readiness marker or SLA check that the report waits on.
• "A file is 3x the usual size." Look for: chunked streaming so memory doesn't scale with file size, an anomaly alert on row count, and a policy decision: load and alert, or quarantine until confirmed.

Rubric: Strong: Defines the unit of idempotency (bank, date) and makes reruns overwrite it atomically; lands raw files immutably with checksums; validates a data contract by column name with row-count and control-total checks before loading; backfills reuse the same task with bounded parallelism; concrete alerts tied to the 09:00 SLA. Handles corrected files that remove rows. Acceptable: Gets idempotent reruns via delete-then-insert or MERGE on UTR and adds schema checks, but needs prompting on atomicity for readers, corrected files with deletions, or observability. Red flag: Keeps appending and dedupes downstream 'with DISTINCT', relies on 'just don't rerun it', parses amounts as floats, or treats backfill as a separate one-off script.

SR-E-D02 · Caching layer for a slow product-pricing API

Easy · Design · Senior (5–8 years) · caching, Redis, cache-aside, TTL, invalidation, cache stampede, FastAPI · 35 min

Scenario:

An e-commerce company's pricing service (FastAPI, 12 pods) answers GET /price?sku=...&pincode=... for the product page and the cart. Each request calls a rules engine and two internal services; p50 is 180 ms and p99 is 1.4 s. Traffic is ~4,000 requests/s normally and ~25,000 requests/s during the Big Billion-style sale, concentrated on a few thousand hot SKUs. The rules engine falls over above ~6,000 requests/s.

Prices change when merchandisers edit them (a few thousand edits a day, bursts at sale start) and when scheduled offers start or end on the hour. Product wants p99 under 150 ms during the sale, and the cart must never charge a stale price.

Design the caching.

Clarifying questions:

• How stale may a product-page price be? (Up to 60 seconds is fine on the product page.) And in the cart and checkout? (Must be current; checkout re-validates.)
• Does the price depend on the user (loyalty tier, coupons) or only on SKU and pincode? (SKU + pincode + a small set of customer segments; coupons are applied later in the cart.)
• How many distinct (SKU, pincode-zone) combinations are there? (~2M SKUs, ~40 pricing zones, but fewer than 50k combinations are hot.)
• Do merchandiser edits publish an event? (Yes, a price.changed event on Kafka with SKU and zone.)
• Is there already a Redis cluster, and what's the memory budget?
• {'What should happen if the rules engine is down': 'serve the last known price or an error?'}

Reference design:

Key design: cache the computed price per (sku, zone, segment), not per pincode (map pincode to zone first, which collapses ~19,000 pincodes into ~40 zones and multiplies the hit rate). Key: price:v3:{sku}:{zone}:{segment}, where v3 is a schema version so a deploy that changes the value format doesn't read old entries.

Two tiers, cache-aside: 1. In-process LRU per pod (e.g. 20k entries, TTL 5 s): absorbs the hottest SKUs during the sale with no network hop. Short TTL bounds staleness across pods. 2. Redis (TTL 60 s plus random jitter of ±10%, so entries created together don't expire together): shared across pods. Values are small JSON or msgpack with price in paise, currency, computed_at, and the price version from the source.

Read path: local → Redis → compute, then write back to both. Use an async Redis client so a Redis call doesn't block the event loop, with a tight timeout (e.g. 20 ms); on Redis error, treat it as a miss and compute: the cache must never be a hard dependency.

Invalidation: a consumer of price.changed deletes (or overwrites with the new price) the Redis keys for that SKU and zone. Scheduled offers: cache entries carry valid_until = the next scheduled change, and TTL is min(60 s, valid_until - now), so an offer ending at 12:00 isn't served at 12:00:40. In-process entries rely on their 5 s TTL (or a pub/sub broadcast if 5 s is too stale).

Stampede protection: when a hot key expires at 25k requests/s, hundreds of requests would recompute it at once and knock over the rules engine. Use request coalescing in-process (one asyncio future per key; concurrent callers await the same future), plus a short Redis lock (SET key:lock NX PX 2000) so only one pod recomputes, and/or stale-while-revalidate: keep serving the old value for a few seconds while one background refresh runs. Pre-warm the top SKUs before the sale starts.

Correctness boundary: the cart and checkout read through a fresh=True path that bypasses the caches (or checks the source version), so the product page may show a price that is up to a minute old but the charged price is always current. If the two differ, the cart shows the updated price before payment.

Observability: hit ratio per tier, recompute rate, coalesced-wait count, Redis latency and errors, staleness (now - computed_at) histogram, and a canary that compares cached vs freshly computed prices for a sample of SKUs.

Trade-offs:

• TTL-only vs event-driven invalidation. TTL-only is simple and self-healing but bounds staleness only by the TTL; events give near-instant updates but can be lost or delayed, so keep the TTL as a backstop.
• Delete-on-change vs write-through on change. Deleting is simpler and avoids racing writers; writing the new value avoids a miss storm on hot SKUs when merchandisers edit thousands of prices at sale start.
• In-process tier. Big latency and load win; costs memory per pod and creates per-pod inconsistency for up to its TTL. Keep it small and short-lived.
• Serve stale on backend failure. Better availability for the product page; unacceptable for checkout. Make the policy explicit per caller.

Deep-dive probes:

• "At 12:00 the sale starts, 3,000 SKUs get new prices, and the rules engine falls over." Look for: stampede analysis (mass invalidation at once), write-through of new prices from the event instead of delete, pre-warming, request coalescing, a concurrency limit on recomputation, and serving stale for the product page while it recovers.
• "A customer saw ₹1,299 on the product page and was charged ₹1,499." Look for: the explicit freshness contract per caller, checkout bypassing or validating the cache with a version, and showing the price change in the cart before payment rather than trying to make the product page perfectly fresh.
• "Race: an old recompute writes after the invalidation." Request A misses, computes with the old price; the edit lands and deletes the key; A then writes the old price back for 60 s. Look for: versioned values (write only if the version is newer, via a Lua script or WATCH), or short TTLs on writes from recomputes that started before the last invalidation.
• "Redis has a 200 ms latency spike." Look for: client timeouts, falling back to compute (with the concurrency limit protecting the rules engine), a circuit breaker on Redis, and the in-process tier carrying the hot set.

Rubric: Strong: Normalises the key (pincode to zone, versioned prefix), layers in-process and Redis tiers with jittered TTLs, invalidates on events with TTL as a backstop, handles scheduled price changes, protects against stampedes with coalescing or locks, and draws a hard freshness line at checkout. Treats Redis as optional and names concrete cache metrics. Acceptable: Redis cache-aside with a TTL and event-based deletion; recognises the stampede and checkout-staleness problems when prompted and proposes reasonable fixes. Red flag: Caches everything with a long TTL and no invalidation, makes Redis a hard dependency, uses functools.lru_cache in each pod as the whole design, or doesn't distinguish product-page staleness from what the customer is charged.

SR-ET-O01 · wait_for timeout around to_thread, then a retry

Easy & Tricky · Output · Senior (5–8 years) · asyncio, cancellation, asyncio.to_thread, timeouts, idempotency · 5 min

Question: A checkout endpoint calls a blocking payment SDK through asyncio.to_thread and protects it with a timeout. On timeout, the client retries. What does this print?

import asyncio
import time

charges: list[str] = []


def charge_card(order_id: str, amount: int) -> None:
    time.sleep(0.3)                      # slow payment gateway
    charges.append(order_id)
    print(f"gateway charged {order_id} ₹{amount}")


async def checkout(order_id: str, timeout: float) -> str:
    try:
        await asyncio.wait_for(asyncio.to_thread(charge_card, order_id, 499), timeout)
        return "paid"
    except TimeoutError:
        return "timeout"


async def main() -> None:
    print("attempt 1:", await checkout("ORD-7", timeout=0.1))
    print("charges so far:", len(charges))
    print("attempt 2:", await checkout("ORD-7", timeout=1.0))
    print("charges:", charges)


asyncio.run(main())

Expected output:

attempt 1: timeout
charges so far: 0
gateway charged ORD-7 ₹499
gateway charged ORD-7 ₹499
attempt 2: paid
charges: ['ORD-7', 'ORD-7']

The trap: "attempt 1: timeout, charges so far: 0, then one gateway charged line, attempt 2: paid, charges: ['ORD-7']." The candidate assumes that when wait_for times out it cancels the work, so the first charge never happens. That holds for pure coroutines, and the charges so far: 0 line seems to confirm it.

Explanation:

wait_for cancels the awaitable it was given, here the asyncio future that to_thread uses to wait for the pool thread. A running thread cannot be cancelled. Cancelling the future just stops anyone waiting for it; charge_card keeps sleeping in its thread and completes the charge.

Timeline: at 0.1 s attempt 1 times out (the thread has 0.2 s to go), so charges is still empty when printed. Attempt 2 starts a second thread at about 0.1 s. At 0.3 s the first thread finishes and prints; at about 0.4 s the second one finishes, and attempt 2 returns paid. The customer has been charged twice.

The same is true of any blocking work offloaded with run_in_executor, of process pools, and of a remote server that already received the request: a client-side timeout means "I stopped waiting", not "it didn't happen".

Real-world bug: a duplicated payment, booking or ledger entry after a timeout plus retry. The fix is not a longer timeout: send an idempotency key (here derived from ORD-7) that the gateway or your own table deduplicates on, and on timeout query the status before retrying.

Takeaway: cancellation is cooperative and only reaches code that awaits; threads, processes and remote side effects run to completion regardless.

Follow-ups:

1. "Would async with asyncio.timeout(0.1): (3.11+) instead of wait_for change the output?" No. It cancels the current task at the deadline, which again only stops the await; the thread still finishes and charges. Both are the right tools for bounding how long you wait, neither stops thread work.
2. "How would you make the slow work actually stoppable?" Use a client with its own timeouts (connect/read timeouts on the HTTP session) so the blocking call returns by itself; or move to a natively async client where cancellation reaches the socket; or pass a threading.Event that the work checks between steps. Even then, a request already sent may be processed server-side, so idempotency is still required.
3. "The default executor has a fixed number of threads. What happens under load if the gateway hangs?" Timed-out calls keep occupying threads. Retries add more, the pool (min(32, cpu+4) threads by default) saturates, and every other to_thread call in the process queues behind them: the outage spreads. Use a dedicated bounded executor for the gateway, a circuit breaker, and a cap on retries.

Scoring: Strong: Exact output including two charges and their position before attempt 2: paid. States that threads can't be cancelled, frames it as a duplicate-charge incident, and proposes idempotency keys plus status checks, not just tuning timeouts. Acceptable: Initially predicts one charge, but when asked what cancellation does to a running thread, reasons to the double charge and suggests an idempotency key. Red flag: Insists wait_for kills the thread, or suggests the fix is a larger timeout or removing retries without addressing duplicate side effects.

SR-ET-O02 · lru_cache returns the same list, and a preview mutates it

Easy & Tricky · Output · Senior (5–8 years) · functools.lru_cache, caching, aliasing, cache poisoning · 4 min

Question: A fee service caches per-state fee slabs loaded from the config DB with lru_cache. A new "festive offer preview" endpoint shows what fees would look like if they were waived. What does this print?

from functools import lru_cache


@lru_cache(maxsize=None)
def load_fee_slabs(state: str) -> list[dict]:
    print(f"loading slabs for {state}")
    return [{"upto": 1_000, "fee": 0}, {"upto": 10_000, "fee": 5}]


def fee(state: str, amount: int) -> int:
    for slab in load_fee_slabs(state):
        if amount <= slab["upto"]:
            return slab["fee"]
    return 20


def preview_festive_offer(state: str) -> list[dict]:
    slabs = load_fee_slabs(state)
    for slab in slabs:
        slab["fee"] = 0                  # "just for the preview"
    return slabs


print(fee("KA", 5_000))
preview_festive_offer("KA")
print(fee("KA", 5_000))
print(fee("MH", 5_000))
print(load_fee_slabs.cache_info())

Expected output:

loading slabs for KA
5
0
loading slabs for MH
5
CacheInfo(hits=2, misses=2, maxsize=None, currsize=2)

The trap: "5, 5, 5": the preview only changed its "own" local slabs, so real fees are unaffected. Candidates think of the cache as returning a fresh result, the way calling the uncached function would.

Explanation:

lru_cache stores the object the function returned and hands back that same object on every hit; it does not copy. preview_festive_offer gets the cached list and mutates its dicts in place, so the cache now holds waived fees. Every later fee("KA", ...) call in this process returns 0, a real revenue bug that survives until the process restarts or cache_clear() is called. MH is a separate cache entry, loaded fresh, so it's unaffected.

cache_info(): misses for the first KA call and for MH; hits for the preview's call and the second KA fee lookup, giving 2 and 2.

Why it's nasty in production: the corruption is per-process, so with several workers only some pods return wrong fees, and the effect depends on which worker served a preview first. Restarting "fixes" it, which hides the cause.

Takeaway: only cache values that are immutable (tuples, frozen dataclasses, MappingProxyType), or have the cached function's callers treat results as read-only and copy before modifying. A cache is shared mutable state.

Follow-ups:

1. "How would you make this cache safe by construction rather than by convention?" Return immutable data: a tuple of frozen dataclasses (@dataclass(frozen=True) class Slab: upto: int; fee: int). The preview then has to build new objects (dataclasses.replace(s, fee=0)), and an accidental in-place write raises. Alternatively, cache the raw data privately and return copy.deepcopy from a thin wrapper, at a cost on every call.
2. "Fee slabs change in the config DB a few times a day. What's wrong with lru_cache(maxsize=None) for them?" It never expires, so each process serves stale fees until it restarts, and different pods disagree. Use a TTL cache (or a version number checked cheaply on each call), bounded size, and an explicit invalidation path (pub/sub or a config version bump). Also decide what happens when the reload fails: serve the last known good value and alert.
3. "Someone puts @lru_cache on an instance method of a long-lived service class. What do you flag in review?" The cache is on the function, shared across instances, and keys include self, so it holds strong references to every instance (a memory leak) and requires self to be hashable. Prefer a per-instance cache (functools.cached_property for no-arg values, or a dict on the instance) or a module-level cache keyed on real inputs.

Scoring: Strong: Exact output including cache_info counts. Explains the cache returns the same object, describes the per-process production impact, and proposes immutable return types. Acceptable: Gets 5, 0, 5 once reminded that the cache stores the returned object; fixes it with a copy at the call site. Red flag: Predicts 5, 5, 5, or thinks lru_cache deep-copies results; or proposes clearing the cache after the preview as the fix.

SR-ET-O03 · model_copy(update=...) slips past a validated payout contract

Easy & Tricky · Output · Senior (5–8 years) · pydantic v2, data contracts, validation, model_copy · 5 min

Question: A payouts service treats its Pydantic model as the data contract: amounts must be positive, and validate_assignment=True is on so "nothing invalid can get in after construction". A reversal flow derives a new payout from an existing one. What does this print?

from decimal import Decimal

from pydantic import BaseModel, ConfigDict, Field, ValidationError


class Payout(BaseModel):
    model_config = ConfigDict(validate_assignment=True)

    payout_id: str
    amount: Decimal = Field(gt=0)
    currency: str = "INR"


p = Payout(payout_id="P-1", amount="1500.50")
print(repr(p.amount))

try:
    p.amount = "-20"
except ValidationError as exc:
    print("assignment rejected:", exc.errors()[0]["type"])

q = p.model_copy(update={"amount": "-20", "currency": 42})
print(repr(q.amount), repr(q.currency))

try:
    Payout.model_validate(q.model_dump(warnings=False))
except ValidationError as exc:
    print("revalidation:", sorted(e["type"] for e in exc.errors()))

Expected output:

Decimal('1500.50')
assignment rejected: greater_than
'-20' 42
revalidation: ['greater_than', 'string_type']

The trap: "ValidationError on the model_copy line" (since construction and assignment are both validated), or "Decimal('-20') and '42'" (assuming the update is at least coerced). Both assume every way of producing a model instance runs the validators.

Explanation:

Construction validates and coerces ("1500.50" becomes Decimal('1500.50')), and with validate_assignment=True the assignment p.amount = "-20" is validated too and rejected by the gt=0 constraint (error type greater_than).

model_copy(update=...) does not validate. It copies the instance's __dict__ and writes the update values in as they are, for speed. So q holds the string '-20' in a Decimal field and the int 42 in a str field: a model instance that violates its own contract and type hints. Nothing fails until something downstream does arithmetic on q.amount, writes it to the DB, or serialises it (Pydantic would emit a serialisation warning, silenced here with warnings=False).

Running the dumped data back through model_validate exposes both violations: greater_than for the amount and string_type for the currency (Pydantic's default lax mode does not coerce an int to str).

Takeaway: model_copy(update=...) and model_construct() are trusted fast paths. At a contract boundary, build derived objects through validation: Payout.model_validate({**p.model_dump(), **changes}).

Follow-ups:

1. "Where else can invalid data end up inside a Pydantic model?" model_construct() skips validation entirely, and mutating a mutable field in place (order.items.append(bad)) isn't an assignment, so validate_assignment never sees it. Frozen models plus immutable field types (tuples instead of lists) close most of these paths.
2. "How would you enforce that every derived payout is validated, as a team standard?" Provide a method on the model, e.g. def with_changes(self, **changes) -> Self: return type(self).model_validate({**self.model_dump(), **changes}), make the models frozen=True, and flag model_copy(update= and model_construct in review (or a lint rule) unless there's a comment justifying the trusted path.
3. "This model is also the contract for events published to other teams. What else matters beyond validation?" Versioning and compatibility: publish the JSON Schema (model_json_schema()), make additive changes optional with defaults, never repurpose a field, decide extra='forbid' for inputs vs 'ignore' for consumers, and run contract tests in CI that check old fixtures still validate against the new model.

Scoring: Strong: Exact output. States plainly that model_copy(update=) skips validation and type coercion, names model_construct as the same kind of path, and proposes a validated derive-with-changes helper as a team standard. Acceptable: Expects an error on the copy, but once told it doesn't validate, predicts the raw values and the revalidation errors correctly. Red flag: Assumes type hints are enforced at runtime everywhere, or says validate_assignment covers copies.

SR-ET-O04 · Path traversal through join and a prefix check

Easy & Tricky · Output · Senior (5–8 years) · security, path traversal, os.path, pathlib, input validation · 4 min

Question: An invoice-download endpoint takes a file name from the URL and serves it from /srv/invoices. A reviewer adds a "safety check". What does this print? (POSIX path functions are used so the result is the same on every OS.)

import posixpath
from pathlib import PurePosixPath

BASE = "/srv/invoices"

for name in ["INV-1.pdf", "../.env", "/etc/passwd"]:
    print(posixpath.join(BASE, name))

print(PurePosixPath(BASE) / "/etc/passwd")


def looks_safe(name: str) -> bool:
    full = posixpath.normpath(posixpath.join(BASE, name))
    return full.startswith(BASE)


print(looks_safe("../.env"), looks_safe("../invoices-archive/2023.zip"))

target = PurePosixPath(posixpath.normpath(posixpath.join(BASE, "../invoices-archive/2023.zip")))
print(target, target.is_relative_to(BASE))

Expected output:

/srv/invoices/INV-1.pdf
/srv/invoices/../.env
/etc/passwd
/etc/passwd
False True
/srv/invoices-archive/2023.zip False

The trap: "/srv/invoices//etc/passwd" for the absolute name, and "False False" for the prefix check, i.e. believing join always keeps the base directory and that startswith(BASE) after normpath confines the path.

Explanation:

Three behaviours:

1. join doesn't resolve ... /srv/invoices/../.env is a valid path that the OS resolves to /srv/.env.
2. An absolute component discards everything before it. join(BASE, "/etc/passwd") is /etc/passwd, and pathlib's / operator does the same. A user-supplied absolute path walks straight out of the base directory.
3. A string prefix is not a directory boundary. After normpath, ../.env becomes /srv/.env and is correctly rejected, but ../invoices-archive/2023.zip becomes /srv/invoices-archive/2023.zip, which starts with the string /srv/invoices. The check passes and serves a file from a sibling directory. is_relative_to compares path components, so it correctly says False.

Even the component-wise check is not enough on a real filesystem: normpath is purely lexical and a symlink inside the base can point anywhere. Production code resolves first, then compares: full = (base / name).resolve(); full.is_relative_to(base.resolve()). Better still, don't use client input as a path at all: look the invoice up by ID in the DB, check that the caller owns it, and use the stored server-side path.

Takeaway: confine paths with resolve() plus is_relative_to() (or os.path.commonpath), never with startswith on strings.

Follow-ups:

1. "Where else does this bug class show up in Python services?" Extracting archives: tarfile and zipfile members can contain ../ or absolute names (Zip Slip). On 3.12+ use tar.extractall(path, filter='data') (the default from 3.14), and validate zip member names yourself. Also templating paths from user input, static-file handlers, and upload handlers that keep the client's filename.
2. "Is resolve() + is_relative_to() free of races?" No. Between the check and open(), a symlink could be swapped (TOCTOU) if an attacker can write inside the base directory. Mitigations: base directories that untrusted users can't write to, opening with O_NOFOLLOW or openat-style APIs, or serving from object storage keyed by ID so there is no filesystem path at all.
3. "How would you test this endpoint for traversal?" A parametrised test with ../, absolute paths, URL-encoded %2e%2e%2f (after the framework's decoding), backslashes, a symlink in a temp base dir pointing outside, and a sibling directory sharing the prefix. Assert 404/400 and that nothing outside the base is read.

Scoring: Strong: Exact output. Explains absolute-component behaviour, that normpath is lexical, and why the string prefix check fails on a sibling directory. Proposes resolve() + is_relative_to, and ideally lookup by ID with an ownership check. Acceptable: Gets the join lines right but trusts the startswith check; recognises the sibling-directory hole when shown the invoices-archive case. Red flag: Proposes stripping ../ with str.replace as the fix, or doesn't see why serving user-named files is a security concern.

SR-ET-C01 · Stable shard assignment for user IDs across worker processes

Easy & Tricky · Coding · Senior (5–8 years) · hashing, hash randomisation, partitioning, multiprocessing, determinism · 12 min

Problem:

An event pipeline runs num_shards worker processes (often on different machines). Every event for a given user must go to the same shard so per-user state stays in one place. Producers call shard_for(user_id, num_shards) to pick the shard.

Write shard_for. It must: - return an int in range(num_shards); - return the same shard for the same key in every process, on every machine, after every restart; - spread keys roughly evenly; - raise ValueError if num_shards < 1.

Signature: def shard_for(key: str, num_shards: int) -> int:

Constraints: Keys are arbitrary Unicode strings (user IDs like "u-91827" or emails). Standard library only. Called ~50k times per second per producer, so it should be cheap, but correctness first.

Examples:

• shard_for('u-91827', 8) → an int in 0..7, identical in every process and on every run
• shard_for('u-91827', 0) → ValueError

Hints:

1. Run python -c "print(hash('u-91827'))" twice. What do you see?
2. Which hash functions in the standard library are defined by a specification rather than by the interpreter?
3. How do you turn a digest into an integer?

Reference solution:

import hashlib


def shard_for(key: str, num_shards: int) -> int:
    """Map `key` to a shard in range(num_shards), stable across processes and runs.

    Uses a spec-defined hash (BLAKE2b, 8-byte digest) of the UTF-8 bytes, not
    the built-in `hash()`, whose value for str/bytes is randomised per process.
    """
    if num_shards < 1:
        raise ValueError(f"num_shards must be >= 1, got {num_shards}")
    digest = hashlib.blake2b(key.encode("utf-8"), digest_size=8).digest()
    return int.from_bytes(digest, "big") % num_shards

Complexity: O(len(key)) time to hash, O(1) extra space. BLAKE2b with an 8-byte digest costs well under a microsecond for short keys; int.from_bytes and % are negligible. Modulo bias from a 64-bit value reduced by a small num_shards is around 2^-60, i.e. irrelevant.

Tests:

# ---- tests ----
import os
import subprocess
import sys
from collections import Counter

keys = [f"u-{i}" for i in range(20_000)] + ["asha@example.in", "रवि", ""]

# range
assert all(0 <= shard_for(k, 8) < 8 for k in keys)
assert shard_for("anything", 1) == 0

# roughly even: every shard within 10% of the mean
counts = Counter(shard_for(k, 8) for k in keys)
mean = len(keys) / 8
assert len(counts) == 8 and all(abs(c - mean) < 0.1 * mean for c in counts.values()), counts

# stable across processes with different hash seeds (simulates other workers/restarts)
source = open(__file__, encoding="utf-8").read().split("# ---- tests ----")[0]
probe = keys[:200]
child_code = source + f"\nprint([shard_for(k, 8) for k in {probe!r}])\n"
expected = str([shard_for(k, 8) for k in probe])
for seed in ("1", "12345", "random"):
    env = {**os.environ, "PYTHONHASHSEED": seed}
    out = subprocess.run([sys.executable, "-c", child_code], env=env,
                         capture_output=True, text=True, check=True).stdout.strip()
    assert out == expected, f"shard assignment changed with PYTHONHASHSEED={seed}"

for bad in (0, -3):
    try:
        shard_for("u-1", bad)
    except ValueError:
        pass
    else:
        raise AssertionError(f"num_shards={bad} accepted")

A common wrong solution (the tests catch it):

def shard_for(key: str, num_shards: int) -> int:
    if num_shards < 1:
        raise ValueError(f"num_shards must be >= 1, got {num_shards}")
    return hash(key) % num_shards

Alternatives and tradeoffs:

• hash(key) % n: the obvious answer and wrong. Since Python 3.3, str and bytes hashes are salted per process (PYTHONHASHSEED, a defence against hash-flooding DoS), so every worker and every restart computes a different shard. Setting PYTHONHASHSEED=0 everywhere "works" but disables that protection globally and still ties you to CPython's internal algorithm, which other languages can't reproduce.
• hashlib.md5 / sha256: also correct and portable to Go/Java producers; slightly slower than BLAKE2b, irrelevant at this rate. MD5 is fine here because this isn't a security use (pass usedforsecurity=False on FIPS builds).
• zlib.crc32: fast and stable, but a 32-bit checksum with weaker distribution; acceptable for coarse sharding.
• Consistent hashing / jump consistent hash / rendezvous hashing: needed when num_shards changes (see follow-ups). Plain modulo moves most keys when n changes.

Follow-ups:

1. "Ops wants to go from 8 to 9 shards. What happens with modulo, and what would you use instead?" About 8/9 of keys change shard, so per-user state has to move almost entirely. Jump consistent hash (a few lines, stateless, moves only ~1/9 of keys) or rendezvous hashing (pick the shard with the highest hash(key, shard), also handles removing arbitrary shards) keep movement minimal. A ring with virtual nodes suits weighted or heterogeneous shards. Whatever you pick, the migration still needs a handover plan for in-flight state.
2. "A Java service must route to the same shards. What do you put in the data contract?" The exact algorithm: UTF-8 encoding of the key, BLAKE2b with 8-byte digest, big-endian unsigned integer, modulo n, plus a few golden test vectors both codebases assert in CI. Without the encoding and byte order spelled out, implementations drift.
3. "Where else does per-process hash() randomisation bite in production?" Iteration order of sets of strings differs between processes, so anything that serialises a set (cache keys, file names, 'stable' IDs, test snapshots) changes between runs; and caches keyed on hash(obj) stored outside the process are wrong after restart. Sort before serialising and use hashlib for anything persisted or shared.

Scoring: Strong: Rejects hash() immediately and explains salting per process and why, uses a spec-defined digest with explicit encoding, validates num_shards, and raises resharding (consistent or jump hashing) and cross-language test vectors. Acceptable: Starts with hash(), but recognises the randomisation when asked how it behaves across processes, and switches to hashlib. Red flag: Ships hash(key) % n, or 'fixes' it by setting PYTHONHASHSEED=0 in the Dockerfile without understanding what that disables.

SR-ET-C02 · Crash-safe checkpoint file for a resumable backfill

Easy & Tricky · Coding · Senior (5–8 years) · file I/O, atomic writes, os.replace, fsync, backfills, durability · 15 min

Problem:

A backfill job reprocesses 18 months of GST invoices and writes a JSON checkpoint (last processed date, counts, cursor) after each day, so it can resume after a crash or a pod eviction. Last week a pod was OOM-killed mid-write and the job couldn't restart: the checkpoint file was half-written JSON.

Write write_checkpoint(path, state) so that any reader of path, at any moment, sees either the complete previous checkpoint or the complete new one, never a partial file. If serialisation fails, the previous checkpoint must be untouched and no stray temporary files may be left behind.

Signature: def write_checkpoint(path: str | os.PathLike[str], state: dict[str, Any]) -> None:

Constraints: POSIX filesystem (local disk or a PVC). Standard library only. The state is small (KBs). Non-ASCII text (e.g. customer names, ₹) must round-trip.

Examples:

• write_checkpoint('ckpt.json', {'last_date': '2025-03-31', 'rows': 1204331}) → ckpt.json contains exactly that JSON
• write_checkpoint('ckpt.json', {'last_date': '2025-04-01', 'cursor': object()}) → TypeError; ckpt.json still holds the 2025-03-31 checkpoint; no temp files remain

Hints:

1. What does open(path, 'w') do to the existing file before a single byte of new data is written?
2. Which filesystem operation replaces a file atomically, and what does it require of the source and destination?
3. Where must the temporary file live for that operation to be atomic?
4. After the rename, what still has to happen for the data to survive a power loss?

Reference solution:

import contextlib
import json
import os
import tempfile
from typing import Any


def write_checkpoint(path: str | os.PathLike[str], state: dict[str, Any]) -> None:
    """Atomically replace the JSON checkpoint at `path` with `state`.

    Serialise first (so a bad value fails before touching disk), write to a temp
    file in the same directory, fsync it, then os.replace() it over the target.
    Readers see the old file or the new file, never a partial one.
    """
    path = os.fspath(path)
    payload = json.dumps(state, ensure_ascii=False, sort_keys=True, indent=1)
    directory = os.path.dirname(os.path.abspath(path))

    fd, tmp_path = tempfile.mkstemp(prefix=".ckpt-", suffix=".tmp", dir=directory)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as tmp:
            tmp.write(payload)
            tmp.flush()
            os.fsync(tmp.fileno())          # data on disk before the rename
        os.replace(tmp_path, path)          # atomic on POSIX within one filesystem
    except BaseException:
        with contextlib.suppress(FileNotFoundError):
            os.unlink(tmp_path)
        raise

    # Make the rename itself durable (directory entry), where supported.
    if hasattr(os, "O_DIRECTORY"):
        dir_fd = os.open(directory, os.O_DIRECTORY)
        try:
            os.fsync(dir_fd)
        finally:
            os.close(dir_fd)

Complexity: O(size of state) time and space. Each checkpoint costs two fsync calls (file and directory), typically a few milliseconds on SSD and more on network volumes; at one checkpoint per processed day that's negligible. If checkpoints were per record, batch them.

Tests:

import json
import os
import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as d:
    ckpt = Path(d) / "gst_backfill.json"

    # first write, then overwrite
    write_checkpoint(ckpt, {"last_date": "2025-03-31", "rows": 1_204_331})
    assert json.loads(ckpt.read_text(encoding="utf-8")) == {"last_date": "2025-03-31", "rows": 1_204_331}
    good = {"last_date": "2025-04-01", "rows": 1_250_002, "note": "₹ totals reconciled for रवि"}
    write_checkpoint(str(ckpt), good)
    assert json.loads(ckpt.read_text(encoding="utf-8")) == good

    # a failing write leaves the previous checkpoint intact and no litter
    before = ckpt.read_bytes()
    bad = {"last_date": "2025-04-02", "rows": 1_300_000, "cursor": object()}
    try:
        write_checkpoint(ckpt, bad)
    except TypeError:
        pass
    else:
        raise AssertionError("unserialisable state accepted")
    assert ckpt.read_bytes() == before
    assert sorted(os.listdir(d)) == ["gst_backfill.json"], os.listdir(d)

    # the replace happens from the same directory (same filesystem => atomic)
    calls = []
    real_replace = os.replace

    def spy_replace(src, dst, *a, **kw):
        calls.append((os.path.dirname(os.path.abspath(src)), os.path.abspath(dst)))
        return real_replace(src, dst, *a, **kw)

    os.replace = spy_replace
    try:
        write_checkpoint(ckpt, {"last_date": "2025-04-03"})
    finally:
        os.replace = real_replace
    assert calls == [(os.path.abspath(d), os.path.abspath(ckpt))], calls
    assert json.loads(ckpt.read_text(encoding="utf-8")) == {"last_date": "2025-04-03"}
    assert sorted(os.listdir(d)) == ["gst_backfill.json"]

A common wrong solution (the tests catch it):

import json
import os
from typing import Any


def write_checkpoint(path: str | os.PathLike[str], state: dict[str, Any]) -> None:
    # "with" closes the file, so this looks safe...
    with open(path, "w", encoding="utf-8") as f:
        json.dump(state, f, ensure_ascii=False, sort_keys=True, indent=1)

Alternatives and tradeoffs:

• open(path, 'w') + json.dump (the obvious answer): 'w' truncates the existing file immediately, and json.dump writes in chunks, so a crash or a serialisation error mid-way leaves an empty or partial file and destroys the last good checkpoint.
• Serialise first, then open('w').write(payload): fixes the serialisation-error case but not a crash during the write or an unflushed buffer at power loss.
• Keep two generations (ckpt.json and ckpt.json.prev, or numbered files) and fall back to the previous one if the newest is unreadable: useful on filesystems without atomic rename (some network or object-store mounts).
• Store the checkpoint in the database in the same transaction as the processed output: strictly better when the output already goes to a DB, because the checkpoint and the data can never disagree.

Follow-ups:

1. "The checkpoint is written after the day's output is written. A crash between the two happens. Is the backfill still correct on resume?" It reprocesses that day, so the day's output write must be idempotent: overwrite the day's partition (delete-and-insert in one transaction, or INSERT ... ON CONFLICT DO UPDATE), never blind-append. Checkpoint-after-output gives at-least-once; idempotent output turns it into effectively-once.
2. "This now runs on S3/GCS instead of a local disk. What changes?" There is no rename; an object PUT is already atomic (readers see the old or new object, never partial), so you write the object directly. But you lose cross-object atomicity, and you may want conditional writes (generation/ETag preconditions) so two runs of the job can't overwrite each other's checkpoint.
3. "mkstemp creates the temp file with mode 0600. Does that matter?" After os.replace, the checkpoint has the temp file's permissions and owner, not the old file's. If another user or a sidecar needs to read it, os.chmod the temp file (or copy the old mode with shutil.copymode) before the replace.

Scoring: Strong: Temp file in the same directory, fsync, os.replace, cleanup on failure, serialise before touching disk; explains why 'w' truncation is the bug and mentions directory fsync and idempotent output on resume. Acceptable: Gets temp-plus-rename, but puts the temp file in /tmp (possibly another filesystem, where the rename fails or isn't atomic) or forgets cleanup; corrects with a hint. Red flag: Believes the with block makes the write atomic, or proposes try/except around json.dump to rewrite the old content.

SR-ET-C03 · Verify a signed webhook from a payment provider

Easy & Tricky · Coding · Senior (5–8 years) · security, HMAC, webhooks, constant-time comparison, replay protection, raw bytes vs parsed JSON · 15 min

Problem:

Your service receives payment-status webhooks. Each request carries a header

X-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256>

where the HMAC is computed with your shared secret over the bytes b"<t>." + raw_request_body. During secret rotation the provider sends several v1= entries (one per active secret); the request is valid if any of them matches.

Write verify_webhook. Return True only if the header is well-formed, the timestamp is within tolerance_s seconds of now (in either direction), and a signature matches. Return False for everything else; never raise on bad input.

Signature: def verify_webhook(raw_body: bytes, header: str, secret: bytes, *, now: int, tolerance_s: int = 300) -> bool:

Constraints: Standard library only. The body is JSON with arbitrary key order, whitespace and UTF-8 text (e.g. "₹499 refund"). Treat the header as attacker-controlled.

Examples:

• verify_webhook(body, 't=1767225600,v1=<correct hex>', secret, now=1767225630) → True
• same request, now=1767226200 (10 minutes later) → False (stale: possible replay)
• body with one byte changed → False

Hints:

1. Your web framework hands you request.json(). Is that what the provider signed?
2. How does == on two strings behave timing-wise, and why does it matter for a signature check?
3. What stops an attacker from re-sending a captured valid request tomorrow?

Reference solution:

import hashlib
import hmac


def verify_webhook(
    raw_body: bytes,
    header: str,
    secret: bytes,
    *,
    now: int,
    tolerance_s: int = 300,
) -> bool:
    """Check a `t=...,v1=...` HMAC-SHA256 signature over b"<t>." + raw_body.

    Signs the exact bytes received (never re-serialised JSON), compares in
    constant time, accepts any of several v1 signatures (secret rotation), and
    rejects timestamps outside the tolerance window (replay protection).
    """
    timestamp: int | None = None
    candidates: list[str] = []
    for part in header.split(","):
        name, sep, value = part.strip().partition("=")
        if not sep:
            return False
        if name == "t":
            if timestamp is not None or not (value.isascii() and value.isdecimal()):
                return False
            timestamp = int(value)
        elif name == "v1":
            candidates.append(value)
    if timestamp is None or not candidates:
        return False
    if abs(now - timestamp) > tolerance_s:
        return False

    signed = str(timestamp).encode("ascii") + b"." + raw_body
    expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
    return any(hmac.compare_digest(expected, c) for c in candidates if c.isascii())

Complexity: O(len(body)) for one HMAC computation plus O(k) constant-time comparisons for k candidate signatures. Header parsing is O(len(header)). No allocation beyond the signed byte string.

Tests:

import hashlib
import hmac

SECRET = b"whsec_test_7f3a"
OLD_SECRET = b"whsec_old_19c2"
T = 1_767_225_600

def sign(body: bytes, t: int, secret: bytes = SECRET) -> str:
    return hmac.new(secret, f"{t}.".encode() + body, hashlib.sha256).hexdigest()

# exactly as the provider sent it: compact separators, raw UTF-8 (not \u escapes)
body = '{"event":"refund.processed","amount_paise":49900,"note":"₹499 refund"}'.encode("utf-8")
header = f"t={T},v1={sign(body, T)}"

assert verify_webhook(body, header, SECRET, now=T + 30) is True
assert verify_webhook(body, header, SECRET, now=T - 30) is True        # small clock skew
assert verify_webhook(body, f" t={T} , v1={sign(body, T)} ", SECRET, now=T) is True

# tampering and wrong secret
assert verify_webhook(body.replace(b"49900", b"99900"), header, SECRET, now=T) is False
assert verify_webhook(body, header, b"wrong", now=T) is False

# replay window
assert verify_webhook(body, header, SECRET, now=T + 301) is False
assert verify_webhook(body, header, SECRET, now=T + 300) is True

# rotation: any v1 may match
rotating = f"t={T},v1={sign(body, T, OLD_SECRET)},v1={sign(body, T)}"
assert verify_webhook(body, rotating, SECRET, now=T) is True
assert verify_webhook(body, rotating, OLD_SECRET, now=T) is True

# malformed headers never raise
for bad in ["", "garbage", f"v1={sign(body, T)}", f"t={T}", f"t=abc,v1={sign(body, T)}",
            f"t={T},t={T},v1={sign(body, T)}", f"t={T},v1=", f"t={T},v1=ä{sign(body, T)[1:]}",
            f"t=-{T},v1={sign(body, T)}", f"t=²,v1={sign(body, T)}"]:
    assert verify_webhook(body, bad, SECRET, now=T) is False, bad

A common wrong solution (the tests catch it):

import hashlib
import hmac
import json


def verify_webhook(raw_body, header, secret, *, now, tolerance_s=300):
    try:
        parts = [p.strip().partition("=") for p in header.split(",")]
        t = int(next(v for k, _, v in parts if k == "t"))
        sigs = [v for k, _, v in parts if k == "v1"]
    except (StopIteration, ValueError):
        return False
    if not sigs or abs(now - t) > tolerance_s:
        return False
    # "normalise" the payload first, as the framework already parsed it
    canonical = json.dumps(json.loads(raw_body)).encode()
    expected = hmac.new(secret, f"{t}.".encode() + canonical, hashlib.sha256).hexdigest()
    return any(hmac.compare_digest(expected, s) for s in sigs if s.isascii())

Alternatives and tradeoffs:

• Re-serialising the parsed JSON (the obvious answer when the framework gives you a dict): json.dumps changes separators, escapes non-ASCII to ₹, and may reorder or reformat numbers, so valid requests fail. Teams then "fix" it by disabling verification. Always read the raw body (await request.body() in FastAPI/Starlette) before parsing.
• expected == candidate: functionally correct but short-circuits on the first differing character, leaking timing information an attacker can use to forge a signature byte by byte. hmac.compare_digest takes time independent of where the strings differ.
• Provider SDK helper: prefer it when one exists; this exercise is what those helpers do, and you still need to know to pass them raw bytes.
• Dedupe by event ID in addition to the timestamp window: the window limits replays to 5 minutes; storing processed event IDs makes a replay within the window harmless too.

Follow-ups:

1. "In FastAPI, the endpoint declares payload: RefundEvent as a Pydantic body parameter. How do you get the raw bytes?" Take request: Request and call raw = await request.body() first, verify, and only then RefundEvent.model_validate_json(raw). Or do verification in a dependency that reads the body (Starlette caches it, so later parsing still works). Don't verify inside a model validator: by then you only have parsed data.
2. "Why compare hex strings with compare_digest rather than decoding to bytes first? Does it matter?" Either works if both sides are the same type: compare_digest accepts two ASCII str or two bytes-like objects. Comparing hex is simpler and avoids bytes.fromhex raising on malformed input. The point is a constant-time comparison of equal-length values; a length mismatch returns False quickly, which leaks only the length, and that is public anyway.
3. "How would you roll your own secret over without dropping webhooks?" Accept a list of secrets in config (current plus previous) and verify against each, which this function supports by being called per secret or by taking a list. Rotate the provider to sign with both, deploy the new secret, then remove the old one after the overlap window. Alert on verification failures by reason (stale, bad signature, malformed) so a misconfiguration is visible immediately.

Scoring: Strong: Signs the raw bytes and says why parsing breaks it, uses hmac.compare_digest, enforces the timestamp window in both directions, handles multiple v1 values, and never raises on hostile headers. Mentions event-ID dedupe and reading the raw body in FastAPI. Acceptable: Correct HMAC over raw bytes with compare_digest, but misses rotation or the replay window until asked. Red flag: Signs json.dumps(request.json()), compares with == without knowing why that's a problem, or wraps everything in except Exception: return True.

SR-ET-D01 · "Make it async": a slow invoice-PDF endpoint

Easy & Tricky · Design · Senior (5–8 years) · concurrency, asyncio vs threads vs processes, GIL, CPU-bound work, job queues, profiling · 30 min

Scenario:

A B2B platform's FastAPI service has POST /invoices/{id}/pdf, which renders a GST invoice as PDF (Jinja2 HTML template, then a pure-Python PDF library, plus a QR code) and returns it. Each render takes ~900 ms of CPU for a 3-page invoice and up to 8 s for 200-line invoices. At month-end, customers download ~40 invoices/s for about two hours, p99 latency hits 30 s, health checks time out, and Kubernetes restarts pods, making it worse.

The service runs Uvicorn with 1 worker process per pod on 4-vCPU pods; the endpoint is async def and calls render_pdf(invoice) directly.

A teammate's PR: "Rewrite the PDF rendering to use asyncio and aiofiles so the endpoint is non-blocking." Review the proposal and design the fix.

Clarifying questions:

• {'Is the time really CPU? Has anyone profiled a render? (Yes': "py-spy shows ~85% in the PDF library's layout code, ~10% template rendering, ~5% DB and file I/O.)"}
• Do users need the PDF synchronously, or would "your invoice is ready" plus a download link work? (Interactive downloads want it within a few seconds; bulk "download all March invoices" can be async.)
• How often is the same invoice re-downloaded? Do invoices change after they're issued? (Issued invoices are immutable; most downloads are repeats of the same few thousand invoices at month-end.)
• What's the pod budget? Can we add a separate worker deployment?
• {'Are there other endpoints on the same service that are suffering? (Yes': 'the order-status API on the same pods also times out at month-end.)'}

Reference design:

Why "make it async" is the wrong fix. asyncio gives concurrency for code that waits; it does nothing for code that computes. The render is ~85% CPU in Python, which holds the GIL. Worse, today the CPU-bound call runs inside an async def endpoint, so it blocks the event loop: while one PDF renders, that worker can't accept requests, answer health checks or serve the order-status API. That is why pods get restarted. aiofiles would make the 5% file I/O non-blocking and change nothing else.

Immediate fix (hours): stop blocking the event loop. Either declare the endpoint as plain def (FastAPI runs it in its threadpool) or await anyio.to_thread.run_sync(render_pdf, invoice) / asyncio.to_thread. This restores health checks and other endpoints, but threads still share one GIL, so PDF throughput stays at about one CPU's worth per process.

Real fix: use the cores, and stop rendering the same thing twice. 1. Render once, store, serve many. Issued invoices are immutable, so render on issue (or on first request) and store the PDF in object storage keyed by invoice_id and template version; downloads become a redirect to a pre-signed URL. Most month-end traffic turns into cheap reads. 2. Move rendering to a separate worker pool (Celery/RQ/Arq, or a dedicated deployment consuming a queue), sized by CPU: one process per vCPU. The API enqueues and returns 202 with a job ID for bulk requests; for interactive single downloads it waits a bounded time (e.g. 5 s) for the job, then falls back to "we'll email you". 3. If a separate deployment is overkill, a ProcessPoolExecutor in the API process (loop.run_in_executor(pool, render_pdf, invoice_id)) uses multiple cores, but competes with request handling for CPU and memory and needs care with pool breakage and pickling (send the ID, not the ORM object). 4. Profile the render itself: py-spy or cProfile on a 200-line invoice. Often the template does N+1 DB queries, or fonts and the QR library are re-initialised per render; caching fonts per worker can halve the time.

Protect the service: a concurrency limit on rendering (semaphore or queue length) with 429/503 + Retry-After beyond it, liveness probes that don't depend on the CPU-bound path, and month-end pre-rendering of the previous month's invoices overnight.

Trade-offs:

• Sync endpoint in threadpool vs process pool vs separate workers. Threadpool: zero infra, unblocks the loop, no extra throughput. Process pool in the API: real parallelism, but the API pod's CPU now serves two workloads and a crashed worker breaks the pool. Separate worker deployment: scales independently, isolates failures, adds a queue to operate. With month-end spikes and an existing queue, separate workers win.
• Pre-render vs render on demand. Pre-rendering costs storage and compute for invoices nobody downloads, but turns the peak into reads; render-on-first-request plus storage is a good middle ground.
• Synchronous response vs 202 + polling or notification. Synchronous is simpler for clients; async scales and survives spikes. Use both: short wait, then fall back.
• More Uvicorn workers per pod (e.g. 4 processes on 4 vCPUs) is a cheap partial fix: real parallelism, but each worker's event loop is still blocked while it renders unless the render is offloaded too.

Deep-dive probes:

• "Prove the event loop is blocked." Look for: asyncio debug mode (PYTHONASYNCIODEBUG=1, slow-callback warnings above loop.slow_callback_duration), a loop-lag metric (schedule a callback every 100 ms and measure delay), py-spy dump showing the main thread in the PDF library, and health-check latency correlating with renders.
• "Why not just use threads instead of processes for rendering?" Look for: the GIL serialises Python bytecode, so threads don't add CPU throughput for pure-Python work; they help only when the library releases the GIL in C code. Free-threaded CPython builds (experimental in 3.13, officially supported but not the default from 3.14) need every C extension in the stack to support them; not a quick production fix.
• "The worker pool is behind by 20 minutes at month-end." Look for: queue depth and age as the scaling signal (KEDA or similar), priorities for interactive over bulk jobs, idempotent jobs keyed by invoice ID so retries and duplicate requests don't re-render, and pre-rendering to shrink the peak.
• "Template change: last year's invoices must still render as issued." Look for: storing the rendered PDF as the legal record (immutable, versioned key), and not re-rendering historical invoices with a new template.

Rubric: Strong: Rejects async as a fix for CPU-bound work and explains the GIL and event-loop blocking; separates the immediate fix (stop blocking the loop) from the real fix (process-level parallelism plus render-once-and-store); uses profiling data to decide; adds admission control and queue-based scaling. Acceptable: Recognises the work is CPU-bound and proposes a process pool or more workers, but misses that the async def endpoint is blocking the event loop, or misses caching immutable PDFs, until prompted. Red flag: Approves the asyncio/aiofiles rewrite, proposes threads as the throughput fix, or answers only with 'add more pods'.

SR-ET-D02 · Renaming a column and an event field without downtime

Easy & Tricky · Design · Senior (5–8 years) · migrations, expand/contract, rolling deploys, backward compatibility, data contracts, Alembic · 30 min

Scenario:

The orders service (FastAPI + SQLAlchemy + Alembic on PostgreSQL, 10 pods with rolling deploys) has a column orders.amount holding the order total in rupees as NUMERIC(12,2). Finance wants it stored as integer paise in a column named amount_paise, and the service also publishes an order.created event (JSON on Kafka) with field amount that four other teams consume, including a Spark job and a Go service.

A developer's plan: one PR with an Alembic migration that renames the column and converts the values (ALTER TABLE orders RENAME COLUMN amount TO amount_paise; UPDATE orders SET amount_paise = amount_paise * 100; ALTER ... TYPE BIGINT), updates the model and the event field, runs the migration in the deploy pipeline, and ships. "It's a rename; it'll take a minute."

orders has 180 million rows and takes ~300 writes/s. Review the plan and design the rollout.

Clarifying questions:

• How do deploys work? Do old and new pods run at the same time during a rollout, and does the migration run before or after the new pods start? (Rolling deploy over ~10 minutes; migration runs as a pre-deploy job.)
• Can we roll back the application quickly if something breaks, and is the migration expected to be reversible?
• Who consumes order.created, and do we know their deploy schedules? Is there a schema registry or versioned contract?
• Are there readers of the orders table other than this service (BI tools, reporting replicas, CDC into the warehouse)?
• Are there amounts with fractions of a paisa anywhere, or values that would overflow?
• What's the lock-timeout and statement-timeout policy on this database?

Reference design:

Why the one-step plan fails. During a rolling deploy, old and new code run at the same time. After the migration, old pods still query orders.amount, which no longer exists, so every request on them errors until the rollout finishes. Rolling back the app doesn't help, because the schema moved. The single UPDATE of 180M rows holds row locks for a long time, generates a huge amount of WAL, and the ALTER ... TYPE rewrites the table under an ACCESS EXCLUSIVE lock, blocking all reads and writes for minutes. Renaming the event field breaks four consumers who deploy on their own schedules: the Go service drops orders with a missing amount, and the Spark job silently reads nulls.

Expand / migrate / contract, each step deployable and reversible on its own:

1. Expand (schema). ALTER TABLE orders ADD COLUMN amount_paise BIGINT NULL (a nullable column with no default is a metadata-only change, near-instant; run with a short lock_timeout and retry so it doesn't queue behind long transactions and block traffic).
2. Dual write (code). Deploy code that writes both amount and amount_paise on every insert or update (converting with Decimal, int(amount * 100) after checking there's no fractional paisa), and still reads amount. Old pods that write only amount are still around for a few minutes, so step 3 must catch their rows too.
3. Backfill in batches. A Python job updates WHERE amount_paise IS NULL AND id BETWEEN ? AND ? in batches of ~10k rows, committing each batch, throttled (sleep between batches, monitor replication lag), resumable from the last ID, and idempotent (re-running is harmless). Re-run until no NULLs remain after the dual-write rollout has finished.
4. Verify. SELECT count(*) WHERE amount_paise IS NULL OR amount_paise <> amount * 100 returns 0; then add the constraint without a long lock: ADD CONSTRAINT ... CHECK (amount_paise IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT.
5. Switch reads (code). Read amount_paise behind a feature flag; compare in shadow mode for a while if the risk justifies it.
6. Contract. Once no code reads or writes amount (and BI and CDC consumers have moved), stop writing it, then drop it in a later release.

Event contract, same pattern. Add amount_paise to order.created alongside amount (additive change; consumers ignore unknown fields), announce a deprecation date, track which consumers have migrated (consumer lag on reading the new field, or just asking the four teams), and remove amount only after all have switched, ideally as a new event version (order.created.v2) if the registry supports it. Contract tests in CI check that the event still validates against the published schema.

Every step can be rolled back without data loss, because the old column stays correct until the very end.

Trade-offs:

• Expand/contract vs a maintenance window. A 20-minute window with the one-step plan is simpler and sometimes acceptable for small internal tables; for a 180M-row, 300 writes/s table with external consumers, the window would be long and the rollback story poor.
• Dual-write in application code vs a database trigger. Triggers catch every writer (including old pods and ad-hoc scripts) and remove the race in step 2, but they hide logic in the database and add per-write cost; app dual-write is explicit and testable. A trigger for the transition period is a reasonable choice here because old pods keep writing during the rollout.
• Backfill speed vs production impact. Bigger batches finish sooner but hold locks longer and spike replication lag; throttle against lag and p99 latency, not a fixed rate.
• Keeping the old event field for months costs a little payload size and some confusion; removing it early breaks consumers you don't control.

Deep-dive probes:

• "Between steps 2 and 3, an old pod inserts an order with only amount." Look for: understanding that rolling deploys overlap, so the backfill (or a trigger) must cover rows written by old code; the verification query catches stragglers before the read switch.
• "The ADD COLUMN migration hung and the site went down for 4 minutes." Look for: the ACCESS EXCLUSIVE lock request queued behind a long-running transaction (an analytics query), and every subsequent query queued behind the lock request. Fix: SET lock_timeout = '2s' with retries, and checking for long transactions before migrating.
• "How do you test this in CI?" Look for: running the migrations against a production-like schema, a test that old code works against the expanded schema (N-1 compatibility), Alembic upgrade/downgrade tests, and contract tests for the event with fixtures of both old and new shapes.
• "Which Python details matter in the conversion?" Look for: Decimal arithmetic (Decimal('19.99') * 100 is exactly 1999.00), never float(amount) * 100 (19.99 becomes 1998.9999999999998, and int() truncates it to 1998), asserting there's no fractional paisa, and BIGINT headroom.

Rubric: Strong: Identifies old/new code overlap during rolling deploys, table-rewrite locking and the external consumers as the three failure modes; lays out expand → dual write → batched idempotent backfill → verify → switch reads → contract, with each step independently reversible; handles the event with additive change plus deprecation; mentions lock timeouts and Decimal conversion. Acceptable: Proposes add-new-column, backfill, switch, drop, but misses the rolling-deploy overlap or the event consumers until prompted. Red flag: Approves the single migration, proposes renaming and fixing consumers 'at the same time', or backfills 180M rows in one UPDATE.

SR-M-O01 · Mock, spec and autospec against a renamed parameter

Medium · Output · Senior (5–8 years) · testing strategy, unittest.mock, autospec, contract drift, test doubles · 8 min

Question: A refund notifier was written against a draft of the SMS gateway client in which the second parameter was called message. The released client calls it text. The team's unit test uses a mock gateway. What does this print for each of the three kinds of mock?

from unittest import mock


class SmsGateway:
    def send(self, phone: str, text: str, *, priority: str = "normal") -> str:
        raise RuntimeError("real network call")


def notify_refund(gateway, phone, amount):
    # Written against a draft API that called the second parameter `message`
    return gateway.send(phone, message=f"Refund of ₹{amount} processed", priority="high")


def check(gateway):
    try:
        status = notify_refund(gateway, "+919800000001", 499)
        gateway.send.assert_called_once()
        return f"PASS ({status})"
    except Exception as e:
        return f"{type(e).__name__}: {e}"


loose = mock.Mock()
loose.send.return_value = "queued"
print(f"{'Mock':10}", check(loose))

typed = mock.Mock(spec=SmsGateway)
typed.send.return_value = "queued"
print(f"{'spec':10}", check(typed))

strict = mock.create_autospec(SmsGateway, instance=True)
strict.send.return_value = "queued"
print(f"{'autospec':10}", check(strict))

print(f"{'.sendd?':10}", hasattr(loose, "sendd"), hasattr(typed, "sendd"), hasattr(strict, "sendd"))

Expected output:

Mock       PASS (queued)
spec       PASS (queued)
autospec   TypeError: missing a required argument: 'text'
.sendd?    True False False

Explanation:

Three test doubles, three different levels of checking:

• Mock() accepts any attribute and any call. loose.send(...) with a keyword the real method doesn't have is recorded happily, assert_called_once() passes, and the test is green while production would raise TypeError on the first refund.
• Mock(spec=SmsGateway) restricts attribute names to those that exist on the class, so typed.sendd is an AttributeError (hence False on the last line). But the child typed.send is still a plain Mock with no signature: the bad keyword passes. This is the half-measure many teams think is "safe".
• create_autospec(SmsGateway, instance=True) copies the signatures recursively. Calling send(phone, message=..., priority=...) is checked against send(self, phone, text, *, priority="normal") and fails with the same TypeError the real object would raise: missing a required argument: 'text'. instance=True makes it model an instance, so self is bound and not counted.

Why a senior cares: the bug here is contract drift between two units, and it is exactly what a mocked unit test is supposed to protect you from. A mock that doesn't know the contract gives you coverage numbers without protection. Team standard: autospec=True (or create_autospec) by default for collaborators you own, plus a small number of contract or integration tests against the real client (or a fake that the client's owners maintain).

Takeaway: spec checks names, autospec checks names and call signatures; a plain Mock checks nothing.

Follow-ups:

1. "Why isn't autospec the default in unittest.mock?" It is slower (it introspects the spec object, recursively, on first attribute access) and it can't see attributes created in __init__ (instance attributes aren't on the class), so some tests need extra setup. Those costs are small compared to tests that silently pass; most teams make autospec=True the default in mock.patch and opt out deliberately.
2. "Autospec now passes. Can this code still break in production against the real gateway?" Yes. Autospec only checks the shape of the call: names, arity, keywords. It knows nothing about behaviour: the real gateway might reject numbers without +91, cap messages at 160 characters, or return "QUEUED" instead of "queued". That's what a contract test (run against the provider's sandbox, or a verified fake shared by both teams) or a consumer-driven contract is for. Mocks prove you called something correctly; they can't prove the something does what you assume.
3. "You're setting testing standards for a team of eight. Where do you draw the line between mocking and using real dependencies?" Mock at boundaries you don't own and can't run cheaply (payment gateways, SMS, third-party HTTP), always with autospec or a typed fake. Use the real thing when it's cheap and deterministic: an in-memory or containerised Postgres/SQLite for repository code, a real FastAPI TestClient for request handling. Don't mock your own pure functions or data classes. Add a thin layer of contract tests per external provider, and keep a handful of end-to-end smoke tests in CI against sandboxes.

Scoring: Strong: All four lines right; explains that spec restricts attribute names but not call signatures and that autospec binds signatures (including why instance=True matters for self); connects this to contract drift and proposes a team standard plus contract tests for what autospec can't catch. Acceptable: Gets the Mock and autospec lines right but assumes spec= also checks the signature; corrects it once asked what typed.send is. Red flag: Believes a passing mocked test proves the integration works, or says the Mock line raises because message is wrong; can't name any way to make mocks check the real interface.

SR-M-O02 · A renamed field slips through a lenient event contract

Medium · Output · Senior (5–8 years) · data contracts, pydantic v2, schema evolution, extra fields, defaults · 8 min

Question: Your service consumes OrderEvent messages from the orders team's Kafka topic. Their v2 producer renamed amount_paise to amount_inr without telling you. Your consumer validates every message with Pydantic. What does this print?

from pydantic import BaseModel, ConfigDict, ValidationError


class OrderEvent(BaseModel):                  # the consumer's copy of the contract
    order_id: str
    amount_paise: int = 0
    is_cod: bool = False


class OrderEventV2(BaseModel):                # same fields, tightened
    model_config = ConfigDict(extra="forbid")
    order_id: str
    amount_paise: int
    is_cod: bool = False


v1 = {"order_id": "ORD-9001", "amount_paise": "149900", "is_cod": "no"}
v2 = {"order_id": "ORD-9001", "amount_inr": 1499.0, "is_cod": "yes"}   # producer renamed a field

for model in (OrderEvent, OrderEventV2):
    for name, payload in [("v1", v1), ("v2", v2)]:
        try:
            print(model.__name__, name, "->", model.model_validate(payload))
        except ValidationError as err:
            print(model.__name__, name, "->", sorted((d["type"], d["loc"][0]) for d in err.errors()))

Expected output:

OrderEvent v1 -> order_id='ORD-9001' amount_paise=149900 is_cod=False
OrderEvent v2 -> order_id='ORD-9001' amount_paise=0 is_cod=True
OrderEventV2 v1 -> order_id='ORD-9001' amount_paise=149900 is_cod=False
OrderEventV2 v2 -> [('extra_forbidden', 'amount_inr'), ('missing', 'amount_paise')]

Explanation:

Lax mode coerces. By default Pydantic v2 validates in lax mode: the numeric string "149900" becomes the int 149900, and the strings "no" / "yes" become False / True (Pydantic accepts yes/no, true/false, on/off, 1/0 for booleans). So v1 parses as you'd hope.

Unknown fields are ignored and defaults fill the gap. The model's default config is extra="ignore". In v2 the key amount_inr is unknown, so it is silently dropped; amount_paise is missing, so its default 0 is used. The result is a valid OrderEvent saying the order is worth ₹0. Nothing raises, nothing logs, and the revenue dashboard quietly drops to zero for every v2 order.

The tightened model fails loudly. OrderEventV2 makes amount_paise required (no default) and sets extra="forbid". Now v2 produces two errors, missing for amount_paise and extra_forbidden for amount_inr, which is exactly the diagnosis you want in the dead-letter queue. v1 still parses, so tightening didn't break the existing producer.

The design lesson is about defaults on business-critical fields: a default turns "the producer didn't send it" into "the producer sent zero". Use defaults only where absence genuinely means that value.

Takeaway: in a consumer, extra="ignore" plus a default on a required business field is how a field rename becomes silent data corruption.

Follow-ups:

1. "Isn't extra="forbid" too brittle? The producer adding any new field would now break your consumer." Yes, and that's the real trade-off. Forward compatibility normally wants consumers to tolerate new fields. A common middle ground: keep extra="ignore" (or "allow") but make business fields required with no defaults, and log or count unknown fields (model_extra with extra="allow") so a rename shows up on a dashboard instead of in the revenue numbers. Where the topic has a schema registry (Avro/Protobuf/JSON Schema with compatibility checks), enforce it there at publish time, so a breaking rename can't be produced at all.
2. "Would strict=True have caught this?" Not the rename: strict mode controls type coercion, not unknown or missing fields. With strict mode, "149900" and "no" in v1 would also be rejected, which breaks the existing producer if it sends JSON strings. Strict is a separate decision about how much coercion you trust; the rename is caught by required fields and by the extra setting.
3. "How would you roll out the rename properly if you owned both sides?" Expand and contract: the producer sends both amount_paise and amount_inr for a period; consumers move to the new field (accepting either, preferring the new one, via a validator or AliasChoices); once every consumer is on the new field (checked through metrics or the registry), the producer drops the old one. Version the event (schema_version) so consumers can branch explicitly when semantics change, not just names.

Scoring: Strong: All four lines exact; explains lax coercion of "149900" and "yes", that extra="ignore" drops amount_inr and the default supplies 0; argues against defaults on business fields and discusses the forward-compatibility trade-off of forbid (registry, metrics on unknown fields, expand/contract). Acceptable: Gets the v2 silent-zero line and the two errors, but is unsure about the "no" / "yes" coercion, or proposes extra="forbid" everywhere without noticing the forward-compatibility cost. Red flag: Expects Pydantic to raise on the unknown field by default, or says v2 raises a type error; doesn't see why a ₹0 order passing validation is a production incident.

SR-M-O03 · Reading cProfile call counts before and after caching

Medium · Output · Senior (5–8 years) · profiling, cProfile, pstats, recursion, lru_cache, N+1 calls · 10 min

Question: A nightly sales report got slow. You profile it, then wrap the two helper functions in lru_cache and profile again. The script prints only the ncalls column that pstats would show, so the numbers are deterministic. What does it print?

import cProfile
import pstats
from functools import lru_cache

PARENT = {"phones": "electronics", "electronics": None, "kurtas": "fashion", "fashion": None}
RATES = {"USD": 83.0, "AED": 22.6, "INR": 1.0}


def fx_rate(currency):                 # in production: an HTTP call to the rates service
    return RATES[currency]


def category_path(cat):                # recursive walk up the category tree
    parent = PARENT[cat]
    return cat if parent is None else category_path(parent) + "/" + cat


def report(orders):
    rows = []
    for amount, currency, cat in orders:
        rows.append((amount * fx_rate(currency), category_path(cat)))
    return sorted(rows, key=lambda r: r[0], reverse=True)


orders = [(10, "USD", "phones"), (500, "INR", "kurtas"), (40, "AED", "phones"), (7, "USD", "kurtas")] * 250


def profile(fn):
    prof = cProfile.Profile()
    prof.runcall(fn, orders)
    # pstats keys: (file, line, name) -> (primitive calls, total calls, tottime, cumtime, callers)
    for (_, _, name), (primitive, total, *_) in sorted(pstats.Stats(prof).stats.items()):
        if name in ("report", "fx_rate", "category_path", "<lambda>"):
            ncalls = str(total) if total == primitive else f"{total}/{primitive}"
            print(f"  {name:14} ncalls {ncalls}")


print("before:")
profile(report)

fx_rate = lru_cache(maxsize=None)(fx_rate)
category_path = lru_cache(maxsize=None)(category_path)
print("after:")
profile(report)

Expected output:

before:
  fx_rate        ncalls 1000
  category_path  ncalls 2000/1000
  report         ncalls 1
  <lambda>       ncalls 1000
after:
  fx_rate        ncalls 3
  category_path  ncalls 4/2
  report         ncalls 1
  <lambda>       ncalls 1000

Explanation:

The lines come out in source order because the stats are sorted by (file, line, name).

Before caching. There are 1,000 orders.

• fx_rate runs once per order: 1000. In production that is 1,000 HTTP calls for 3 distinct currencies, the classic N+1 shape.
• category_path is called 1,000 times from report, and each of those makes one recursive call for the parent (phones → electronics, kurtas → fashion). That's 2,000 calls in total, of which 1,000 are primitive, meaning not made from inside another active call of the same function. pstats prints this as total/primitive: 2000/1000. When you see a split ncalls, the function is recursive, and the second number is how often it was entered from outside.
• The sort key lambda runs once per element (1,000), not once per comparison. sorted(key=...) computes each key once and then compares the keys.

After caching. lru_cache wraps the original functions and the module-level names are rebound to the wrappers. cProfile counts calls of the original Python functions, which now run only on cache misses.

• fx_rate: three distinct currencies give three misses: 3.
• category_path: the recursive call inside the function body looks up the global name category_path, which is now the cached wrapper. So the recursion is cached too. Misses are phones (primitive) → electronics (recursive), and kurtas (primitive) → fashion (recursive): 4/2.
• The lambda and report are unchanged.

Takeaway: in a profile, read ncalls before tottime. A call count proportional to the input for something that depends on far fewer distinct values points to an N+1 problem, and caching or batching fixes it.

Follow-ups:

1. "The rates actually change during the day. Is lru_cache(maxsize=None) on fx_rate acceptable in a long-running service?" No. It never expires and never bounds memory. In a nightly batch job, caching for the duration of one run is exactly right (rates are snapshotted per report, which is also good for reproducibility). In a service, use a TTL cache keyed by currency, or better, fetch all rates once per request or batch (get_rates([...])), which is the batching fix for N+1 and also makes the report internally consistent.
2. "cProfile showed the problem here, but in production you can't run cProfile on the API pods. What do you use?" A sampling profiler such as py-spy (py-spy top --pid / py-spy record -o flame.svg) attaches to a running process with low overhead and without code changes; austin or scalene are alternatives. cProfile is deterministic tracing and adds large overhead to call-heavy code, which skews results toward small functions. For I/O-bound N+1 issues, distributed tracing (OpenTelemetry spans per outbound call) shows the 1,000 HTTP calls directly.
3. "Why might tottime mislead you in the before profile if fx_rate really made an HTTP call?" cProfile measures wall-clock time per function, so time spent blocked on the network is attributed to whichever function made the socket call (recv inside the HTTP client), and fx_rate shows up in cumtime rather than tottime. Look at cumtime and the call counts to find the culprit, and remember cProfile's own overhead inflates functions that are called millions of times.

Scoring: Strong: Exact output, including 2000/1000 and 4/2; explains primitive vs total calls, that the sort key runs once per element, and that the recursive call goes through the rebound global name; turns this into the N+1 diagnosis and discusses TTL/batching and sampling profilers for production. Acceptable: Gets the before block and fx_rate 3 right but misses that the recursion is also cached (answers category_path 2/2 or 4), and fixes it once asked what name the function body looks up. Red flag: Assumes the key function is called once per comparison (n log n), or can't explain what the a/b form of ncalls means; treats profiling as reading tottime only.

SR-M-O04 · An idempotent upsert that a backfill still corrupts

Medium · Output · Senior (5–8 years) · idempotency, backfills, upsert, sqlite3, versioning, data pipelines · 8 min

Question: A wallet-balance table is fed by a live stream of account snapshots, each with a monotonically increasing version. An old day's file is replayed by a backfill job, and the backfill job itself gets retried. Both use an upsert. What does this print?

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE balance (account TEXT PRIMARY KEY, rupees INTEGER, version INTEGER)")

UPSERT = """
INSERT INTO balance (account, rupees, version) VALUES (?, ?, ?)
ON CONFLICT(account) DO UPDATE SET rupees = excluded.rupees, version = excluded.version
"""
GUARDED = UPSERT + " WHERE excluded.version > balance.version"

live = [("ACC-1", 1000, 1), ("ACC-1", 1500, 2), ("ACC-2", 300, 1), ("ACC-1", 900, 3)]
backfill = [("ACC-1", 1500, 2), ("ACC-2", 300, 1), ("ACC-2", 250, 2)]   # replay of an old day's file

for name, sql in [("plain", UPSERT), ("guarded", GUARDED)]:
    db.execute("DELETE FROM balance")
    for row in live:
        db.execute(sql, row)
    for row in backfill:
        db.execute(sql, row)
    for row in backfill:                        # the backfill job is retried
        db.execute(sql, row)
    print(name, db.execute("SELECT * FROM balance ORDER BY account").fetchall())

cur = db.execute(GUARDED, ("ACC-1", 1000, 1))
print("rowcount for a stale replay:", cur.rowcount)

Expected output:

plain [('ACC-1', 1500, 2), ('ACC-2', 250, 2)]
guarded [('ACC-1', 900, 3), ('ACC-2', 250, 2)]
rowcount for a stale replay: 0

Explanation:

Plain upsert: last write wins. ON CONFLICT ... DO UPDATE SET col = excluded.col overwrites with whatever row arrives last. After the live stream, ACC-1 is at version 3 (₹900). The backfill then replays version 2 (₹1500) and overwrites it: the balance goes back in time. ACC-2 ends at version 2 (₹250), which happens to be right, because the backfill also contained its newest row.

Note that the plain upsert is idempotent: running the backfill twice gives the same table as running it once. That is the trap. Idempotency only says "repeating this is harmless". It doesn't say "applying this out of order is harmless", and backfills, retries and replays are out of order by nature.

Guarded upsert: monotonic. With WHERE excluded.version > balance.version, the DO UPDATE runs only when the incoming row is newer. Stale rows become no-ops, so ACC-1 stays at version 3. Inserts of new keys are unaffected, because the WHERE belongs to the conflict branch only.

Rowcount. A stale replay hits the conflict branch, the guard is false, and no row changes: rowcount is 0. That's a useful metric: count "stale, skipped" rows per backfill so you can see how much of the replay was ignored.

Takeaway: make writes order-independent, not just repeatable. Compare a version or an event timestamp in the write itself (here in SQL, atomically), not in Python before the write.

Follow-ups:

1. "The source doesn't have a version number, only an updated_at timestamp. Is guarding on that equally safe?" Mostly, with caveats: two updates in the same timestamp tick tie (use >= plus a tie-breaker such as an event ID or log sequence number, or accept last-arrival on ties); clocks on different producers can skew, so updated_at should come from a single authority (the source database's commit time or CDC log position), not from each app server's clock. A CDC log position (Postgres LSN, MySQL binlog position) is the best version you can get for free.
2. "Why do the check inside the SQL statement instead of SELECT version in Python and then deciding whether to UPDATE?" Read-then-write is a race: two writers (the live consumer and the backfill) can both read version 2 and both write, and the slower one wins regardless of version. The conditional upsert is a single atomic statement, so the database serialises it on the row. The same idea in other stores: UPDATE ... WHERE version < :v, DynamoDB condition expressions, Redis WATCH/Lua.
3. "The backfill also needs to delete accounts that were closed on that day. How does the guard extend to deletes?" Don't hard-delete. Write a tombstone (deleted = true with the event's version) through the same guarded upsert, so a late replay of an older 'update' can't resurrect the account. Physically purge tombstones later, once no replay can be older than them (after the backfill window or the retention period).

Scoring: Strong: Exact output; explains that the plain upsert is idempotent yet not order-safe, that the guard applies only to the conflict branch, why rowcount is 0; proposes atomic version guards, discusses updated_at ties and skew and tombstones for deletes. Acceptable: Gets the plain and guarded tables right but reasons only as 'the backfill overwrote it'; needs prompting to separate idempotency from ordering, or is unsure about the rowcount. Red flag: Expects the plain upsert to keep version 3 (thinks upsert 'merges' by recency), or proposes fixing it with a Python-side read-then-write and doesn't see the race.

SR-M-C01 · Circuit breaker for a flaky bank API

Medium · Coding · Senior (5–8 years) · resilience, circuit breaker, state machines, dependency injection, testability, thread safety · 30 min

Problem:

Your checkout service calls a partner bank's API. When the bank has an outage, every checkout waits for a 10-second timeout, the worker pool fills up, and the whole site goes down with it. Implement a circuit breaker:

• closed (normal): calls go through. After failure_threshold consecutive failures, the circuit opens. A success resets the count.
• open: calls fail immediately with CircuitOpenError, without calling the bank, until reset_timeout seconds have passed since the circuit opened.
• half-open (after the timeout): exactly one trial call is let through. If it succeeds, the circuit closes. If it fails, the circuit opens again with a fresh timer. Other calls made while the trial is in flight fail fast.
• Only exceptions listed in trip_on count as failures (default: any Exception). Other exceptions, such as a ValueError for bad input, propagate without changing the state. A tripping exception is re-raised to the caller unchanged.
• Time comes from an injectable clock (default time.monotonic) so tests don't sleep.
• state returns "closed", "open" or "half_open".

Multiple worker threads share one breaker per dependency.

Signature: class CircuitBreaker: def __init__(self, failure_threshold: int, reset_timeout: float, *, trip_on: tuple[type[BaseException], ...] = (Exception,), clock: Callable[[], float] = time.monotonic) -> None def call(self, fn: Callable[..., Any], *args: Any, **kwargs: Any) -> Any @property def state(self) -> str

Constraints: Thread-safe for concurrent calls, but the lock must not be held while fn runs (the bank call can take seconds). Standard library only.

Examples:

• cb = CircuitBreaker(3, 30, clock=clock); three consecutive ConnectionErrors from cb.call(bank.charge, 100) → cb.state == 'open'; the next cb.call raises CircuitOpenError without calling bank.charge
• 30 seconds later (clock advanced), cb.call(bank.charge, 100) and the bank is back → returns the bank's result; cb.state == 'closed'
• fail, fail, success, fail, fail with threshold 3 → still 'closed': failures must be consecutive

Hints:

1. Write down the three states and every transition (what moves you from each state to each other state) before writing code.
2. Store when the circuit opened rather than a separate 'half-open' flag; the state can be computed from the clock.
3. How do you guarantee only one trial call in half-open when several threads arrive at once, without holding the lock during the call?
4. Which exceptions should count? What if the caller passes a negative amount?

Reference solution:

import threading
import time
from typing import Any, Callable


class CircuitOpenError(RuntimeError):
    """Raised instead of calling the dependency while the circuit is open."""


class CircuitBreaker:
    """Stop calling a failing dependency; probe it again after a cool-down.

    closed    -> calls go through; `failure_threshold` consecutive tripping
                 failures open the circuit.
    open      -> calls fail fast with CircuitOpenError for `reset_timeout` seconds.
    half_open -> exactly one trial call is let through; success closes the
                 circuit, failure re-opens it with a fresh timer.
    """

    def __init__(self, failure_threshold: int, reset_timeout: float, *,
                 trip_on: tuple[type[BaseException], ...] = (Exception,),
                 clock: Callable[[], float] = time.monotonic) -> None:
        if failure_threshold < 1 or reset_timeout <= 0:
            raise ValueError("failure_threshold must be >= 1 and reset_timeout > 0")
        self._threshold = failure_threshold
        self._reset_timeout = reset_timeout
        self._trip_on = trip_on
        self._clock = clock
        self._lock = threading.Lock()
        self._failures = 0
        self._opened_at: float | None = None      # None means closed
        self._trial_in_flight = False

    @property
    def state(self) -> str:
        with self._lock:
            return self._state_locked()

    def _state_locked(self) -> str:
        if self._opened_at is None:
            return "closed"
        if self._clock() - self._opened_at >= self._reset_timeout:
            return "half_open"
        return "open"

    def call(self, fn: Callable[..., Any], *args: Any, **kwargs: Any) -> Any:
        with self._lock:                           # decide, but never hold the lock during fn()
            state = self._state_locked()
            if state == "open" or (state == "half_open" and self._trial_in_flight):
                raise CircuitOpenError("circuit open; failing fast")
            is_trial = state == "half_open"
            if is_trial:
                self._trial_in_flight = True
        try:
            result = fn(*args, **kwargs)
        except self._trip_on:
            with self._lock:
                if is_trial:
                    self._trial_in_flight = False
                self._failures += 1
                if is_trial or self._failures >= self._threshold:
                    self._opened_at = self._clock()
            raise
        except BaseException:
            with self._lock:                       # non-tripping error: no state change
                if is_trial:
                    self._trial_in_flight = False
            raise
        with self._lock:
            if is_trial:
                self._trial_in_flight = False
            self._failures = 0
            self._opened_at = None
        return result

Complexity: O(1) time and space per call. The lock is held only for a few comparisons, never across the network call.

Tests:

class FakeClock:
    def __init__(self) -> None:
        self.now = 1000.0
    def __call__(self) -> float:
        return self.now


class Bank:
    def __init__(self) -> None:
        self.calls = 0
        self.up = True
    def charge(self, amount: int) -> str:
        self.calls += 1
        if not self.up:
            raise ConnectionError("bank timeout")
        return f"ok {amount}"


def expect(exc_type, fn, *args):
    try:
        fn(*args)
    except exc_type:
        return
    raise AssertionError(f"expected {exc_type.__name__}")


clock, bank = FakeClock(), Bank()
cb = CircuitBreaker(failure_threshold=3, reset_timeout=30, clock=clock)
assert cb.call(bank.charge, 100) == "ok 100" and cb.state == "closed"

# consecutive failures, not total: a success in between resets the count
bank.up = False
expect(ConnectionError, cb.call, bank.charge, 1)
expect(ConnectionError, cb.call, bank.charge, 1)
bank.up = True
cb.call(bank.charge, 1)
bank.up = False
expect(ConnectionError, cb.call, bank.charge, 1)
expect(ConnectionError, cb.call, bank.charge, 1)
assert cb.state == "closed"
expect(ConnectionError, cb.call, bank.charge, 1)     # third in a row
assert cb.state == "open"

# open: fail fast without touching the bank
calls = bank.calls
expect(CircuitOpenError, cb.call, bank.charge, 1)
clock.now += 29.9
expect(CircuitOpenError, cb.call, bank.charge, 1)
assert bank.calls == calls

# half-open probe fails -> open again with a FRESH timer
clock.now += 0.1
assert cb.state == "half_open"
expect(ConnectionError, cb.call, bank.charge, 1)
assert bank.calls == calls + 1 and cb.state == "open"
clock.now += 29
expect(CircuitOpenError, cb.call, bank.charge, 1)

# half-open probe succeeds -> closed, counter reset
clock.now += 1
bank.up = True
assert cb.call(bank.charge, 5) == "ok 5" and cb.state == "closed"
bank.up = False
expect(ConnectionError, cb.call, bank.charge, 1)
expect(ConnectionError, cb.call, bank.charge, 1)
assert cb.state == "closed"

# only ONE trial call while half-open (a nested call during the probe is rejected)
cb2 = CircuitBreaker(failure_threshold=1, reset_timeout=10, clock=clock)
expect(ConnectionError, cb2.call, bank.charge, 1)
clock.now += 10
seen = []
def probe():
    try:
        cb2.call(lambda: "second")
    except CircuitOpenError:
        seen.append("rejected")
    return "probe ok"
assert cb2.call(probe) == "probe ok" and seen == ["rejected"] and cb2.state == "closed"

# errors outside trip_on (bad input) propagate but never trip the breaker
cb3 = CircuitBreaker(failure_threshold=1, reset_timeout=10, trip_on=(ConnectionError,), clock=clock)
def bad_input():
    raise ValueError("negative amount")
expect(ValueError, cb3.call, bad_input)
assert cb3.state == "closed"
print("all tests passed")

A common wrong solution (the tests catch it):

import time
from typing import Any, Callable


class CircuitOpenError(RuntimeError):
    pass


class CircuitBreaker:
    # Looks right, but: counts TOTAL failures (a success never resets them),
    # and in half-open lets every caller through instead of one probe.
    def __init__(self, failure_threshold: int, reset_timeout: float, *,
                 trip_on: tuple[type[BaseException], ...] = (Exception,),
                 clock: Callable[[], float] = time.monotonic) -> None:
        self._threshold, self._reset_timeout = failure_threshold, reset_timeout
        self._trip_on, self._clock = trip_on, clock
        self._failures = 0
        self._opened_at: float | None = None

    @property
    def state(self) -> str:
        if self._opened_at is None:
            return "closed"
        if self._clock() - self._opened_at >= self._reset_timeout:
            return "half_open"
        return "open"

    def call(self, fn: Callable[..., Any], *args: Any, **kwargs: Any) -> Any:
        if self.state == "open":
            raise CircuitOpenError("circuit open; failing fast")
        try:
            result = fn(*args, **kwargs)
        except self._trip_on:
            self._failures += 1
            if self._failures >= self._threshold:
                self._opened_at = self._clock()
            raise
        self._opened_at = None
        return result

Alternatives and tradeoffs:

• Failure-rate window instead of consecutive count: open when, say, more than 50% of the last 20 calls (or the last 10 seconds) failed. More robust for high-traffic dependencies with occasional blips; needs a ring buffer or time-bucketed counters. Libraries such as pybreaker or purgatory, and service-mesh outlier detection (Envoy, Istio), implement these.
• Count slow calls as failures: a bank that answers in 9.9 s against a 10 s timeout is effectively down. Wrap the call with a timeout, and count timeouts (or calls above a latency threshold) as tripping failures.
• Async version: the same state machine with an asyncio.Lock, or no lock at all if all decisions happen between awaits on one event loop, and await fn().
• The broken version counts total failures (a success never resets them) and lets every caller through in half-open, so after an outage the recovering bank is hit by the entire backlog at once, the thundering-herd problem the breaker exists to prevent.

Follow-ups:

1. "You run 40 pods with one breaker each, in process. What are the consequences, and would you share state in Redis?" Each pod learns about the outage independently: up to 40 × threshold failed calls before all are open, and up to 40 probes in half-open. That's usually acceptable and much more robust than a shared breaker, which adds a network dependency on the hot path and can itself fail. Shared state makes sense only when the dependency is extremely sensitive to probes or is rate-limited per client. A middle ground is in-process breakers plus a shared kill switch (a feature flag) that operators can flip.
2. "What should the checkout do when it gets CircuitOpenError?" Degrade, don't just return a 500: offer other payment methods (UPI instead of this bank's netbanking), queue the payment for later if the business allows it, or return a clear 503 with Retry-After. And make the breaker observable: emit a metric and a log line on every state transition, with the dependency name, and alert on 'open for more than N minutes'. A breaker that opens silently looks like a mysterious drop in conversions.
3. "How do retries and the circuit breaker interact? Where does each go?" Retries go inside the breaker for a single logical call (or the breaker counts each attempt), and the breaker must short-circuit retries: once it's open, a retry loop shouldn't keep trying. Otherwise retries multiply load on a struggling dependency. Bound retries with a budget (for example at most 10% extra requests), use jittered backoff, and only retry idempotent operations or ones with an idempotency key, which a bank charge requires.

Scoring: Strong: Clean state machine with consecutive-failure reset, fresh timer on half-open failure, exactly one probe; lock not held during fn; trip_on filtering; injectable clock and deterministic tests; discusses observability, degradation and retry/breaker interaction. Done in about 30 minutes. Acceptable: Correct closed/open/half-open transitions, but allows multiple half-open probes or holds the lock across the call; fixes it when asked about concurrency. Red flag: Uses time.sleep in tests or reads time.time() directly so it can't be tested; or counts every exception, including bad input, as a dependency failure, without seeing the problem.

SR-M-C02 · Plan pending schema migrations safely

Medium · Coding · Senior (5–8 years) · migrations, deploy safety, checksums, ordering, failing loudly, release engineering · 25 min

Problem:

Your team's services run a small in-house migration runner at deploy time (think a stripped-down Alembic or Flyway). Migration files are named NNNN_description (four digits, then lowercase letters, digits and underscores) and the database records each applied migration in a schema_migrations table as (name, checksum).

Write plan_migrations(available, applied, *, allow_out_of_order=False) that returns the names of the migrations to run, in numeric order, or raises MigrationError when it isn't safe to run anything:

1. A name that doesn't match NNNN_description → error.
2. Two available migrations with the same number (two branches both added 0007_...) → error naming both.
3. A migration recorded as applied that doesn't exist in the code → error (the database is ahead of this build, for example during an application rollback).
4. An applied migration whose checksum no longer matches its file → error (someone edited a migration that already ran in production). Use the provided checksum(); it normalises CRLF to LF so a Windows checkout doesn't look like an edit.
5. A pending migration numbered below the highest applied one (a branch merged late) → error, unless allow_out_of_order=True, in which case it's included in numeric order.

Calling it when everything is applied returns [], so it's safe on every deploy.

Signature: def plan_migrations(available: dict[str, str], applied: list[tuple[str, str]], *, allow_out_of_order: bool = False) -> list[str]:

Constraints: Up to a few thousand migrations. available maps name → SQL text, in arbitrary order. Error messages must name the offending migration(s), because they are read by whoever is on call during a failed deploy.

Examples:

• plan_migrations({'0001_create_orders': ..., '0002_add_gstin': ..., '0010_index_gstin': ..., '0003_add_city': ...}, []) → ['0001_create_orders', '0002_add_gstin', '0003_add_city', '0010_index_gstin']
• same files; applied = 0001, 0002, 0010 with matching checksums → MigrationError: out-of-order migrations ['0003_add_city'] are older than applied 0010 ...
• same, with allow_out_of_order=True → ['0003_add_city']

Hints:

1. Validate everything first and decide at the end; don't return a partial plan if a later check fails.
2. Sort by the integer prefix, not by the dict's order. Would sorting the names as strings also work? When would it stop working?
3. What should happen if the code is older than the database? Is skipping silently ever safe?

Reference solution:

import hashlib
import re


class MigrationError(Exception):
    """The database and the code disagree; a human must look before anything runs."""


_NAME = re.compile(r"^(\d{4})_[a-z0-9_]+$")


def checksum(sql: str) -> str:
    """Stable fingerprint of a migration body (line endings normalised)."""
    return hashlib.sha256(sql.replace("\r\n", "\n").encode()).hexdigest()[:12]


def plan_migrations(available: dict[str, str], applied: list[tuple[str, str]],
                    *, allow_out_of_order: bool = False) -> list[str]:
    """Return the names of migrations to run, in order, or raise MigrationError.

    available: migration name -> SQL text, e.g. {"0001_create_orders": "CREATE TABLE ..."}
    applied:   (name, checksum) rows from the schema_migrations table
    """
    by_number: dict[int, str] = {}
    for name in available:
        m = _NAME.match(name)
        if not m:
            raise MigrationError(f"bad migration name: {name!r}")
        number = int(m.group(1))
        if number in by_number:
            raise MigrationError(f"two migrations numbered {number:04d}: "
                                 f"{by_number[number]}, {name}")
        by_number[number] = name

    applied_names: set[str] = set()
    for name, recorded in applied:
        if name not in available:
            raise MigrationError(f"{name} is applied in the database but missing from the code "
                                 "(deploying an older build?)")
        if checksum(available[name]) != recorded:
            raise MigrationError(f"{name} was edited after it was applied")
        applied_names.add(name)

    highest_applied = max((int(n[:4]) for n in applied_names), default=0)
    pending = [by_number[n] for n in sorted(by_number) if by_number[n] not in applied_names]
    late = [name for name in pending if int(name[:4]) < highest_applied]
    if late and not allow_out_of_order:
        raise MigrationError(f"out-of-order migrations {late} are older than applied "
                             f"{highest_applied:04d}; rebase them or pass allow_out_of_order=True")
    return pending

Complexity: O(n log n) for sorting n migrations, plus O(total SQL size) to compute checksums. O(n) space.

Tests:

def expect_error(fragment, *args, **kwargs):
    try:
        plan_migrations(*args, **kwargs)
    except MigrationError as e:
        assert fragment in str(e), str(e)
        return
    raise AssertionError("expected MigrationError")


m = {
    "0001_create_orders": "CREATE TABLE orders (id INTEGER PRIMARY KEY);",
    "0002_add_gstin": "ALTER TABLE orders ADD COLUMN gstin TEXT;",
    "0010_index_gstin": "CREATE INDEX ix_orders_gstin ON orders (gstin);",
    "0003_add_city": "ALTER TABLE orders ADD COLUMN city TEXT;",
}
row = lambda name: (name, checksum(m[name]))

# fresh database: everything, in NUMERIC order (dict order is irrelevant)
assert plan_migrations(m, []) == ["0001_create_orders", "0002_add_gstin",
                                  "0003_add_city", "0010_index_gstin"]
# partially applied: only the rest; fully applied: nothing (safe to call on every deploy)
assert plan_migrations(m, [row("0001_create_orders"), row("0002_add_gstin")]) == \
    ["0003_add_city", "0010_index_gstin"]
assert plan_migrations(m, [row(n) for n in sorted(m)]) == []
# CRLF vs LF checkout of the same file must not look like an edit
crlf = dict(m, **{"0001_create_orders": m["0001_create_orders"].replace(";", ";\r\n")})
assert checksum(crlf["0001_create_orders"]) == checksum(m["0001_create_orders"].replace(";", ";\n"))

# an applied migration was edited
expect_error("0002_add_gstin was edited",
             dict(m, **{"0002_add_gstin": "ALTER TABLE orders ADD COLUMN gstin VARCHAR(15);"}),
             [row("0001_create_orders"), row("0002_add_gstin")])
# database is ahead of the code being deployed (rollback of the app, not the schema)
expect_error("0011_drop_city is applied",
             m, [row("0001_create_orders"), ("0011_drop_city", "abc")])
# two branches both added 0003
expect_error("two migrations numbered 0003",
             dict(m, **{"0003_add_pincode": "ALTER TABLE orders ADD COLUMN pincode TEXT;"}), [])
expect_error("bad migration name", dict(m, **{"4_fix": "SELECT 1;"}), [])
# 0003 merged late, after 0010 already ran in prod
applied = [row("0001_create_orders"), row("0002_add_gstin"), row("0010_index_gstin")]
expect_error("out-of-order", m, applied)
assert plan_migrations(m, applied, allow_out_of_order=True) == ["0003_add_city"]
print("all tests passed")

A common wrong solution (the tests catch it):

import hashlib


class MigrationError(Exception):
    pass


def checksum(sql: str) -> str:
    return hashlib.sha256(sql.replace("\r\n", "\n").encode()).hexdigest()[:12]


def plan_migrations(available: dict[str, str], applied: list[tuple[str, str]],
                    *, allow_out_of_order: bool = False) -> list[str]:
    # "Run whatever isn't in the table yet": trusts the table blindly.
    done = {name for name, _ in applied}
    return sorted(name for name in available if name not in done)

Alternatives and tradeoffs:

• Use Alembic in a real SQLAlchemy codebase: it uses a revision graph (each migration names its down_revision) instead of numbers, which turns the 'two branches both added 0007' problem into explicit branch heads that must be merged (alembic merge). Flyway-style numbered files, as here, are simpler to read but need the duplicate and out-of-order checks.
• Timestamps instead of sequence numbers (20261004_1130_add_gstin) make collisions between branches rare but make out-of-order application common, so allow_out_of_order effectively becomes the default and migrations must be written to be independent of each other.
• The broken version just runs 'whatever isn't in the table yet', sorted by name: it silently runs a late-merged migration after a newer one, ignores edited migrations, and happily deploys code that's older than the database.

Follow-ups:

1. "The planner says go. Twenty pods start at once during a rolling deploy. What stops them all running migrations concurrently?" Take a database-level lock around plan-and-apply: Postgres pg_advisory_lock(<constant>) or LOCK TABLE schema_migrations IN EXCLUSIVE MODE in a transaction, then re-plan after acquiring it (another pod may have just finished). Better still, run migrations as a separate one-off step in the pipeline (a Kubernetes Job or a pre-deploy hook) instead of at application start-up, so a slow migration doesn't block pods from becoming ready.
2. "Should each migration run in a transaction? What about CREATE INDEX CONCURRENTLY?" On Postgres, yes by default: DDL is transactional, so a failed migration rolls back cleanly and the schema_migrations row is inserted in the same transaction. CREATE INDEX CONCURRENTLY can't run inside a transaction, so the runner needs a per-migration opt-out, and such migrations must be written to be re-runnable (IF NOT EXISTS, and drop an INVALID index left by a failed attempt). On MySQL, DDL causes implicit commits, so every migration must be idempotent or small enough to repair by hand.
3. "Someone edited an applied migration to fix a typo in a comment, and the deploy is blocked by the checksum error. What do you do?" Don't 'fix' the checksum in the table as a habit, but as a reviewed one-off it's acceptable if the SQL semantics didn't change: revert the edit (preferred: applied migrations are immutable history), or run a documented repair command that updates the stored checksum with an audit trail. Then add a CI check that fails any PR modifying an existing migration file.

Scoring: Strong: All five rules, validates fully before returning, numeric ordering, clear error messages naming the migrations; reasons about rollbacks (DB ahead of code), concurrency of runners and transactional DDL. Done in about 25 minutes with tests for each rule. Acceptable: Correct ordering and pending detection with checksum validation, but misses the duplicate-number or out-of-order rule until prompted; adds them quickly. Red flag: Returns set(available) - set(applied) with no ordering or validation; or proposes silently skipping unknown or edited migrations to 'keep deploys green'.

SR-M-C03 · Consistent hashing ring for a sharded cache

Medium · Coding · Senior (5–8 years) · scaling, caching, sharding, consistent hashing, bisect, hashlib · 30 min

Problem:

Your cart service shards its cache across several Redis nodes, choosing a node with nodes[hash(key) % len(nodes)]. During the Big Billion Days sale you added a fourth node, about 75% of keys moved, the cache hit rate collapsed, and the database fell over.

Implement a consistent hashing ring with virtual nodes:

• HashRing(nodes, vnodes=100): each node is placed on the ring at vnodes pseudo-random positions.
• get(key) returns the node owning the first position clockwise from the key's hash (wrapping around past the end). An empty ring raises LookupError.
• add(node) is idempotent. remove(node) raises KeyError for an unknown node.
• The mapping must be identical in every process and on every machine, whatever order the nodes were given in.
• Adding a node may move keys only to the new node. Removing a node may move only its own keys.

Signature: class HashRing: def __init__(self, nodes: Iterable[str] = (), vnodes: int = 100) -> None def add(self, node: str) -> None def remove(self, node: str) -> None def get(self, key: str) -> str @property def nodes(self) -> set[str]

Constraints: Up to ~50 nodes and millions of lookups per second across the fleet, so get must be O(log n), not a scan. add/remove are rare and may be O(n). Standard library only.

Examples:

• ring = HashRing(['cache-a', 'cache-b', 'cache-c']); ring.get('cart:42') → one of the three nodes, the same one on every pod and after every restart
• ring.add('cache-d') with 6,000 keys → about a quarter of the keys move, all of them to cache-d
• HashRing().get('x') → LookupError

Hints:

1. Why can't you use the built-in hash() for strings across processes?
2. Keep a sorted list of ring positions. Which bisect function finds 'the first position clockwise'?
3. What happens at the very end of the ring?
4. Why place each node at many positions instead of one? Try vnodes=1 with three nodes and count keys per node.

Reference solution:

import bisect
import hashlib
from collections.abc import Iterable


def _h(text: str) -> int:
    # Stable across processes and machines, unlike hash() with PYTHONHASHSEED randomisation
    return int.from_bytes(hashlib.md5(text.encode()).digest()[:8], "big")


class HashRing:
    """Consistent hashing with virtual nodes, for sharding cache keys across servers."""

    def __init__(self, nodes: Iterable[str] = (), vnodes: int = 100) -> None:
        if vnodes < 1:
            raise ValueError("vnodes must be >= 1")
        self._vnodes = vnodes
        self._points: list[int] = []          # sorted hash positions on the ring
        self._owner: dict[int, str] = {}      # position -> node
        self._nodes: set[str] = set()
        for node in nodes:
            self.add(node)

    @property
    def nodes(self) -> set[str]:
        return set(self._nodes)

    def add(self, node: str) -> None:
        if node in self._nodes:
            return
        for i in range(self._vnodes):
            point = _h(f"{node}#{i}")
            if point in self._owner:          # 64-bit collision: astronomically rare, but be explicit
                continue
            self._owner[point] = node
            bisect.insort(self._points, point)
        self._nodes.add(node)

    def remove(self, node: str) -> None:
        if node not in self._nodes:
            raise KeyError(node)
        self._points = [p for p in self._points if self._owner[p] != node]
        self._owner = {p: n for p, n in self._owner.items() if n != node}
        self._nodes.discard(node)

    def get(self, key: str) -> str:
        if not self._points:
            raise LookupError("ring is empty")
        i = bisect.bisect_right(self._points, _h(key))
        return self._owner[self._points[i % len(self._points)]]   # wrap around

Complexity: get: O(log(N·V)) with a binary search over N nodes × V virtual nodes. add: O(V log(N·V)) hashing plus O(N·V) per insort in the worst case; remove: O(N·V). Space O(N·V).

Tests:

from collections import Counter

keys = [f"cart:{i}" for i in range(6000)]
ring = HashRing(["cache-a", "cache-b", "cache-c"], vnodes=100)
before = {k: ring.get(k) for k in keys}

# deterministic: a fresh ring (another pod) maps every key the same way
assert before == {k: HashRing(["cache-c", "cache-a", "cache-b"]).get(k) for k in keys}

# reasonably balanced with virtual nodes
load = Counter(before.values())
assert set(load) == {"cache-a", "cache-b", "cache-c"}
assert all(1400 < n < 2600 for n in load.values()), load

# adding a node only moves keys TO the new node, roughly 1/4 of them
ring.add("cache-d")
after = {k: ring.get(k) for k in keys}
moved = [k for k in keys if before[k] != after[k]]
assert all(after[k] == "cache-d" for k in moved)
assert 900 < len(moved) < 2100, len(moved)

# removing a node only moves the keys it owned, and they go back where they were
ring.remove("cache-d")
assert {k: ring.get(k) for k in keys} == before
ring.remove("cache-b")
for k in keys:
    if before[k] != "cache-b":
        assert ring.get(k) == before[k]
    else:
        assert ring.get(k) in {"cache-a", "cache-c"}

ring.add("cache-a")                 # idempotent
assert ring.nodes == {"cache-a", "cache-c"}
try:
    HashRing().get("x")
    raise AssertionError("expected LookupError")
except LookupError:
    pass
try:
    ring.remove("cache-z")
    raise AssertionError("expected KeyError")
except KeyError:
    pass
print("all tests passed")

A common wrong solution (the tests catch it):

import hashlib
from collections.abc import Iterable


def _h(text: str) -> int:
    return int.from_bytes(hashlib.md5(text.encode()).digest()[:8], "big")


class HashRing:
    # Modulo sharding dressed up as a ring: stable and balanced, but adding or
    # removing a node remaps most keys.
    def __init__(self, nodes: Iterable[str] = (), vnodes: int = 100) -> None:
        self._nodes: list[str] = sorted(set(nodes))

    @property
    def nodes(self) -> set[str]:
        return set(self._nodes)

    def add(self, node: str) -> None:
        if node not in self._nodes:
            self._nodes = sorted(self._nodes + [node])

    def remove(self, node: str) -> None:
        self._nodes.remove(node)

    def get(self, key: str) -> str:
        if not self._nodes:
            raise LookupError("ring is empty")
        return self._nodes[_h(key) % len(self._nodes)]

Alternatives and tradeoffs:

• Virtual-node count: more vnodes give better balance at the cost of memory and add/remove time. With 6,000 keys in this exercise, vnodes=100 gave 2,090 / 2,056 / 1,854 keys per node, while vnodes=1 gave 2,782 / 1,844 / 1,374. Weight bigger nodes by giving them more vnodes.
• Rendezvous (highest-random-weight) hashing: for each key, score every node with hash(node, key) and pick the highest. It has no ring and perfect minimal movement, at O(N) per lookup, which is fine for tens of nodes.
• Jump consistent hash (Lamping & Veach): O(log N), no memory, perfectly balanced, but nodes must be numbered 0..N-1 and can only be added or removed at the end, so it suits shard numbers rather than named servers.
• Redis Cluster uses fixed hash slots (16,384, CRC16(key) % 16384) assigned to nodes, so rebalancing moves slots explicitly rather than relying on hashing. If you can use the cluster's own sharding, do.
• The broken version is modulo hashing behind the same API: balanced and stable, but adding a fourth node remaps about 75% of keys, the incident in the problem statement.

Follow-ups:

1. "A node is removed and its keys move to its neighbours on the ring. What happens to the database during that minute, and how do you soften it?" Every moved key is a cache miss, so the database briefly sees 1/N of the traffic uncached. That's survivable only if the DB has headroom. Soften it with request coalescing (single-flight) so each hot key is loaded once, by warming the new owner before switching traffic (dual reads during migration), by adding nodes one at a time off-peak, and with a stale-while-revalidate fallback so a short miss storm doesn't become a 'hit the DB synchronously' storm.
2. "Two app versions are deployed side by side during a rollout, and the new one uses vnodes=160. What breaks?" The two versions disagree on key placement, so the same key is written to one node and read from another: miss rates go up, and worse, stale values are read from the old location after the new owner was updated. Ring parameters (hash function, vnode count, node names) are part of the data contract. Change them only with a migration plan, and keep them in shared config rather than code defaults.
3. "Why hashlib.md5 here? Isn't MD5 broken?" MD5 is broken for security (collisions can be crafted), but here we only need a fast, well-distributed, stable hash, and an attacker choosing keys to unbalance shards is usually not in the threat model. If it is (user-controlled keys on a public API), use a keyed hash such as hashlib.blake2b(key, key=secret) so placement can't be predicted. Faster non-cryptographic options (xxhash, murmur) need third-party packages.

Scoring: Strong: Sorted positions + bisect_right + wrap-around, hashlib-based stable hashing, vnodes, idempotent add, correct remove; tests that prove minimal movement, not just 'returns a node'; discusses miss storms on rebalance and ring config as a contract. About 30 minutes. Acceptable: Working ring with stable hashing but no virtual nodes, or an O(n) scan in get; adds them when asked about balance and lookup cost. Red flag: Uses built-in hash() (different per process because of hash randomisation) or modulo hashing and can't explain why adding a node moves most keys.

SR-M-D01 · Real-time fraud scoring in the payment path

Medium · Design · Senior (5–8 years) · model serving, FastAPI, latency budgets, GIL and worker processes, feature stores, model versioning, fallbacks, observability · 40 min

Scenario:

A payments company authorises card and UPI transactions. Before authorisation, the payment service must call a fraud-scoring service and get a score within 60 ms at p99 (the whole authorisation has a 300 ms budget). Traffic is ~1,500 transactions/s normally and ~6,000/s at peaks (salary day, festive sales).

The data science team has trained a gradient-boosted tree model (LightGBM, ~300 MB in memory, ~2 ms CPU per prediction) in a notebook. It uses ~80 features: some come with the request (amount, merchant category, device ID), and some are aggregates such as "number of transactions on this card in the last 10 minutes" and "average ticket size for this merchant over 30 days". They retrain weekly and want to ship new models without engineering involvement.

Design the scoring service in Python.

Clarifying questions:

• What happens to a payment if scoring is slow or down: approve, decline, or fall back to rules? (Fall back to a rules engine; never block payments on the model.)
• Is the decision made by the fraud service (approve/decline/step-up to OTP) or just a score that the payment service thresholds?
• How fresh must the velocity features be? (Last-10-minute counts must include transactions up to a few seconds ago.)
• Where do the batch aggregates come from today, and how were the training features computed (same code or a separate SQL)?
• What are the audit requirements? (RBI and the card networks expect you to explain a decline; keep inputs, model version and score per decision.)
• How often may thresholds change, and who owns them?

Reference design:

Request path (target p99 ≤ 60 ms):

1. Payment service → POST /score (FastAPI) with the transaction. Hard client-side timeout of 60 ms; on timeout or error the payment service uses the rules engine and tags the decision fallback=true.
2. Online features: one round trip to an online feature store (Redis Cluster or similar) using an async client with a 15 ms timeout, an MGET/pipeline of all keys (card, device, merchant). Missing features take the defaults the model was trained with; if the store times out, score with request-only features using a degraded model, or fall back.
3. Inference: LightGBM predict is CPU-bound (~2 ms) and holds the GIL for most of that time, so a single process can't use more than one core. Run N Uvicorn/Gunicorn worker processes per pod (≈ vCPUs) and call predict directly in the request (2 ms is acceptable on the loop), or in a dedicated thread pool if the library releases the GIL (LightGBM's C++ predict does, so threads can help; measure). At 6,000/s × 2 ms ≈ 12 cores of pure inference, so ~24 cores with headroom across pods.
4. Response: {score, model_version, reason_codes, decision}; write the decision record asynchronously (to Kafka, not inline to a DB) for audit and training labels.

Model loading and versioning: models are immutable artifacts in a registry (MLflow or an S3 path plus a manifest with version, feature list, training data window and validation metrics). Each pod loads the model at startup (FastAPI lifespan) and before taking traffic, checks that the model's expected feature list matches the feature schema the service builds, then runs warm-up predictions; readiness passes only after that. Load in the master process before forking Gunicorn workers so the 300 MB is shared copy-on-write (Python reference counting touches objects, so verify actual RSS, or accept N × 300 MB). Rolling out a model = changing the manifest version and doing a rolling restart, not hot-swapping a global inside running workers.

Safe rollout: shadow mode first (new model scores every request in parallel, decisions still from the old one; compare score distributions and decline rates offline), then a canary by hashing the card ID (stable assignment), then full. Automated guardrails: block promotion if the decline rate at the configured threshold moves by more than X% on shadow traffic.

Feature consistency (training/serving skew): velocity features are computed by a streaming job (Kafka → Flink, or a Python consumer with Redis INCR plus TTL buckets per minute) and written to the online store, and the same definitions are used to produce point-in-time-correct training data. A daily job compares the online features logged at decision time against offline recomputation and alerts on drift.

Observability: per-stage latency (feature fetch, inference, total) as histograms, fallback rate, feature-missing rate per feature, score distribution per model version, decline rate, and model version in every log line and trace span.

Trade-offs:

• Python service vs a dedicated model server (Triton, a compiled tree runtime such as Treelite, or ONNX Runtime): Python with FastAPI is easiest for the team and fine at ~2 ms per prediction; a compiled runtime cuts CPU cost by several times if inference dominates the bill. Start in Python, keep the model artifact portable.
• Processes vs threads: processes give real parallelism regardless of the GIL but multiply memory; threads share the model but only help if prediction releases the GIL. Measure before choosing.
• Per-request vs micro-batching: batching improves throughput but adds queueing delay, which eats the 60 ms budget. Tree models are cheap per row, so per-request is usually right here.
• Fail open (rules fallback) vs fail closed: failing open keeps payments flowing during an outage, at the cost of fraud exposure; make fallback visible (metric, alert) and time-boxed.
• Weekly retrain with no engineering involvement vs safety: self-service needs automated gates (schema check, offline metrics, shadow comparison), or it becomes self-service production incidents.

Deep-dive probes:

• "p99 is 45 ms most of the day but jumps to 400 ms for a few seconds every minute." Look for: GC pauses or a periodic task on the event loop, Redis latency spikes, CPU throttling from Kubernetes CPU limits (CFS quota), and a blocking call in an async def endpoint; how to tell them apart (per-stage histograms, py-spy dump during a spike, container throttling metrics).
• "The new model declines 3× more transactions in production than in offline evaluation." Look for: training/serving skew (a feature computed differently online, a missing feature defaulting to 0 instead of the training-time default, a unit change such as rupees vs paise), checking the feature schema at load time, and shadow-mode comparison before promotion.
• "A regulator asks why transaction X was declined three months ago." Look for: immutable decision records with the input features, model version and threshold version; models kept in the registry for the retention period; reason codes (top SHAP contributions computed offline or at decision time if the budget allows).
• "Traffic doubles for Diwali." Look for: load testing at 2× with realistic feature-store load, horizontal pod autoscaling on CPU plus pre-scaling before the event (autoscalers react too slowly to a sale that starts at midnight), and capacity on the feature store, which usually saturates before the scorers do.

Rubric: Strong: Explicit latency budget per stage with timeouts and a defined fallback; reasons about the GIL and CPU-bound inference (worker processes, threads only if predict releases the GIL, memory cost of N copies); startup loading with readiness and schema checks; versioned artifacts with shadow and canary rollout; addresses training/serving skew and audit records; concrete metrics. Quantifies capacity. Acceptable: Reasonable FastAPI service with model loaded at startup, Redis features and a fallback, but vague on processes vs threads or on rollout safety; gets there with probing. Red flag: Loads the model per request or pickles it from S3 on demand, runs CPU-bound inference in an async def endpoint and expects asyncio to parallelise it, has no fallback when scoring is slow, or ships new models by overwriting a file in place.

SR-M-D02 · An internal Python SDK used by 40 teams

Medium · Design · Senior (5–8 years) · library design, packaging, semantic versioning, dependency management, typing, sync and async APIs, deprecation, testing · 40 min

Scenario:

Your platform team owns the internal payments API (REST, OAuth2 client credentials). Today ~40 teams call it with hand-written requests code: each team has its own retry logic (some retry non-idempotent POST /charges, causing double charges), half of them log full card tokens, and a breaking change last quarter took four weeks of chasing teams to roll out.

You're asked to build and own an official Python SDK, acme-payments, that every team will use. Consumers range from Django monoliths on Python 3.10 to FastAPI services on 3.12, Airflow DAGs, and notebooks. Some teams are still on Pydantic v1.

Design the SDK: its API, packaging, versioning, and how you'll evolve it.

Clarifying questions:

• Which Python versions must be supported, and for how long? (3.10–3.12 now; 3.13 within the year.)
• Do consumers need async? (The FastAPI services do; Django and Airflow are sync.)
• Is there an internal package index (Artifactory, CodeArtifact), and how do teams pin dependencies (lock files, ranges)?
• Does the payments API have an OpenAPI spec that is kept accurate? Could we generate from it?
• Which operations are idempotent, and does the API accept an Idempotency-Key header?
• What observability stack do teams use (OpenTelemetry, structured logging)?

Reference design:

API surface (small, typed, hard to misuse): - PaymentsClient (sync, built on httpx.Client) and AsyncPaymentsClient (httpx.AsyncClient) with the same method names, e.g. client.charges.create(amount_paise=..., currency="INR", idempotency_key=...). Share request building, validation and response parsing in a sans-I/O core so the two clients can't drift; only the transport differs. - Explicit types: request and response models as frozen dataclasses or TypedDicts (or Pydantic v2 models, see trade-offs), money as integer paise, enums for statuses, and a py.typed marker so mypy and pyright use the annotations. - Errors as a small hierarchy: PaymentsError → APIError(status, code, request_id) → RateLimited, NotFound, Conflict, ServerError; TransportError for network problems. Never leak httpx exceptions; consumers shouldn't depend on your transport. - Safe defaults built in: connect and read timeouts always set; retries with backoff and jitter only for idempotent methods or when an idempotency key is present (the SDK generates one per logical call and reuses it across retries); Retry-After honoured; card tokens and secrets redacted in repr and logs. - Configuration via constructor arguments with environment-variable fallbacks (ACME_PAYMENTS_BASE_URL), never global mutable state. OAuth token caching and refresh inside the client, thread-safe for the sync client and task-safe for the async one. - Observability hooks: an OpenTelemetry span per call (if opentelemetry-api is installed, optional dependency) and a structured logging logger named acme_payments with a NullHandler, so the library never configures logging for its host.

Packaging: pyproject.toml with a standard build backend (hatchling or setuptools); pure-Python wheel published to the internal index from CI on tag. Minimal, loosely bounded dependencies (httpx>=0.25,<1), because tight pins in a library cause resolver conflicts in 40 different apps. Optional extras (acme-payments[otel]). Support matrix tested in CI across Python 3.10–3.12 and the lowest and highest allowed versions of each dependency.

Versioning and evolution: semantic versioning with a written public-API policy (everything not underscore-prefixed and documented is public). Additive changes in minors; removals only in majors, after at least one minor with DeprecationWarning (with stacklevel pointing at the caller) and a changelog entry. Keep the previous major on a maintenance branch for security fixes for N months. Track adoption: the SDK sends a User-Agent: acme-payments/2.3.1 python/3.12, so the API team can see which versions are still in use before removing anything server-side.

Testing for consumers: ship a test double in the package (acme_payments.testing.FakePaymentsClient that behaves like the API, including idempotency and error cases) and httpx.MockTransport helpers, so teams don't hand-roll mocks that drift. Inside the SDK: unit tests on the sans-I/O core, contract tests against the API's staging environment in CI, and type-checking the examples.

Trade-offs:

• Generated (from OpenAPI) vs hand-written: generation keeps endpoints in sync and scales with API size, but generated code is often awkward (huge flat namespaces, weak errors). A hybrid is common: generate the models and low-level calls, and hand-write a thin ergonomic layer with retries, pagination helpers and errors.
• Pydantic models vs dataclasses for responses: Pydantic gives validation and serialisation for free, but forcing Pydantic v2 on teams still on v1 blocks their adoption. Dataclasses or TypedDicts with a small hand-written parser have zero dependency cost. A library should be stingy with heavy dependencies.
• Sync and async in one package vs two: one package with a shared core avoids duplicated logic; two packages give smaller dependency footprints. Calling async from sync via asyncio.run inside the library is a trap (it fails inside running loops, e.g. Jupyter).
• Retries in the SDK vs in callers: centralising them fixes the double-charge problem everywhere, but defaults must be conservative and overridable, because the SDK doesn't know each caller's latency budget.
• Monorepo vendoring vs index releases: if all 40 teams are in one monorepo, a single version moving in lockstep removes version skew entirely; with polyrepos you need semver discipline and adoption tracking.

Deep-dive probes:

• "Team X can't install 2.0 because it requires httpx>=0.27 and their other dependency pins httpx<0.26." Look for: why libraries use wide ranges, testing against the lower bound in CI, avoiding unnecessary minimum bumps, and vendoring or replacing a dependency only as a last resort.
• "A consumer reports duplicate charges after upgrading." Look for: how idempotency keys are generated (once per logical call, reused across retries, exposed so callers can pass their own business key), which status codes trigger retries, a regression test with a transport that drops the connection after the server commits, and an incident process (yank the release from the index, publish a patch).
• "How do you know it's safe to remove the deprecated charge() function?" Look for: User-Agent or telemetry on deprecated calls, a code search across repos, warnings escalated to errors in consumers' CI, a published removal date, and a major-version bump.
• "The async client sometimes raises 'attached to a different loop' in a FastAPI app." Look for: client lifecycle tied to the app lifespan (create on startup, close on shutdown), never creating an AsyncClient at import time or sharing one across event loops, and documenting that ownership.

Rubric: Strong: Small typed API with sync and async parity over a shared core; safe defaults for timeouts, idempotent-only retries and redaction; library hygiene (NullHandler logging, no global config, loose dependency ranges, optional extras, py.typed); explicit semver and deprecation policy with adoption telemetry; ships a fake for consumers' tests; discusses generated vs hand-written and dependency-conflict trade-offs. Acceptable: Sensible client class with retries, typed models and a versioning story, but pins dependencies tightly, forces Pydantic v2, or has no plan for consumer testing or deprecation; improves when probed. Red flag: Wraps requests with module-level global configuration, retries every method including non-idempotent POSTs, configures the root logger from the library, or has no versioning or compatibility policy for 40 consumers.

SR-MT-O01 · A retried load that reports success and loses rows

Medium & Tricky · Output · Senior (5–8 years) · retries, generators, iterator exhaustion, idempotency, data pipelines, decorators · 10 min

Question: A loader stages rows and writes them to the warehouse in one go. The first attempt hits a connection reset after two rows, so the retry decorator calls it again. It's run once with a list and once with a generator. What does this print?

import functools

def retry(times):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return fn(*args, **kwargs)
                except ConnectionError as e:
                    print(f"  attempt {attempt} failed: {e}")
            raise RuntimeError("gave up")
        return wrapper
    return deco

warehouse = []
flaky = iter([True, False])               # first bulk insert drops the connection

@retry(times=3)
def load_batch(rows):
    staged = []
    for row in rows:
        staged.append(row)
        if len(staged) == 2 and next(flaky, False):
            raise ConnectionError(f"reset after {len(staged)} rows")
    warehouse.extend(staged)
    return len(staged)

def parse(lines):
    for line in lines:
        order_id, amount = line.split(",")
        yield order_id, int(amount)

lines = ["A1,100", "A2,250", "A3,75", "A4,30"]
print("list:", load_batch(list(parse(lines))), warehouse)
warehouse.clear(); flaky = iter([True, False])
print("gen: ", load_batch(parse(lines)), warehouse)

Expected output:

  attempt 1 failed: reset after 2 rows
list: 4 [('A1', 100), ('A2', 250), ('A3', 75), ('A4', 30)]
  attempt 1 failed: reset after 2 rows
gen:  2 [('A3', 75), ('A4', 30)]

The trap: gen: 4 with all four rows, the same as the list run. The decorator looks like it "calls the function again from scratch", and the generator looks like "the data". Tempting because retry decorators are usually tested with lists or scalar arguments, where re-calling really is a fresh start.

Explanation:

Two mechanisms combine.

1. A retry re-invokes the function with the same argument objects. wrapper holds on to args and calls fn(*args) again. It doesn't (and can't) recreate the inputs. For a list that is harmless: a for loop over a list creates a fresh iterator each time, so attempt 2 starts at A1.

2. A generator is a one-shot iterator with its position stored inside it. The first attempt pulled A1 and A2 from parse(lines) before the reset. Attempt 2 iterates the same generator object, which continues from where it stopped: A3, A4. staged is a fresh local list each attempt, so the first two rows were thrown away with the failed attempt's staged.

Why this is nasty in production: nothing raises. The retry "succeeded", the function returned a count (2), the job is green, and two rows are silently missing. Any row count you log comes from the same consumed iterator, so it agrees with the wrong result. Only reconciliation against the source (expected 4, loaded 2) catches it.

(The print inside the retry appears before list: / gen: because the arguments of the outer print are evaluated first, which includes running load_batch.)

Takeaway: retry re-playable inputs only. Materialise one batch (a list, or a re-openable source such as a file path plus an offset) before the retry boundary, or put the retry inside, around the single network call.

Follow-ups:

1. "How would you make the retry decorator itself defend against this?" It can refuse one-shot iterators: before the first attempt, check the arguments with isinstance(arg, collections.abc.Iterator) (generators, map, file objects) and raise a TypeError that says to pass a sequence. That is cheap and catches the bug at the first call in tests. Silently calling list() on the iterator is worse, because it hides a memory cost the caller didn't choose.
2. "Where should the retry boundary sit in a pipeline that streams 50 million rows from a file into a warehouse?" Around the smallest unit you can re-run cheaply and idempotently, usually one batch (say 10,000 rows) that has been materialised in memory, with the network write as the retried operation. The batch is identified by (file, offset range), so a retry, or a restart after a crash, resends exactly the same rows. The warehouse write must be idempotent per batch (a MERGE on keys, or a staging table swapped in atomically), because the 'connection reset' may happen after the server committed.
3. "What monitoring would have caught the generator version quickly?" Per-batch reconciliation: rows read from the source vs rows acknowledged by the sink, emitted as metrics and compared at the end of the job, failing the job on a mismatch. Also count retries per job; a retry rate going from 0 to non-zero is worth an alert in its own right.

Scoring: Strong: Exact output, including the print order; names both mechanisms (the retry reuses argument objects, and the generator resumes from its internal position, losing the rows staged in the failed attempt); points out the silent success; proposes a re-playable batch at the retry boundary, an idempotent sink and reconciliation. Acceptable: Realises the generator run differs and that rows are lost, but gets the count or the missing rows wrong at first (e.g. says it raises StopIteration or loads 0 rows); corrects after tracing the generator's position. Red flag: Predicts identical output for both runs and, after seeing it, blames the decorator rather than the iterator; or proposes 'retry with more attempts'.

SR-MT-O02 · Stored exceptions that keep a large batch alive

Medium & Tricky · Output · Senior (5–8 years) · memory, tracebacks, frames, reference cycles, garbage collection, weakref · 10 min

Question: A worker keeps recent exceptions in a list for its /health page. Memory grows by about 400 MB per failed batch until someone clears the list, and sometimes even after that. The Batch class stands in for a large DataFrame, and a weakref shows whether it is still alive. Automatic GC is disabled so that collection happens only where the snippet asks for it. What does this print? (CPython)

import gc
import weakref

gc.disable()          # make collection timing explicit for the demo


class Batch:          # stands in for a 400 MB DataFrame
    pass


recent_errors = []    # shown on the /health page


def process(batch_id, keep_alias):
    batch = Batch()
    probe = weakref.ref(batch)
    try:
        raise ValueError(f"bad GSTIN in batch {batch_id}")
    except ValueError as e:
        recent_errors.append(e)
        if keep_alias:
            last_error = e
    return probe


p1 = process(1, keep_alias=False)
p2 = process(2, keep_alias=True)
print("after return:", p1() is not None, p2() is not None)

recent_errors.clear()
print("after clear: ", p1() is not None, p2() is not None)

gc.collect()
print("after gc:    ", p1() is not None, p2() is not None)

Expected output:

after return: True True
after clear:  False True
after gc:     False False

The trap: False False on the first line, on the reasoning that batch is a local variable and locals die when the function returns. Some candidates who know about tracebacks then answer False False on the second line too, expecting clear() to free both, and miss the cycle.

Explanation:

Two mechanisms combine.

1. An exception keeps its stack frames, and frames keep their locals. e.__traceback__ references the frame of process, and that frame references every local, including batch. Storing e in recent_errors therefore keeps each batch alive after process returns: True True. The /health list is effectively a list of DataFrames.

2. Whether the memory comes back on clear() depends on whether there's a reference cycle.

• Batch 1: Python deletes the except ... as e name at the end of the except block (an implicit del e), precisely to avoid a cycle. The only path to the frame is recent_errors → exception → traceback → frame. clear() drops the last reference, CPython's reference counting frees the chain at once, and p1() is dead.
• Batch 2: last_error = e creates a second local that is not auto-deleted. Now the frame references the exception (through last_error) and the exception references the frame (through its traceback): a cycle. Reference counts never reach zero, so only the cyclic garbage collector can free it. With GC disabled it survives clear(); gc.collect() frees it.

In a real service GC is enabled, but generation-2 collections are infrequent, so cycles holding hundreds of MB can sit there for a long time. Under memory pressure that is an OOM kill rather than a slow leak.

Takeaway: never keep exception objects longer than the request. Store traceback.format_exception(e) strings or a small error record, and avoid binding the exception to another local in the frame that raised it.

Follow-ups:

1. "How would you have found this in production without reading the code?" Watch RSS per worker over time (a sawtooth that only drops at deploys or restarts), then take snapshots: tracemalloc (compare two snapshots grouped by traceback), or gc.get_referrers / objgraph.show_backrefs on a large object to see the chain list → ValueError → traceback → frame → DataFrame. memray attached to a canary is the heavier-duty option. The tell-tale sign is frame or traceback objects on the path to your data.
2. "You need the stack trace on the health page. How do you keep it without keeping the frames?" Format it immediately: ''.join(traceback.format_exception(e)) and store the string with the batch ID and time in a bounded deque(maxlen=50). Alternatives are traceback.TracebackException(...), which captures a lightweight summary (without locals unless capture_locals=True), or traceback.clear_frames(e.__traceback__) to drop the locals while keeping the exception. Setting e.__traceback__ = None loses the trace entirely.
3. "Same bug, but in an asyncio service: a task fails and its exception is stored by task.exception() in a list. Anything different?" Same mechanism: the exception's traceback references the coroutine's frames and therefore its locals. Also note that a failed Task object itself holds its exception, so keeping a list of finished tasks (for example in a 'background jobs' registry) leaks in the same way; remove tasks in a done-callback and store a summary instead.

Scoring: Strong: Exact output; explains traceback → frame → locals, the implicit del e at the end of the except block, and the cycle created by last_error; knows that refcounting frees acyclic garbage immediately while cycles wait for the GC; proposes formatting tracebacks into a bounded structure and names tools (tracemalloc, gc.get_referrers, memray) to find it. Acceptable: Knows that stored exceptions keep locals alive (first line right) but expects both to be freed on clear(); understands the cycle once shown that last_error lives in the same frame. Red flag: Says locals are always freed on return and can't see how the list holds batch; or 'fixes' the leak by calling gc.collect() after every batch.

SR-MT-O03 · A library's deprecation warning that its users never see

Medium & Tricky · Output · Senior (5–8 years) · library design, deprecation, warnings, stacklevel, warning filters, API evolution · 10 min

Question: You maintain paysdk, an internal payments library used by many teams. You're deprecating pay() in favour of charge(). The snippet builds two small modules in memory (paysdk, and a consumer module billing) and runs with Python's default warning filters. showwarning is replaced only so that warnings print to stdout. What does it print?

import sys
import types
import warnings


def show(message, category, filename, lineno, *rest):
    where = filename if filename in ("paysdk.py", "billing.py") else "main script"
    print(f"  {category.__name__} ({where}, line {lineno}): {message}")


warnings.showwarning = show       # print to stdout so the demo is easy to read


def make_module(name, source):
    mod = types.ModuleType(name)
    exec(compile(source, f"{name}.py", "exec"), mod.__dict__)
    sys.modules[name] = mod
    return mod


paysdk = make_module("paysdk", '''
import warnings

def charge(amount, *, currency="INR"):
    return f"charged {amount} {currency}"

def pay(amount):
    warnings.warn("pay() is deprecated; use charge()", DeprecationWarning)
    return charge(amount)

def pay_v2(amount):
    warnings.warn("pay_v2() is deprecated; use charge()", DeprecationWarning, stacklevel=2)
    return charge(amount)
''')

billing = make_module("billing", '''
import paysdk

def monthly_invoice():
    return paysdk.pay_v2(999)
''')

print("1", paysdk.pay(100))
print("2", paysdk.pay_v2(100))
for _ in range(2):
    print("3", paysdk.pay_v2(200))
print("4", billing.monthly_invoice())

Expected output:

1 charged 100 INR
  DeprecationWarning (main script, line 44): pay_v2() is deprecated; use charge()
2 charged 100 INR
  DeprecationWarning (main script, line 46): pay_v2() is deprecated; use charge()
3 charged 200 INR
3 charged 200 INR
4 charged 999 INR

The trap: A warning on every one of the five calls (or at least on 1, 2 and 4), on the belief that warnings.warn(..., DeprecationWarning) always prints something. This is what most library authors assume when they add the warning, and why many deprecations reach the removal release with nobody having seen them.

Explanation:

Two mechanisms decide whether a warning is shown.

1. stacklevel decides which module the warning is attributed to. A warning is attributed to the module and line of the frame stacklevel levels up the stack. With the default stacklevel=1, pay()'s warning belongs to paysdk.py, the line that called warnings.warn. With stacklevel=2, pay_v2()'s warning belongs to the caller: the main script for calls 2 and 3, and billing.py for call 4.

2. The default filters show DeprecationWarning only when it is attributed to __main__. Since Python 3.7 the default filter list includes default::DeprecationWarning:__main__ followed by ignore::DeprecationWarning. So:

• Call 1 is attributed to paysdk, so it is ignored. With the default stacklevel, a library's own deprecation warning is never shown to anyone by default.
• Call 2 is attributed to the main script, so it is shown, pointing at the user's line (44), which is the line they need to change.
• Call 3 is shown once, not twice. The default action prints the first occurrence per location (module + line), and both loop iterations come from line 46.
• Call 4 is attributed to billing, which is not __main__, so it is ignored. Most real usage of a library comes from other modules, not from the entry script, so even with stacklevel=2 most callers see nothing in production.

This matches the intent of the defaults: DeprecationWarning is aimed at developers, who are expected to see it in their test runs (pytest shows them), not at end users.

Takeaway: pass stacklevel so the warning points at the caller, and don't rely on default filters for visibility. Make your consumers' CI run with -W error::DeprecationWarning (or pytest filterwarnings = error), or use FutureWarning when the audience is end users.

Follow-ups:

1. "You have 40 consuming teams and want to remove pay() in the next major version. What's your deprecation process?" Warn with stacklevel=2 and a message naming the replacement and the removal version; document it in the changelog; tell consumers to turn paysdk deprecations into errors in their test config (filterwarnings = error::DeprecationWarning:paysdk doesn't work because the module field matches the caller, so filter on the message or a custom PaysdkDeprecationWarning subclass); measure real usage, by emitting a metric or log on each deprecated call or by searching the monorepo or dependency graph, so you know who still calls it; then remove in a major version under semver, with the old API as a no-op shim only if the usage data says you need it.
2. "Why might a custom warning class (class PaysdkDeprecationWarning(DeprecationWarning)) be better than a bare DeprecationWarning?" Consumers can target it precisely: -W error::paysdk.PaysdkDeprecationWarning or filterwarnings with that category escalates only your deprecations without breaking on every third-party library's warnings. It also lets you add fields (removal version, replacement) and test for it with pytest.warns(PaysdkDeprecationWarning).
3. "pay() is called through a decorator, @deprecated("use charge"), that wraps the function. What stacklevel does the wrapper need?" One more level for each wrapper frame: inside the decorator's wrapper, stacklevel=2 points at the caller of wrapper, which is correct only if warn is called directly in wrapper. If it is called from a helper inside the wrapper, use 3. Python 3.13 adds warnings.deprecated (PEP 702), which does this for you and is also understood by type checkers; on 3.12 the same decorator is available as typing_extensions.deprecated.

Scoring: Strong: Exact output, including that call 3 prints once and call 4 prints nothing; explains stacklevel attribution, the __main__-only default filter and once-per-location; proposes a deprecation process (custom category, CI escalation, usage metrics, semver removal) suited to a library with many consumers. Acceptable: Knows DeprecationWarning is hidden by default and that stacklevel=2 points at the caller, but expects call 3 to print twice or call 4 to print; corrects after the filter list is shown. Red flag: Expects every call to warn; or has never used stacklevel and suggests print() or logging as the deprecation mechanism for a shared library.

SR-MT-O04 · Thread-local request context in a thread pool

Medium & Tricky · Output · Senior (5–8 years) · threading.local, ThreadPoolExecutor, request context, security, concurrency · 8 min

Question: A legacy service keeps the current user in a threading.local() and handles requests on a thread pool. The auth middleware sets ctx.user only when the request carries a token; public endpoints have none. With one worker thread (so the output is deterministic), what does this print?

import threading
from concurrent.futures import ThreadPoolExecutor

ctx = threading.local()


def current_user():
    return getattr(ctx, "user", "anonymous")


def handle(request):
    if "user" in request:                 # auth middleware sets it only when a token is present
        ctx.user = request["user"]
    ctx.calls = getattr(ctx, "calls", 0) + 1
    return f"{request['path']} as {current_user()} (call {ctx.calls} on this thread)"


requests = [
    {"path": "/orders", "user": "priya"},
    {"path": "/health"},
    {"path": "/orders", "user": "arjun"},
    {"path": "/catalog"},
]

with ThreadPoolExecutor(max_workers=1) as pool:
    for line in pool.map(handle, requests):
        print(line)

print("main thread:", current_user(), getattr(ctx, "calls", 0))

Expected output:

/orders as priya (call 1 on this thread)
/health as priya (call 2 on this thread)
/orders as arjun (call 3 on this thread)
/catalog as arjun (call 4 on this thread)
main thread: anonymous 0

The trap: /health as anonymous and /catalog as anonymous, with every call numbered 1, on the belief that thread-local means request-local. That holds in a thread-per-request server, where each request gets a new thread, and it is how many Flask-era codebases were written before moving to a pool.

Explanation:

Two mechanisms combine.

1. threading.local is per thread, not per task or request. Each thread sees its own attribute dictionary on ctx. The main thread never set anything, so it sees anonymous and 0 calls, while the worker's values are invisible to it.

2. A pool reuses its threads. ThreadPoolExecutor(max_workers=1) runs all four requests on the same worker thread, one after another. Nothing clears the thread-local between tasks. So:

• /health carries no token, so the middleware doesn't touch ctx.user and the request inherits priya from the previous request on that thread.
• /catalog likewise inherits arjun.
• ctx.calls keeps counting across requests (1–4), which shows the state is per thread, not per request.

With more workers the leak doesn't go away. It becomes non-deterministic, depending on which thread picks up which request, which makes it much harder to reproduce. In a real system this is a security incident: an anonymous request can act as, or see data of, the previous authenticated user on that thread, and audit logs attribute actions to the wrong person.

Takeaway: per-request state must be set and cleared at the request boundary (try/finally), or passed explicitly. In asyncio code use contextvars, which are scoped per task.

Follow-ups:

1. "What's the minimal fix in this code, and what's the better fix?" Minimal: always assign at the start of every request (ctx.user = request.get("user")) and clear in a finally (del ctx.user or reset to None) in the one place requests enter, a middleware or wrapper, so no handler can forget. Better: stop using ambient state for identity. Pass the user explicitly (FastAPI's dependency injection, Depends(get_current_user)), or use a ContextVar set with a token and reset with var.reset(token) in finally. That works for threads (when the executor copies the context) and for asyncio tasks.
2. "Would replacing threading.local with a contextvars.ContextVar alone fix it in this thread pool?" Not by itself. ThreadPoolExecutor.submit/map does not copy the caller's context; the task runs in the worker thread's own context, which persists across tasks, so a ContextVar.set in one task is still visible to the next task on that thread. You need contextvars.copy_context().run(handle, req) per task (as asyncio.to_thread does), or set/reset(token) in a finally. The discipline of resetting at the boundary matters more than the storage mechanism.
3. "How would you write a test that catches this class of bug?" Run an authenticated request followed by an unauthenticated one through the same worker (a single-thread executor, or the server with one worker), and assert the second sees no user. Make it a standard regression test for the middleware. Also add a check in the request wrapper that the context is empty on entry and log or alert if not, which catches leaks introduced later by code that sets context in unexpected places.

Scoring: Strong: Exact output; explains per-thread storage plus pool thread reuse and the conditional assignment; calls out the security impact and non-determinism with more workers; gives the set-and-reset-at-the-boundary fix, prefers explicit passing or ContextVars, and knows executors don't copy context by default. Acceptable: Realises the user leaks to /health but expects calls to reset per request, or misses that the main thread sees nothing; reaches the try/finally fix with a prompt. Red flag: Treats thread-local as request-local, or proposes adding a lock (which doesn't address the leak at all).

SR-MT-C01 · Async micro-batcher for a model-serving endpoint

Medium & Tricky · Coding · Senior (5–8 years) · model serving, asyncio, batching, futures, cancellation, error propagation, latency vs throughput · 35 min

Problem:

A FastAPI service scores fraud risk one transaction per request, but the model server (GPU) is much cheaper per row when called with a batch. Write an asyncio micro-batcher that sits between the endpoint and the model:

• await batcher.predict(x) returns the model's output for x.
• Pending inputs are sent as one await predict_batch(xs) call when the batch reaches max_batch items, or max_wait seconds after the batch's first item arrived, whichever comes first. A lone request must not wait for company forever.
• predict_batch returns a list of outputs in input order. If it raises, every caller in that batch gets that exception, and other batches are unaffected. If it returns the wrong number of results, every caller in the batch gets a RuntimeError; never hand anyone a possibly misaligned row.
• A caller that is cancelled (client disconnected, request timeout) before its batch is sent must not be sent to the model. Cancelling one caller must never affect the others.
• await batcher.close() sends whatever is pending immediately and waits for in-flight batches (used at shutdown).

Signature: class MicroBatcher(Generic[X, Y]): def __init__(self, predict_batch: Callable[[list[X]], Awaitable[list[Y]]], max_batch: int, max_wait: float) -> None async def predict(self, x: X) -> Y async def close(self) -> None

Constraints: Single event loop (one per worker process). Batches may run concurrently with each other. Typical settings: max_batch=32, max_wait=0.005. Standard library only.

Examples:

• max_batch=2; gather(predict(1), predict(2), predict(3), predict(4)) with a model returning x * 10 → [10, 20, 30, 40]; the model saw [[1, 2], [3, 4]]
• max_batch=8, max_wait=0.05; a single predict(7) → 70, after about 50 ms, in a batch of one
• max_batch=2; predict(1) and predict(-1) where the model raises on -1 → both callers get the ValueError; the next batch succeeds

Hints:

1. Give each caller its own Future, and resolve it from the task that runs the batch.
2. loop.call_later returns a handle. When the batch is flushed because it's full, what should happen to the timer started by its first item?
3. When a caller awaiting a future is cancelled, what happens to the future? How can the flush notice?
4. Who holds a reference to the batch task so it isn't garbage-collected mid-flight?

Reference solution:

import asyncio
from collections.abc import Awaitable, Callable
from typing import Generic, TypeVar

X = TypeVar("X")
Y = TypeVar("Y")


class MicroBatcher(Generic[X, Y]):
    """Group concurrent single predictions into batched model calls.

    A batch is sent when it reaches `max_batch` items, or `max_wait` seconds
    after its FIRST item arrived, whichever comes first. Each caller gets its
    own result; if the batch call fails, every caller in that batch gets the error.
    """

    def __init__(self, predict_batch: Callable[[list[X]], Awaitable[list[Y]]],
                 max_batch: int, max_wait: float) -> None:
        if max_batch < 1 or max_wait <= 0:
            raise ValueError("max_batch must be >= 1 and max_wait > 0")
        self._predict_batch = predict_batch
        self._max_batch = max_batch
        self._max_wait = max_wait
        self._pending: list[tuple[X, asyncio.Future[Y]]] = []
        self._timer: asyncio.TimerHandle | None = None
        self._running: set[asyncio.Task[None]] = set()     # keep references to batch tasks

    async def predict(self, x: X) -> Y:
        loop = asyncio.get_running_loop()
        fut: asyncio.Future[Y] = loop.create_future()
        self._pending.append((x, fut))
        if len(self._pending) >= self._max_batch:
            self._flush()
        elif self._timer is None:                           # first item of a new batch
            self._timer = loop.call_later(self._max_wait, self._flush)
        return await fut                                    # cancelling this cancels only `fut`

    def _flush(self) -> None:
        if self._timer is not None:
            self._timer.cancel()                            # the deadline belonged to THIS batch
            self._timer = None
        batch = [(x, f) for x, f in self._pending if not f.done()]   # drop cancelled callers
        self._pending = []
        if batch:
            task = asyncio.get_running_loop().create_task(self._run(batch))
            self._running.add(task)
            task.add_done_callback(self._running.discard)

    async def _run(self, batch: list[tuple[X, asyncio.Future[Y]]]) -> None:
        try:
            results = await self._predict_batch([x for x, _ in batch])
            if len(results) != len(batch):
                raise RuntimeError(f"model returned {len(results)} results for {len(batch)} inputs")
        except Exception as exc:
            for _, fut in batch:
                if not fut.done():
                    fut.set_exception(exc)
            return
        for (_, fut), y in zip(batch, results):
            if not fut.done():                              # caller may have given up meanwhile
                fut.set_result(y)

    async def close(self) -> None:
        """Send whatever is pending and wait for all in-flight batches."""
        self._flush()
        while self._running:
            await asyncio.gather(*self._running, return_exceptions=True)

Complexity: O(1) amortised per predict plus O(b) per batch of size b to build the input list and resolve futures. Memory is O(pending + in-flight items).

Tests:

import asyncio


class FakeModel:
    def __init__(self) -> None:
        self.batches: list[list[int]] = []
    async def predict_batch(self, xs: list[int]) -> list[int]:
        self.batches.append(list(xs))
        await asyncio.sleep(0.01)                 # GPU time
        if -1 in xs:
            raise ValueError("NaN in features")
        if 99 in xs:
            return [x * 10 for x in xs][:-1]      # buggy model server drops a row
        return [x * 10 for x in xs]


async def scenario() -> None:
    # size-triggered: 5 concurrent requests, batches of at most 2, results routed correctly
    m = FakeModel()
    b = MicroBatcher(m.predict_batch, max_batch=2, max_wait=5.0)
    out = await asyncio.wait_for(asyncio.gather(*(b.predict(i) for i in range(1, 5))), 1.0)
    assert out == [10, 20, 30, 40] and m.batches == [[1, 2], [3, 4]]

    # time-triggered: a lone request is not stuck waiting for company
    m = FakeModel()
    b = MicroBatcher(m.predict_batch, max_batch=8, max_wait=0.05)
    assert await asyncio.wait_for(b.predict(7), 1.0) == 70 and m.batches == [[7]]

    # the deadline belongs to the batch: a size flush cancels the old timer
    m = FakeModel()
    b = MicroBatcher(m.predict_batch, max_batch=3, max_wait=1.0)
    first = [asyncio.create_task(b.predict(i)) for i in (1, 2, 3)]     # flushed at t=0 by size
    await asyncio.sleep(0.4)
    d = asyncio.create_task(b.predict(4))                             # new batch, deadline t=1.4
    await asyncio.sleep(0.8)                                           # t=1.2: past the OLD deadline
    e = asyncio.create_task(b.predict(5))
    await asyncio.gather(*first, d, e)
    assert m.batches == [[1, 2, 3], [4, 5]], m.batches

    # a failing batch fails all of its callers, and only them
    m = FakeModel()
    b = MicroBatcher(m.predict_batch, max_batch=2, max_wait=5.0)
    r = await asyncio.gather(b.predict(1), b.predict(-1), b.predict(3), b.predict(4),
                             return_exceptions=True)
    assert [type(x).__name__ for x in r[:2]] == ["ValueError", "ValueError"] and r[2:] == [30, 40]

    # wrong number of results: everyone in the batch gets an error, nobody gets a wrong row
    r = await asyncio.gather(b.predict(5), b.predict(99), return_exceptions=True)
    assert all(isinstance(x, RuntimeError) for x in r)

    # a caller that gives up before the flush is not sent to the model
    m = FakeModel()
    b = MicroBatcher(m.predict_batch, max_batch=3, max_wait=0.05)
    gone = asyncio.create_task(b.predict(1))
    stay = asyncio.create_task(b.predict(2))
    await asyncio.sleep(0)
    gone.cancel()
    assert await stay == 20 and m.batches == [[2]]

    # close() flushes a partial batch without waiting for the deadline
    m = FakeModel()
    b = MicroBatcher(m.predict_batch, max_batch=10, max_wait=30.0)
    t = asyncio.create_task(b.predict(6))
    await asyncio.sleep(0)
    await asyncio.wait_for(b.close(), 1.0)
    assert await t == 60


asyncio.run(scenario())
print("all tests passed")

A common wrong solution (the tests catch it):

import asyncio
from collections.abc import Awaitable, Callable
from typing import Generic, TypeVar

X = TypeVar("X")
Y = TypeVar("Y")


class MicroBatcher(Generic[X, Y]):
    """Group concurrent single predictions into batched model calls.

    A batch is sent when it reaches `max_batch` items, or `max_wait` seconds
    after its FIRST item arrived, whichever comes first. Each caller gets its
    own result; if the batch call fails, every caller in that batch gets the error.
    """

    def __init__(self, predict_batch: Callable[[list[X]], Awaitable[list[Y]]],
                 max_batch: int, max_wait: float) -> None:
        if max_batch < 1 or max_wait <= 0:
            raise ValueError("max_batch must be >= 1 and max_wait > 0")
        self._predict_batch = predict_batch
        self._max_batch = max_batch
        self._max_wait = max_wait
        self._pending: list[tuple[X, asyncio.Future[Y]]] = []
        self._timer: asyncio.TimerHandle | None = None
        self._running: set[asyncio.Task[None]] = set()     # keep references to batch tasks

    async def predict(self, x: X) -> Y:
        loop = asyncio.get_running_loop()
        fut: asyncio.Future[Y] = loop.create_future()
        self._pending.append((x, fut))
        if len(self._pending) >= self._max_batch:
            self._flush()
        elif self._timer is None:                           # first item of a new batch
            self._timer = loop.call_later(self._max_wait, self._flush)
        return await fut                                    # cancelling this cancels only `fut`

    def _flush(self) -> None:
        self._timer = None                                  # forgets to cancel the old timer
        batch = list(self._pending)
        self._pending = []
        if batch:
            task = asyncio.get_running_loop().create_task(self._run(batch))
            self._running.add(task)
            task.add_done_callback(self._running.discard)

    async def _run(self, batch: list[tuple[X, asyncio.Future[Y]]]) -> None:
        try:
            results = await self._predict_batch([x for x, _ in batch])
            if len(results) != len(batch):
                raise RuntimeError(f"model returned {len(results)} results for {len(batch)} inputs")
        except Exception as exc:
            for _, fut in batch:
                if not fut.done():
                    fut.set_exception(exc)
            return
        for (_, fut), y in zip(batch, results):
            if not fut.done():                              # caller may have given up meanwhile
                fut.set_result(y)

    async def close(self) -> None:
        """Send whatever is pending and wait for all in-flight batches."""
        self._flush()
        while self._running:
            await asyncio.gather(*self._running, return_exceptions=True)

Alternatives and tradeoffs:

• A queue and a single consumer task: predict puts (x, future) on an asyncio.Queue; a background loop does first = await q.get(), then gathers more with asyncio.wait_for(q.get(), remaining) until full or the deadline passes. That gives one batch in flight at a time (natural backpressure toward a single GPU) and easier shutdown via a sentinel, but you must start and stop the loop with the app's lifespan.
• Server-side dynamic batching: NVIDIA Triton, TorchServe and Ray Serve (@serve.batch) do exactly this inside the model server. If you control the serving stack, prefer that to batching in every client.
• Tuning: max_wait adds directly to p50 latency at low traffic, while max_batch caps the GPU efficiency gain at high traffic. Measure throughput and p99 at your real request rates and choose per model. Many teams adapt max_wait to load.
• The broken version forgets to cancel the first item's timer when a batch is flushed for being full, so the stale timer later flushes the next batch early (smaller batches than configured), and it sends cancelled callers to the model, wasting GPU time.

Follow-ups:

1. "Your service runs 4 Uvicorn worker processes, each with its own batcher. At low traffic, batches are mostly of size 1. Why, and what would you do?" Requests are spread across 4 independent batchers, so each sees a quarter of the traffic and its max_wait window rarely catches a second request. Options: fewer workers per pod (async I/O doesn't need many when the GPU call is awaited), move batching into the model server so all clients share one batcher, or accept it, since at low traffic there's GPU headroom anyway and batching matters only at peak. Check the batch-size histogram metric before optimising.
2. "What metrics would you emit from this component?" A batch-size histogram, the time each item waited in the queue (time from predict to flush), model call latency, batch error count by exception type, the number of cancelled callers dropped before flush, and the in-flight batch count. Queue wait plus model latency explains the endpoint's p99; a rising queue wait with full batches means you are throughput-bound and need more replicas.
3. "The model call itself hangs for 30 seconds. What happens to callers, and what should?" Every caller in that batch hangs too, and with unbounded concurrent batches more pile up behind a sick GPU. Wrap predict_batch in asyncio.timeout(...) so the whole batch fails fast with a TimeoutError, cap in-flight batches with a semaphore so a hung model applies backpressure instead of piling up work, and let the endpoint map the timeout to a 503 so the load balancer can retry on another replica (scoring is read-only, so it is safe to retry).

Scoring: Strong: Per-caller futures; size flush cancels the pending timer; deadline measured from the batch's first item; cancelled callers filtered at flush and done() checked before setting results; errors and length mismatches fanned out to the batch only; task references kept; close() implemented. Discusses latency/throughput tuning, metrics and timeouts. About 35 minutes. Acceptable: Working size- and time-triggered batching with correct result routing, but misses the stale-timer bug or cancelled callers; fixes them when the test is shown. Red flag: Awaits the model call inline in the caller that fills the batch (so that caller's cancellation cancels everyone's batch), or sleeps max_wait on every request; can't explain how results are routed back to the right caller.

SR-MT-C02 · Safe commit offsets for a parallel Kafka consumer

Medium & Tricky · Coding · Senior (5–8 years) · stream processing, at-least-once delivery, offset management, concurrency, redelivery, data loss · 30 min

Problem:

A consumer reads one Kafka partition of payment events and hands each message to a pool of worker threads, so messages finish out of order. Periodically it commits an offset, and after a crash or rebalance it resumes from the message after the committed one. Write the bookkeeping class that decides what is safe to commit.

• OffsetTracker(committed): committed is the last offset whose effects are already durable.
• start(offset) -> bool: called when a message is received. Return False (skip it) if it is a redelivery: at or below the commit point, currently in flight, or already finished. Otherwise record it as in flight and return True.
• finish(offset): the message's side effects are durable.
• commit_point() -> int: advance and return the highest offset that is safe to commit. Never commit past a message that is still in flight, because if the process dies, that message would never be processed. The commit point never moves backwards.

Offsets in a partition increase but are not necessarily contiguous: compaction and transaction markers leave gaps, so offset 105 may simply not exist. Messages arrive in offset order, but redeliveries can repeat earlier ones.

Signature: class OffsetTracker: def __init__(self, committed: int) -> None def start(self, offset: int) -> bool def finish(self, offset: int) -> None def commit_point(self) -> int

Constraints: One tracker per partition, called from the consumer's poll loop (single thread); workers report completion back to that loop. Thousands of messages in flight at most. Kafka itself stores the next offset to read, so the poll loop commits commit_point() + 1. Standard library only.

Examples:

• OffsetTracker(99); start 100..103; finish 101, 103; commit_point() → 99: 100 is still running
• then finish(100); commit_point() → 101: 102 is still running
• offsets 104, 106, 108 (105 and 107 don't exist); finish 104 and 108; commit_point() → 105: just below the oldest in-flight offset (106)

Hints:

1. What is the smallest offset you must not commit past?
2. If nothing is in flight, what is the right commit point, given that the offsets may have gaps?
3. Which data structure tells you the oldest in-flight offset cheaply? Is a set good enough at this scale?
4. Why is 'commit the highest finished offset' the classic data-loss bug?

Reference solution:

class OffsetTracker:
    """Decide which offset of ONE partition is safe to commit when messages
    finish out of order (a worker pool) and the broker may redeliver.

    `committed` is the last offset whose effects are durable; on restart the
    consumer resumes after it. Offsets increase but need not be contiguous
    (compaction and transaction markers leave gaps).
    """

    def __init__(self, committed: int) -> None:
        self.committed = committed
        self._in_flight: set[int] = set()
        self._done_max = committed            # highest finished offset so far

    def start(self, offset: int) -> bool:
        """Register a received message. False means: skip it, it's a redelivery."""
        if offset <= self.committed or offset in self._in_flight:
            return False
        if offset <= self._done_max:          # already finished above the commit point
            return False                      # (only correct if offsets are never reused)
        self._in_flight.add(offset)
        return True

    def finish(self, offset: int) -> None:
        """Mark a started message as fully processed (side effects durable)."""
        self._in_flight.remove(offset)        # KeyError on an unknown offset: a caller bug
        self._done_max = max(self._done_max, offset)

    def commit_point(self) -> int:
        """Advance and return the highest offset that is safe to commit."""
        if self._in_flight:
            safe = min(self._in_flight) - 1   # never pass the oldest unfinished message
        else:
            safe = self._done_max
        self.committed = max(self.committed, safe)
        return self.committed

Complexity: start/finish: O(1). commit_point: O(k) for k messages in flight using min() over a set; use a heap with lazy deletion (or a sorted container) to make it O(log k) if k is large. Space O(k).

Tests:

t = OffsetTracker(committed=99)
for off in (100, 101, 102, 103):
    assert t.start(off)

# 101 and 103 finish first: 100 is still running, so nothing can be committed
t.finish(101); t.finish(103)
assert t.commit_point() == 99
# 100 finishes: commit up to 101 only, because 102 is still in flight
t.finish(100)
assert t.commit_point() == 101
t.finish(102)
assert t.commit_point() == 103

# gaps in offsets (compacted topic): 105 and 107 don't exist
assert t.start(104) and t.start(106) and t.start(108)
t.finish(104); t.finish(108)
assert t.commit_point() == 105            # 106 is still running; 105 never existed, harmless
t.finish(106)
assert t.commit_point() == 108

# redelivery after a rebalance: already committed -> skip, not reprocessed
assert t.start(103) is False and t.start(108) is False
# a duplicate of a message that is still being processed -> skip
assert t.start(109) and t.start(109) is False
# duplicate of a message finished above the commit point (110 done, 109 not)
assert t.start(110)
t.finish(110)
assert t.commit_point() == 108
assert t.start(110) is False
t.finish(109)
assert t.commit_point() == 110
# commit point never goes backwards
assert t.commit_point() == 110 and t.committed == 110

# a fresh tracker after restart starts from the committed offset
t2 = OffsetTracker(committed=110)
assert t2.commit_point() == 110 and t2.start(111)
print("all tests passed")

A common wrong solution (the tests catch it):

class OffsetTracker:
    # Commits the highest FINISHED offset. Fast and simple, but if the process
    # dies, anything below it that was still running is never reprocessed.
    def __init__(self, committed: int) -> None:
        self.committed = committed
        self._seen: set[int] = set()

    def start(self, offset: int) -> bool:
        if offset <= self.committed or offset in self._seen:
            return False
        self._seen.add(offset)
        return True

    def finish(self, offset: int) -> None:
        self.committed = max(self.committed, offset)

    def commit_point(self) -> int:
        return self.committed

Alternatives and tradeoffs:

• Heap of in-flight offsets with lazy deletion makes commit_point O(log k); worth it only with tens of thousands in flight.
• Per-key ordering instead of per-partition: libraries such as Confluent's parallel consumer process different keys in parallel but the same key in order, and track offsets the same way, encoding completed offsets above the commit point in the commit metadata so fewer messages are reprocessed after a restart.
• Commit only after a batch fully completes (process a poll batch, wait for all of it, commit its last offset). Much simpler and often fast enough; you give up some parallelism because one slow message holds up the batch.
• The broken version commits the highest finished offset. It looks fine in a happy-path test, but if the process dies while 100 is still running and 103 is committed, message 100 is never processed: silent data loss.

Follow-ups:

1. "After a crash, a new consumer resumes from the commit point and reprocesses 100–103, even though 101 and 103 had finished. Is that a bug?" It's the at-least-once contract: anything above the commit point may be redelivered, and this tracker's in-memory 'finished' set is lost with the process. So the processing itself must be idempotent: deduplicate on an event ID in the same transaction as the side effect, or make the write a version-guarded upsert. The tracker minimises duplicates in the normal case; idempotency makes duplicates harmless in the crash case.
2. "One message takes 10 minutes (a slow downstream). What happens to commits and to the consumer, and what do you do?" The commit point is stuck just below it, so lag appears to grow even though other messages are processed, and a crash would replay 10 minutes of work. Also, if in-flight work keeps growing, memory grows. Bound in-flight messages (pause the partition when the limit is reached), put a timeout on processing, and send messages that keep failing or timing out to a retry or dead-letter topic so they stop blocking the commit point. Alert on 'oldest in-flight age', not just consumer lag.
3. "How does this change if the side effect is writing to Postgres, and you want effectively-once?" Store the consumed offset in the same Postgres transaction as the side effect (a table (topic, partition) → offset) and, on startup, seek to the stored offset instead of using Kafka's committed offset. Then the effect and the progress marker commit atomically, and a crash can't separate them. Kafka transactions give a similar guarantee only when both input and output are Kafka topics.

Scoring: Strong: Commit point = min(in-flight) − 1, or the highest finished offset when nothing is in flight; handles gaps, redeliveries and the never-backwards rule; explains why 'max finished' loses data; ties it to idempotent processing, bounded in-flight and dead-letter handling. About 30 minutes. Acceptable: Correct for contiguous offsets (advances while the next offset is done) but stalls on gaps; or misses duplicate detection for in-flight offsets; fixes with a hint about compaction. Red flag: Commits the latest finished or latest received offset, and doesn't see the data-loss scenario even when a crash is described.

SR-MT-C03 · Bulk API client with partial failures and stable idempotency keys

Medium & Tricky · Coding · Senior (5–8 years) · idempotency, retries, partial failure, exponential backoff, API clients, backfills · 30 min

Problem:

A nightly job pushes settled orders to the finance team's ledger API in bulk. The API is partially failing by design: for each item it answers "ok", "retryable" (throttled, or a shard briefly down) or "rejected" (invalid, never retry). Sometimes the whole call raises ConnectionError, and then you don't know whether the server applied the items: a read timeout after the server committed looks exactly like a connect failure.

The API deduplicates on an idempotency_key per item. Write bulk_send(records, send, ...):

• Each item sent is {"idempotency_key": key, "record": record}. send(items) returns {key: status} or raises ConnectionError.
• Retry only items that are not settled: retryable ones, ones missing from the reply, and all of them after a ConnectionError. Never resend "ok" or "rejected" items.
• Up to max_attempts calls in total, sleeping base_delay * 2 ** (attempt - 1) between attempts (0.5, 1.0, 2.0, ...) via the injected sleep. Don't sleep after the last attempt or when nothing is left.
• Return {order_id: "ok" | "rejected" | "failed"}, where "failed" means attempts were exhausted.
• A retry after an ambiguous failure, and a re-run of the whole job tomorrow, must not apply any record twice.

Also write idempotency_key(record).

Signature: def idempotency_key(record: dict) -> str: def bulk_send(records: list[dict], send: Callable[[list[dict]], dict[str, str]], *, max_attempts: int = 4, base_delay: float = 0.5, sleep: Callable[[float], None] = time.sleep) -> dict[str, str]:

Constraints: Records are JSON-serialisable dicts with a unique order_id within one call. Up to a few thousand records per call (chunking is the caller's job). Standard library only.

Examples:

• 4 records, one with amount -5; the API answers normally → {'O0': 'ok', 'O1': 'ok', 'O2': 'rejected', 'O3': 'ok'}; one call, no sleep
• same, but O1 is 'retryable' once → second call contains only O1; sleep(0.5) once; O1 ends 'ok'
• first call: the server applies everything, then the connection drops → the retry resends with the SAME keys; the server applies each order exactly once

Hints:

1. When exactly must the key be created so that attempt 3 and tomorrow's re-run use the same one?
2. What should a record's status be if it's missing from the server's reply?
3. Derive the key from what? A random UUID per record, created once, survives retries, but does it survive a re-run of the job?

Reference solution:

import hashlib
import json
import time
from collections.abc import Callable


def idempotency_key(record: dict) -> str:
    """Same record -> same key, on every attempt and every re-run of the job."""
    canonical = json.dumps(record, sort_keys=True, separators=(",", ":"), default=str)
    return hashlib.sha256(canonical.encode()).hexdigest()[:32]


def bulk_send(records: list[dict], send: Callable[[list[dict]], dict[str, str]], *,
              max_attempts: int = 4, base_delay: float = 0.5,
              sleep: Callable[[float], None] = time.sleep) -> dict[str, str]:
    """Push records to a bulk API that may partially fail. Returns order_id -> final status.

    `send(items)` takes [{"idempotency_key": ..., "record": ...}] and returns
    {idempotency_key: "ok" | "retryable" | "rejected"}, or raises ConnectionError
    when the outcome of the whole call is unknown.
    Final statuses: "ok", "rejected" (never retried) or "failed" (attempts exhausted).
    """
    pending = {idempotency_key(r): r for r in records}       # keys fixed BEFORE the first attempt
    final: dict[str, str] = {}
    for attempt in range(1, max_attempts + 1):
        items = [{"idempotency_key": k, "record": r} for k, r in pending.items()]
        try:
            statuses = send(items)
        except ConnectionError:
            statuses = {}                     # unknown outcome: retry all of them, same keys
        for key in list(pending):
            status = statuses.get(key, "retryable")         # missing from the reply = unknown
            if status in ("ok", "rejected"):
                final[pending.pop(key)["order_id"]] = status
        if not pending:
            break
        if attempt < max_attempts:
            sleep(base_delay * 2 ** (attempt - 1))
    for record in pending.values():
        final[record["order_id"]] = "failed"
    return final

Complexity: O(n × attempts) to build batches, plus O(total record size) to hash each record once. O(n) extra space.

Tests:

class FakeLedgerAPI:
    """Dedups on idempotency_key, like a real payments/ledger API."""
    def __init__(self, script: list[str], flaky: set[str] = frozenset()) -> None:
        self.script = list(script)        # per call: "normal", "drop_after_apply", "down"
        self.flaky = set(flaky)           # order_ids that return "retryable" once
        self.seen_keys: dict[str, str] = {}
        self.applied: list[str] = []      # order_ids, once per real write
        self.batches: list[list[str]] = []

    def send(self, items: list[dict]) -> dict[str, str]:
        mode = self.script.pop(0) if self.script else "normal"
        self.batches.append([i["record"]["order_id"] for i in items])
        if mode == "down":
            raise ConnectionError("connect timeout")
        out = {}
        for item in items:
            key, rec = item["idempotency_key"], item["record"]
            if rec["amount"] <= 0:
                out[key] = "rejected"
            elif rec["order_id"] in self.flaky:
                self.flaky.discard(rec["order_id"])
                out[key] = "retryable"
            else:
                if key not in self.seen_keys:
                    self.seen_keys[key] = rec["order_id"]
                    self.applied.append(rec["order_id"])
                out[key] = "ok"
        if mode == "drop_after_apply":
            raise ConnectionError("read timeout")   # server applied, client never heard back
        return out


recs = [{"order_id": f"O{i}", "amount": a} for i, a in enumerate([100, 250, -5, 999])]

# happy path: one call, rejected record reported, nothing retried
api, slept = FakeLedgerAPI([]), []
assert bulk_send(recs, api.send, sleep=slept.append) == \
    {"O0": "ok", "O1": "ok", "O2": "rejected", "O3": "ok"}
assert len(api.batches) == 1 and slept == []

# partial failure: only the retryable record is sent again
api, slept = FakeLedgerAPI([], flaky={"O1"}), []
assert bulk_send(recs, api.send, sleep=slept.append)["O1"] == "ok"
assert api.batches[1] == ["O1"] and slept == [0.5]

# ambiguous failure: server applied, connection dropped -> retry must NOT double-apply
api, slept = FakeLedgerAPI(["drop_after_apply"]), []
result = bulk_send(recs, api.send, sleep=slept.append)
assert result == {"O0": "ok", "O1": "ok", "O2": "rejected", "O3": "ok"}
assert sorted(api.applied) == ["O0", "O1", "O3"], api.applied

# outage: exponential backoff, then "failed" without crashing the job
api, slept = FakeLedgerAPI(["down"] * 10), []
assert set(bulk_send(recs, api.send, max_attempts=4, sleep=slept.append).values()) == {"failed"}
assert slept == [0.5, 1.0, 2.0] and len(api.batches) == 4

# re-running the whole job later (e.g. a backfill) is also safe: same keys
api = FakeLedgerAPI([])
bulk_send(recs, api.send, sleep=lambda s: None)
bulk_send(recs, api.send, sleep=lambda s: None)
assert sorted(api.applied) == ["O0", "O1", "O3"]
assert idempotency_key({"a": 1, "b": 2}) == idempotency_key({"b": 2, "a": 1})
print("all tests passed")

A common wrong solution (the tests catch it):

import itertools
import time
from collections.abc import Callable

_counter = itertools.count()


def idempotency_key(record: dict) -> str:
    # A fresh key per send, like uuid4() inside the retry loop
    return f"{record['order_id']}-{next(_counter)}"


def bulk_send(records: list[dict], send: Callable[[list[dict]], dict[str, str]], *,
              max_attempts: int = 4, base_delay: float = 0.5,
              sleep: Callable[[float], None] = time.sleep) -> dict[str, str]:
    pending = list(records)
    final: dict[str, str] = {}
    for attempt in range(1, max_attempts + 1):
        items = [{"idempotency_key": idempotency_key(r), "record": r} for r in pending]
        try:
            statuses = send(items)
        except ConnectionError:
            statuses = {}
        still = []
        for item in items:
            status = statuses.get(item["idempotency_key"], "retryable")
            if status in ("ok", "rejected"):
                final[item["record"]["order_id"]] = status
            else:
                still.append(item["record"])
        pending = still
        if not pending:
            break
        if attempt < max_attempts:
            sleep(base_delay * 2 ** (attempt - 1))
    for record in pending:
        final[record["order_id"]] = "failed"
    return final

Alternatives and tradeoffs:

• Key = business identity + version (f\"{order_id}:{settlement_version}\") instead of a content hash. It's more explicit and survives harmless changes such as a reordered or added field, but you need a reliable version. A content hash treats any change as a new operation: a corrected amount gets a new key and is applied again. For a ledger that may be exactly wrong, so decide deliberately which one you want.
• uuid4() generated once per record and persisted (in an outbox table, say) before the first attempt: survives retries and re-runs, at the cost of a write.
• Jitter: with many workers retrying at the same moment, add random jitter (delay * random.uniform(0.5, 1.5)) to avoid synchronised retry waves; inject the random source for tests.
• Honour Retry-After if the API sends it on throttled items, instead of your own schedule.
• The broken version builds keys inside the retry loop (as uuid4() per attempt would), so after an ambiguous ConnectionError the server sees new keys and applies the same orders twice.

Follow-ups:

1. "The ledger API's dedup window is 24 hours. Your job's backfill re-sends a month of data. What happens?" Keys older than 24 hours have been forgotten, so the backfill applies everything again: idempotency keys protect retries, not replays beyond the window. For backfills you need idempotency in the data: the API should upsert on a natural key (order_id plus version) or reject duplicates permanently, or the job must check what's already in the ledger (a reconciliation diff) and send only the missing records. Ask the API owners which guarantee they offer; it's part of the contract.
2. "How do you report the result of the job so that finance trusts it?" Emit counts by final status (ok / rejected / failed), the amount totals per status, and the list of rejected and failed order IDs with reasons to a dead-letter table for follow-up. Fail the job (non-zero exit, alert) if anything is failed; rejected is a data-quality issue routed to the owning team. Reconcile the totals against the source the next morning.
3. "Twenty instances of this job run in parallel during a backfill and the ledger API starts throttling everyone. How do you keep it from getting worse?" Exponential backoff with jitter per instance, plus a global concurrency limit: fewer parallel workers, or a shared token bucket in Redis. Treat throttling as a signal to slow down overall, not just to retry this batch. A retry budget (stop retrying when more than X% of calls are retries) and a circuit breaker around the API stop a retry storm from making an outage worse.

Scoring: Strong: Keys fixed before the first attempt and stable across re-runs; retries only unsettled items; missing-from-reply treated as retryable; correct backoff schedule with no trailing sleep; failed status on exhaustion; discusses key derivation trade-offs (content hash vs business key), dedup windows vs backfills, and jitter or global rate limiting. Acceptable: Correct partial-failure retry and backoff, but derives keys per attempt or uses uuid4() per run; understands the ambiguous-failure double-apply once the drop-after-apply case is described. Red flag: Retries the whole batch, including ok and rejected items, with no keys, or treats ConnectionError as 'nothing happened'; no notion that a timeout can follow a successful write.

SR-MT-D01 · Outbound webhook delivery for 20,000 merchants

Medium & Tricky · Design · Senior (5–8 years) · webhooks, at-least-once delivery, retries and backoff, ordering, tenant isolation, queues, security, idempotency · 45 min

Scenario:

A payment gateway notifies merchants of events (payment.captured, payment.failed, refund.processed, settlement.completed) by POSTing JSON to a URL each merchant registers. There are ~20,000 active merchants and ~3 million events/day, peaking at ~500/s during sales. Merchant endpoints vary wildly: big merchants answer in 50 ms, while a small Shopify-style store in Indore might take 20 s or be down for a day.

Today a Django view fires the webhook synchronously right after the payment is captured, with requests.post(url, timeout=30) and no retries. Merchants complain about missed webhooks, and last week one merchant's endpoint hanging for 30 s tied up the workers and delayed payment captures for everyone.

Design a webhook delivery system in Python. Note that two goals pull against each other: retrying until delivered and delivering in order per payment.

Clarifying questions:

• What delivery guarantee do we promise merchants: at-least-once with retries for how long? (At least once, retried for 72 hours.)
• Do merchants need ordering? (Per payment, yes; refund.processed must not arrive before payment.captured for the same payment. No ordering across payments.)
• Can merchants re-fetch event state from our API, and can they ask us to replay events?
• How do merchants verify a webhook is genuinely from us?
• Are some events more urgent than others (captures vs daily settlement summaries)?
• What's the SLO? (p95 first-attempt delivery within 5 s of the event.)

Reference design:

Decouple from the payment path (transactional outbox): the capture transaction inserts the event into an outbox table in the same database transaction as the payment state change. A relay process publishes outbox rows to Kafka (or a Postgres-backed queue) and marks them sent. A capture can no longer be slowed by a merchant, and no event is lost if the process crashes after commit.

Delivery workers: async Python workers (asyncio + httpx.AsyncClient) consume events, sign and POST them. I/O-bound work with long waits is what asyncio is good at: one process can hold thousands of in-flight requests. Strict timeouts: connect 3 s, total 10 s. Response 2xx = delivered; 410 = endpoint gone (disable it); everything else, including timeouts, is retryable.

Retries without blocking others: a failed delivery is not retried inline. It's written to a retry schedule (a deliveries table with next_attempt_at, or delay topics such as 1 min, 5 min, 30 min, 2 h, 6 h) with exponential backoff plus jitter, up to 72 hours, then marked failed, alerted to the merchant (email or dashboard), and kept for manual replay.

The interaction: retries vs ordering. If event 1 for payment P is retrying and event 2 for P is delivered meanwhile, the merchant sees them out of order. Options: - Per-key ordering: partition by payment_id and don't deliver event N+1 for a payment while event N is pending. This preserves order, but one failing event blocks later events for that payment only. Implement it with a per-payment sequence number and a check that the previous sequence was delivered or abandoned. - Don't promise order; make it unnecessary (what Stripe and Razorpay largely do): include created_at, a per-payment sequence, and the full current object state in every event, and tell merchants to fetch the latest state from the API or ignore older sequences. Delivery stays fully parallel, and the merchant's handler becomes order-tolerant. - Recommended: the second as the contract, plus a per-payment sequence, which is cheap and turns ordering into the merchant's simple comparison rather than our head-of-line-blocking problem.

Tenant isolation (the other interaction: retries vs fairness): a merchant that is down generates a growing retry backlog, and a slow merchant holds connections. Apply per-merchant concurrency limits (e.g. at most 10 in flight per endpoint, an asyncio.Semaphore per merchant in each worker plus a global token bucket in Redis), and a per-endpoint circuit breaker: after N consecutive failures, park that merchant's deliveries until a probe succeeds. Separate queues or priority lanes for slow or failing merchants keep healthy merchants under the 5 s SLO.

Security and idempotency: HMAC-SHA256 signature over timestamp + "." + raw_body with a per-merchant secret (support two active secrets during rotation); a stable event_id header so merchants can deduplicate, since at-least-once means duplicates will happen. Block SSRF: resolve and reject private, loopback and link-local addresses for merchant URLs, re-checking on every delivery (DNS can change), and don't follow redirects.

Observability and operations: per-merchant success rate, latency, retry backlog and oldest pending event age; a merchant-facing delivery log with request and response snippets; a replay API (by event ID or time range), rate-limited.

Trade-offs:

• Strict per-payment ordering vs throughput: blocking on the previous event guarantees order but creates head-of-line blocking and complex state; order-tolerant events with sequence numbers push a little work to merchants but keep delivery simple and parallel.
• Kafka vs a Postgres queue (SELECT ... FOR UPDATE SKIP LOCKED): Kafka scales and gives partition ordering, but scheduled retries are awkward (delay topics). A Postgres deliveries table makes per-row next_attempt_at, per-merchant queries and replay trivial, and is plenty for ~35 events/s average and 500/s peaks with good indexes.
• asyncio workers vs Celery with threads: asyncio handles thousands of slow sockets per process cheaply; Celery is familiar and has retries built in, but per-task retries with long countdowns sit in the broker and make per-merchant fairness harder.
• Retry window length: longer windows recover more deliveries after merchant outages but grow the backlog and the replay storm when the merchant comes back. Rate-limit the drain.

Deep-dive probes:

• "A big merchant was down for 6 hours and just came back; 400,000 queued events are now being delivered." Look for: draining at a controlled rate (the per-merchant limit, ramping up), delivering the newest state first or collapsing superseded events for the same payment, protecting other merchants' SLO, and coordinating with the merchant.
• "A merchant says they received payment.captured twice and shipped the order twice." Look for: at-least-once being inherent (timeout after their server processed it), the event_id for deduplication, documentation and sample handler code, and checking our own duplicate rate (relay re-publishing outbox rows).
• "Webhooks are delivered but payments are captured 2 s later than before." Look for: the outbox relay polling interval vs Postgres logical replication or CDC (Debezium), and confirming that the capture path no longer does any network I/O.
• "A merchant registers http://169.254.169.254/latest/meta-data/ as their webhook URL." Look for: SSRF defences at registration and at delivery time (DNS rebinding), egress through a proxy that blocks internal ranges, no redirects, and response bodies truncated before being shown in the merchant dashboard.

Rubric: Strong: Removes delivery from the payment path via an outbox; scheduled retries with backoff and jitter instead of inline retries; explicitly resolves the retry-vs-ordering tension (sequence numbers plus full-state events, or per-key blocking with its cost) and the retry-vs-fairness tension (per-merchant limits, circuit breakers, isolation lanes); HMAC signing with rotation, event IDs for dedup, SSRF protection; operational tooling for replay and drain. Acceptable: Queue plus workers with retries and signatures, but treats ordering as free or ignores per-merchant isolation until probed; reaches reasonable answers when shown the slow-merchant and outage scenarios. Red flag: Keeps sending webhooks inside the payment request (or in a thread from it), retries with time.sleep inside a worker, promises exactly-once delivery, or has no answer for one slow merchant delaying everyone.

SR-MT-D02 · A month-end payout job that must run exactly once in effect

Medium & Tricky · Design · Senior (5–8 years) · scheduled jobs, distributed locks, leases, fencing tokens, idempotency, state machines, time zones, reliability · 45 min

Scenario:

A marketplace pays ~60,000 sellers at month-end. A Python job runs at 00:30 IST on the 1st: it computes each seller's payable amount from the ledger, calls the bank's payout API (NEFT/IMPS through a partner bank, ~5 calls/s allowed), and records the payout. The full run takes ~4 hours.

The job runs as a cron entry inside the app container. When the app was scaled to 3 replicas, sellers were paid three times. A quick fix added a Redis lock (SET payout_lock 1 NX EX 3600). Last month, the lock expired after an hour while the job was still running (a slow bank), a second replica started the job, and ~4,000 sellers were paid twice, ₹1.8 crore that finance is still recovering.

Design the payout job so it is safe to schedule redundantly, safe to crash and restart, and never pays a seller twice for the same period. Two mechanisms interact: mutual exclusion with leases and idempotent effects.

Clarifying questions:

• Does the bank's payout API accept a client reference or idempotency key, and does it reject duplicates? For how long? (It rejects a duplicate client_ref within 30 days.)
• Can we query the bank for a payout's status by our reference? (Yes, a status API, eventually consistent within ~10 minutes.)
• Is the payable amount fixed at a cut-off time (the ledger as of 23:59:59 IST on the last day) or computed at payment time?
• What happens with sellers whose bank details fail validation? (They're held and retried after the seller fixes the details.)
• Who needs to be able to see progress or stop the run? (Finance ops, via a dashboard and a kill switch.)
• Do we have Postgres available for job state? (Yes, the ledger lives there.)

Reference design:

Principle: exclusion is an optimisation, idempotency is the guarantee. No lock with a timeout can guarantee a single runner: a process can pause (GC, a VM migration, a 20-minute bank call) past its lease and keep running. So design for "two runners may overlap" and make that harmless.

1. Freeze the plan, then execute it. Step one, as a single transaction: compute payables from the ledger as of the cut-off (2026-09-30 23:59:59 IST, stored as an aware UTC timestamp) and insert one row per seller into payouts(period, seller_id, amount_paise, status, client_ref, attempt, ...) with UNIQUE(period, seller_id). client_ref is deterministic: PAYOUT-2026-09-<seller_id>. Re-running the planning step inserts nothing new (ON CONFLICT DO NOTHING), so the plan is idempotent and auditable before any money moves.

2. Execute rows through a state machine: planned → submitting → submitted → paid | failed | held. A worker claims a batch with UPDATE ... SET status='submitting', lease_owner=:me, lease_until=now()+5 min WHERE status='planned' ... FOR UPDATE SKIP LOCKED, so several workers can run in parallel without claiming the same seller. It calls the bank with client_ref, then sets submitted with the bank's reference.

3. The ambiguous window. If a worker dies after calling the bank but before recording submitted, the row stays submitting with an expired lease. A recovery step must not blindly resubmit. It queries the bank's status API by client_ref first: found → mark submitted; not found (and older than the bank's consistency window) → resubmit with the same client_ref, which the bank deduplicates anyway. Two layers of protection: our state machine and the bank's idempotency.

4. Leader election only for scheduling. Run the trigger from one place: a Kubernetes CronJob with concurrencyPolicy: Forbid, or an orchestrator such as Airflow, instead of cron in every replica. If a lock is still wanted (to avoid wasted work), use a lease with heartbeats (renew every 30 s, expire after 2 min) and fencing tokens: each acquisition gets an increasing number (a Postgres sequence), and every state transition includes WHERE fence <= :my_token and records the token, so a paused old holder's late writes are rejected. Postgres advisory locks (pg_try_advisory_lock) are another option: held per session, released automatically if the connection dies.

5. Rate and safety controls: a token bucket at the bank's 5 calls/s; a per-run cap on total amount (refuse to execute a plan whose total deviates more than X% from last month without approval); a kill switch checked between batches; a dry-run mode.

Observability: counts and amounts per status, rate of bank calls, rows stuck in submitting, the reconciliation result (our submitted vs the bank's statement next morning), and an alert on any client_ref the bank reports as a duplicate, which should never happen.

Trade-offs:

• Redis SET NX EX lock vs Postgres advisory lock vs a CronJob with Forbid: Redis locks are easy but time-based (the incident); advisory locks tie exclusion to a live DB session (no TTL guessing), but a stuck-but-connected process keeps it forever; the scheduler-level Forbid prevents most overlaps without application code. None of them replaces idempotency.
• One big job vs a planned queue of rows: a monolithic loop is simple but restarts from scratch or needs ad-hoc checkpoints; the plan table makes progress, parallelism and restarts natural, at the cost of a schema and a state machine.
• Freezing amounts at plan time vs computing at execution: freezing gives auditability and consistency (finance approves a known total) but needs a correction flow for late adjustments (carried into next month's payout); computing at execution is fresher but makes reruns non-deterministic.
• Parallel workers vs a single worker: parallelism shortens the 4-hour window, but the bank's rate limit is the real bottleneck; beyond that, parallelism only adds coordination risk.

Deep-dive probes:

• "Walk me through exactly how last month's double payment happened, and show me that your design survives the same sequence." Look for: lease expiry with the holder still alive, a second runner starting, both calling the bank for the same sellers; then the fix: a deterministic client_ref deduplicated by the bank, row claiming with SKIP LOCKED, fenced state transitions, recovery that checks status before resubmitting.
• "The job crashed at 02:10 after 35,000 sellers. What happens when it restarts at 02:15?" Look for: rows in planned continue; rows in submitting go through status-check recovery; no re-planning changes amounts; progress visible on the dashboard.
• "Cron says 00:30 on the 1st, but the container runs in UTC." Look for: the 00:30 IST run is 19:00 UTC on the last day of the previous month, so the schedule must be expressed in IST (CronJob timeZone: Asia/Kolkata) and the cut-off computed with zoneinfo, not naive datetimes. A period key derived from the UTC date would point at the wrong month.
• "Finance wants to re-run payouts for 300 sellers whose bank details were fixed." Look for: held → planned transitions for just those rows with the same client_ref (or a new attempt suffix if the first attempt definitively failed at the bank), an audit trail, and approval before execution.

Rubric: Strong: States that locks with TTLs can't guarantee single execution and builds correctness on idempotency: a frozen plan with a unique key per seller and period, deterministic bank references, a claim-and-execute state machine, and status-check recovery for the ambiguous window; uses leases with heartbeats and fencing tokens (or advisory locks) only as an optimisation; handles IST scheduling and cut-offs correctly; adds amount guards, a kill switch and reconciliation. Acceptable: Proposes a better lock (longer TTL with renewal, or advisory locks) plus a per-seller 'paid' flag checked before paying; reaches deterministic references and the ambiguous-failure recovery with probing. Red flag: Fixes the incident by increasing the lock TTL to 6 hours, checks 'already paid?' with a read-then-write outside a transaction, or generates a new bank reference on every attempt.

SR-H-O01 · A "small" embedding cache that holds 234 MB

Hard · Output · Senior (5–8 years) · memory, numpy views, base arrays, tracemalloc, memory leaks, MLE · 8 min

Question: A recommender service keeps the last 100 embedding rows of each recent day in a dict "to save memory". The pod's RSS keeps climbing. What does this print? (tracemalloc tracks NumPy's data buffers; the numbers are rounded to whole MB.)

import tracemalloc
import numpy as np

tracemalloc.start()

def mb():
    return round(tracemalloc.get_traced_memory()[0] / 2**20)

def load_day(day):
    # one day of embeddings for a recommender: 20_000 x 512 float64 = ~78 MB
    return np.full((20_000, 512), float(day))

recent = {}
for day in range(1, 4):
    emb = load_day(day)
    recent[day] = emb[-100:]          # keep only the last 100 rows "to save memory"
    del emb
print("views:", mb(), "MB", recent[3].shape, recent[3].nbytes, recent[3].base is not None)

recent = {day: load_day(day)[-100:].copy() for day in range(1, 4)}
print("copies:", mb(), "MB", recent[3].nbytes, recent[3].base is None)

big = load_day(9)
col = big[:, 0]
print(col.flags["C_CONTIGUOUS"], col.strides, np.shares_memory(col, big))

Expected output:

views: 234 MB (100, 512) 409600 True
copies: 1 MB 409600 True
False (4096,) True

Explanation:

Basic slicing returns a view. emb[-100:] is a new ndarray header (shape, strides, offset) that points into the same data buffer as emb, and keeps a reference to it in .base. del emb only removes the name; the 78 MB buffer stays alive as long as any view of it is alive. Three days × 20,000 × 512 × 8 bytes ≈ 234 MB, even though each cached array reports nbytes == 409600 (400 KB). nbytes describes the view's logical size, not the memory it pins.

.copy() allocates a fresh, contiguous 400 KB buffer and owns it (base is None), so each 78 MB day is freed as soon as the temporary from load_day goes away. Three copies ≈ 1.2 MB, which rounds to 1.

The last line is the same story for a column: big[:, 0] is a strided view (strides == (4096,), one row of 512 × 8 bytes between elements), not contiguous, and it shares (and pins) the whole 78 MB array.

Production shape of this bug: caching slices of large arrays, DataFrame columns backed by a big block, or bytes slices via memoryview. Memory "leaks" with no Python-level reference cycle and no growing object count, so object counters and gc show nothing.

Takeaway: when a small slice must outlive a big array, .copy() it (or np.ascontiguousarray); check .base, not nbytes.

Follow-ups:

1. "How would you find this in a running service where you can't add prints?" Compare tracemalloc snapshots (snapshot.compare_to(old, "lineno")) to see the allocation site that keeps growing: here it points at np.full in load_day, not at the cache line, which is the clue that something else pins it. For RSS vs Python heap, use memray or py-spy dump plus a heap snapshot; a big gap between RSS and tracemalloc usually means native buffers or fragmentation.
2. "Which NumPy operations return views and which return copies?" Views: basic slicing, reshape when possible, .T, ravel when contiguous, view(). Copies: fancy (integer array) indexing, boolean masks, flatten(), most arithmetic. So emb[[-1, -2]] would have copied; use np.shares_memory in tests when it matters.
3. "Does the same thing happen with pandas?" Yes, in subtler ways: a column or a row slice can be backed by a larger block. Since pandas 2.x with Copy-on-Write semantics, mutation is safe, but memory retention is not solved: call .copy() on a small subset you keep long-term (for example a per-merchant cache), and measure with df.memory_usage(deep=True) plus tracemalloc.

Scoring: Strong: Gets both numbers, explains .base pinning vs nbytes, names .copy() as the fix, and describes how they'd diagnose it with snapshots in production. Acceptable: Knows slices are views and reaches ~234 MB with prompting; unsure about the strided column line. Red flag: Believes del emb frees the 78 MB, or trusts nbytes/sys.getsizeof as a measure of what an array keeps alive.

SR-H-O02 · Deprecating a function in an SDK that 40 teams import

Hard · Output · Senior (5–8 years) · library design, warnings, DeprecationWarning, stacklevel, warning filters, API evolution · 8 min

Question: You maintain the internal paysdk package. You added deprecation warnings to two functions a quarter ago, but no team has migrated, and several say they "never saw a warning". showwarning is replaced so warnings print to stdout. The interpreter runs with default warning filters (no -W flag, no PYTHONWARNINGS). What does this print?

import sys
import types
import warnings

def show(message, category, filename, lineno, file=None, line=None):
    print(f"  shown: {category.__name__}: {message}")

warnings.showwarning = show

def make_module(name, source):
    mod = types.ModuleType(name)
    exec(source, mod.__dict__)
    sys.modules[name] = mod
    return mod

paysdk = make_module("paysdk", '''
import warnings
def charge(amount, currency="INR"):
    warnings.warn("charge() is deprecated; use create_payment()", DeprecationWarning)
    return amount
def refund(amount):
    warnings.warn("refund() is deprecated; use create_refund()", DeprecationWarning, stacklevel=2)
    return amount
''')

orders = make_module("orders", '''
import paysdk
def checkout():
    return paysdk.refund(10)
''')

print("app calls charge")
paysdk.charge(100)
print("app calls refund twice")
for _ in range(2):
    paysdk.refund(50)
print("another team's module calls refund")
orders.checkout()
print("under pytest-like filters")
with warnings.catch_warnings():
    warnings.simplefilter("error", DeprecationWarning)
    try:
        orders.checkout()
    except DeprecationWarning as exc:
        print("  raised:", exc)

Expected output:

app calls charge
app calls refund twice
  shown: DeprecationWarning: refund() is deprecated; use create_refund()
another team's module calls refund
under pytest-like filters
  raised: refund() is deprecated; use create_refund()

Explanation:

Three rules decide what a user sees:

1. Default filters ignore DeprecationWarning unless it is attributed to __main__. Since Python 3.7 the default list is roughly default::DeprecationWarning:__main__, then ignore::DeprecationWarning. The audience is the developer running a script, not end users of an application.
2. stacklevel decides which module the warning is attributed to. charge uses the default stacklevel=1, so the warning points at paysdk itself, matches ignore, and nobody ever sees it. refund uses stacklevel=2, so the warning points at the caller. Called from __main__, it is shown; called from orders (another team's library code), it is attributed to orders and ignored again.
3. The default action shows a warning once per location (module + line). The loop calls refund twice from the same line, so it prints once.

Under simplefilter("error", DeprecationWarning) (what pytest -W error::DeprecationWarning or a strict CI does), the same call raises, which is how deprecations actually get noticed.

Takeaway: always pass stacklevel=2 (or more, through wrappers; 3.12 adds skip_file_prefixes), and don't rely on people seeing DeprecationWarning at runtime: make CI turn your package's deprecations into errors, and announce removals through the changelog and versioning policy.

Follow-ups:

1. "Teams still won't see it in production logs. What would you do as the SDK owner?" Make it visible where it matters: ship a pytest plugin or documented filterwarnings = error::DeprecationWarning:paysdk.* setting for consumer CI; route warnings to logs with logging.captureWarnings(True) and count them as a metric (calls to deprecated APIs per service); publish a removal version under semver; and use static signals too, typing_extensions.deprecated / warnings.deprecated (PEP 702, 3.13+) so type checkers and IDEs flag calls.
2. "When would you use FutureWarning or PendingDeprecationWarning instead?" FutureWarning is shown by default everywhere: use it for behaviour changes that end users of an application should see (pandas uses it for changing defaults). PendingDeprecationWarning is ignored even in __main__; it signals "will be deprecated later". DeprecationWarning targets developers of the calling code.
3. "Your deprecated function is called through a decorator or a thin wrapper. What happens to stacklevel=2?" The warning gets attributed to the wrapper's module (inside your package) and is ignored, or points users to the wrong line. Compute the stacklevel for your own wrapper depth, or in 3.12+ use warnings.warn(..., skip_file_prefixes=(os.path.dirname(paysdk.__file__),)) so the first frame outside the package gets the blame.

Scoring: Strong: Exact output; explains the __main__-only default filter, the role of stacklevel, once-per-location dedup, and proposes a CI-enforced deprecation policy. Acceptable: Knows DeprecationWarning is hidden by default and that stacklevel=2 matters, but misses that a call from another library module is also hidden. Red flag: Expects every warning to print every time, or thinks adding a warning is enough to deprecate an API used by many teams.

SR-H-O03 · Keeping the last exception keeps the whole DataFrame alive

Hard · Output · Senior (5–8 years) · memory, exceptions, tracebacks, frames, weakref, reference counting · 8 min

Question: A nightly scoring job keeps self.last_error so a health endpoint can report it. After a few failed days, the worker is OOM-killed. Frame stands in for a 2 GB DataFrame, and a weak reference tells us whether it is still alive. What does this print? (CPython; frees happen when the reference count drops to zero.)

import traceback
import weakref

class Frame:                       # stands in for a 2 GB DataFrame
    def __init__(self, day):
        self.day = day

class Job:
    def __init__(self):
        self.last_error = None

    def score_day(self, day):
        df = Frame(day)
        self.probe = weakref.ref(df)
        raise ValueError(f"bad rows on {day}")

    def run(self, day, keep):
        try:
            self.score_day(day)
        except ValueError as exc:
            if keep == "exception":
                self.last_error = exc
            elif keep == "message":
                self.last_error = str(exc)
            elif keep == "cleared":
                traceback.clear_frames(exc.__traceback__)
                self.last_error = exc
        return self.probe() is not None

job = Job()
for keep in ["nothing", "message", "exception", "cleared"]:
    job.last_error = None
    print(f"{keep:<9} alive={job.run('2026-10-04', keep)}")

try:
    job.score_day("2026-10-05")
except ValueError as exc:
    pass
print("exc" in dir(), job.probe() is None)

Expected output:

nothing   alive=False
message   alive=False
exception alive=True
cleared   alive=False
False True

Explanation:

The reference chain is: exception → __traceback__ → traceback entries → frame objects → the frames' local variables. score_day's frame holds df, so as long as the exception object is reachable, the "2 GB" object is too.

• nothing: when the except block ends, Python deletes exc (it compiles to finally: del exc). The exception's refcount reaches zero, the traceback and the score_day frame go with it, and so does df.
• message: str(exc) keeps only a string. Nothing points to the frame.
• exception: self.last_error = exc keeps the exception, its traceback and every frame in it, with all their locals. Each failed day pins another one: that's the OOM.
• cleared: traceback.clear_frames(tb) clears the locals of every finished frame in the traceback (it skips frames that are still executing), so df is released while the exception, type and message remain.

The last line shows the language rule: exc doesn't exist after the except block ("exc" in dir() is False), precisely to avoid this cycle/retention; and nothing else kept the 2026-10-05 frame alive.

The same leak appears with sys.exc_info() stored on an object, a Future holding an exception, retry helpers that collect "all errors", and error-reporting code that stores tracebacks in a list.

Takeaway: store what you need from an exception (type, message, formatted traceback text, IDs), not the exception object, in anything long-lived.

Follow-ups:

1. "The health endpoint needs the stack trace. How do you keep it without the frames?" Format it once: "".join(traceback.format_exception(exc)) (or traceback.TracebackException.from_exception(exc, capture_locals=False)), store the string, and let the exception go. Better still, log it with logger.exception(...) and send it to Sentry/your error tracker with an ID; the endpoint reports the ID.
2. "Why didn't the garbage collector's cycle detector save us in the exception case?" There is no unreachable cycle to collect: job is reachable, and it holds the exception. GC only frees unreachable objects. (A real cycle also exists in the nothing case if code does e = exc inside the frame that raised; that one is collectable but only when the cyclic GC runs, which is why Python deletes exc eagerly.)
3. "How would you confirm this is the leak in production?" Take two tracemalloc snapshots a day apart and compare by traceback: growth attributed to the DataFrame load site that never shrinks; then gc.get_referrers on one of the leaked objects (or objgraph.show_backrefs) shows the chain frame → traceback → exception → Job.last_error.

Scoring: Strong: All five lines; describes the exception → traceback → frame → locals chain, the implicit del exc, and what to store instead; mentions how they'd prove it with snapshots. Acceptable: Gets exception alive=True with reasoning about tracebacks, unsure about clear_frames or the last line. Red flag: Thinks an exception object is "just a message", or that GC will reclaim memory held by a reachable attribute.

SR-H-O04 · Loading a model artifact runs code before you see the model

Hard · Output · Senior (5–8 years) · security, pickle, deserialization, model serving, supply chain, allow-lists · 8 min

Question: A model-serving service downloads fraud-v7.pkl from a shared bucket that several teams can write to, and loads it with pickle.loads. A reviewer asks whether that's a security problem. What does this print?

import io
import json
import pickle

class Payload:
    def __reduce__(self):
        return (print, ("!! code ran during loads",))

blob = pickle.dumps({"model": "fraud-v7", "weights": Payload()})
print(len(blob) > 0, b"print" in blob)

print("loading...")
obj = pickle.loads(blob)
print("loaded:", obj)

class SafeUnpickler(pickle.Unpickler):
    ALLOWED = {("builtins", "dict"), ("builtins", "list")}
    def find_class(self, module, name):
        if (module, name) not in self.ALLOWED:
            raise pickle.UnpicklingError(f"blocked {module}.{name}")
        return super().find_class(module, name)

try:
    SafeUnpickler(io.BytesIO(blob)).load()
except pickle.UnpicklingError as exc:
    print("safe:", exc)

print(SafeUnpickler(io.BytesIO(pickle.dumps({"weights": [0.1, 0.2]}))).load())
print("json:", json.loads('{"weights": {"__reduce__": ["print", "pwned"]}}'))

Expected output:

True True
loading...
!! code ran during loads
loaded: {'model': 'fraud-v7', 'weights': None}
safe: blocked builtins.print
{'weights': [0.1, 0.2]}
json: {'weights': {'__reduce__': ['print', 'pwned']}}

Explanation:

A pickle is a small program for a stack machine, not a data format. __reduce__ lets an object say "to rebuild me, call this callable with these arguments". pickle.dumps records a reference to builtins.print (that's why b"print" is in the bytes) plus the argument tuple. pickle.loads imports the callable by name and calls it while loading; print returns None, which becomes the value of "weights".

An attacker who can write the file uses os.system, subprocess.Popen or eval instead of print: arbitrary code runs inside your serving process, with its credentials, at load time, before any validation of the "model". Nothing about the Payload class needs to exist on the loading side; the reference is to builtins.print.

SafeUnpickler.find_class is the hook every global lookup goes through. An allow-list (never a deny-list) of exactly the classes you expect blocks the payload with UnpicklingError, while plain containers still load.

json.loads only ever builds dicts, lists, strings, numbers, booleans and None; a key called "__reduce__" is just a string.

Takeaway: never unpickle data that crosses a trust boundary (shared buckets, caches, queues, user uploads). Use a data-only format (JSON, Parquet, safetensors, ONNX), or sign artifacts and verify the signature before loading.

Follow-ups:

1. "The team says "the bucket is internal, only our CI writes to it". Is that enough?" It narrows the risk but makes the bucket's write permission equivalent to code execution in production. Treat it like a deploy pipeline: least-privilege write access, artifact signing (HMAC with a key the serving side holds, or Sigstore-style signatures) verified before load, checksums pinned in the deploy manifest, and audit logs.
2. "Which common ML loaders have this problem?" Anything built on pickle: joblib.load, pickle, pandas.read_pickle, numpy.load(allow_pickle=True), and torch.load for full checkpoints (since PyTorch 2.6 weights_only=True is the default, which restricts the unpickler to tensors and primitive types). Prefer safetensors for weights and a versioned, data-only config for everything else.
3. "Is an allow-listing find_class a complete fix?" It's a strong mitigation if the list is tiny and contains only data classes. It gets fragile when you have to allow large libraries (an allowed class with a dangerous __setstate__ or __reduce__ path reopens the hole), and it doesn't protect against resource exhaustion. Use it as defence in depth, not instead of a data-only format.

Scoring: Strong: Exact output; explains __reduce__ and that loads calls importable callables; proposes data-only formats, signing before load, and least-privilege on the artifact store. Acceptable: Knows pickle is unsafe for untrusted input and gets most of the output, but can't explain the mechanism or the find_class hook. Red flag: Believes unpickling only creates objects of classes defined in the loading code, or that an "internal" source makes pickle safe without further controls.

SR-H-C01 · Single-flight: one pricing call per key during a cache stampede

Hard · Coding · Senior (5–8 years) · concurrency, threading, caching, cache stampede, request coalescing, error propagation · 25 min

Problem:

At 12:00 IST a flash sale starts and the product-price cache entry for the hero SKU expires. Within the same millisecond, 400 request threads miss the cache and all call the pricing service, which falls over. You will put a single-flight layer in front of the loader (the pattern behind Go's singleflight and Guava's LoadingCache):

Implement SingleFlight with:

• do(key, fn): if no call for key is in flight, run fn() in the calling thread and return its result. If a call for key is in flight, don't call fn; wait for the in-flight call and return the same result object, or raise the same exception it raised.
• Nothing is cached: once the in-flight call finishes (success or failure), the next do(key, ...) runs fn again. (Caching is the caller's job; this only collapses concurrent duplicates.)
• Calls for different keys must not block each other.
• pending(key): how many callers are currently attached to the in-flight call for key (the leader plus waiters), 0 if none. Useful for metrics ("coalesced requests") and for tests.

Signature: class SingleFlight: def do(self, key: Hashable, fn: Callable[[], T]) -> T: ... def pending(self, key: Hashable) -> int: ...

Constraints:

• Thread-safe under the standard (GIL) build and free-threaded builds: don't rely on the GIL for atomicity.
• fn may take seconds and may raise any exception; never hold a lock while calling it.
• Standard library only.

Examples:

• 5 threads call sf.do('A1', load_price) while the first is still running → load_price runs once; all 5 threads get the same dict object
• 3 threads call sf.do('C3', flaky) and flaky raises TimeoutError → all 3 threads raise TimeoutError; the next sf.do('C3', ...) calls fn again

Hints:

1. You need a per-key record of the in-flight call: something to wait on, plus a slot for the result or the exception.
2. A threading.Event per call lets followers wait without holding the global lock.
3. Hold the lock only to look up / insert / remove the per-key record, never while running fn.
4. Think about the failure path: what happens to the per-key record if fn raises? Use try/finally.
5. Order matters at completion: store the result, remove the key, then wake the followers.

Reference solution:

import threading
from collections.abc import Callable, Hashable
from typing import Any, TypeVar

T = TypeVar("T")


class _Call:
    __slots__ = ("done", "result", "error", "callers")

    def __init__(self) -> None:
        self.done = threading.Event()
        self.result: Any = None
        self.error: BaseException | None = None
        self.callers = 1


class SingleFlight:
    """Coalesce concurrent calls for the same key into one execution.

    Callers that arrive while a call for `key` is in flight wait for it and get the
    same result (or the same exception). Nothing is cached: once the call finishes,
    the next caller for that key starts a new execution.
    """

    def __init__(self) -> None:
        self._lock = threading.Lock()
        self._calls: dict[Hashable, _Call] = {}

    def do(self, key: Hashable, fn: Callable[[], T]) -> T:
        with self._lock:
            call = self._calls.get(key)
            leader = call is None
            if leader:
                call = self._calls[key] = _Call()
            else:
                call.callers += 1

        if not leader:
            call.done.wait()
            if call.error is not None:
                raise call.error
            return call.result

        try:
            call.result = fn()            # run outside the lock: other keys stay unblocked
        except BaseException as exc:
            call.error = exc
            raise
        finally:
            with self._lock:
                del self._calls[key]      # always clean up, so failures are never cached
            call.done.set()               # wake followers only after result/error is stored
        return call.result

    def pending(self, key: Hashable) -> int:
        """Number of callers attached to the in-flight call for `key` (0 if none)."""
        with self._lock:
            call = self._calls.get(key)
            return call.callers if call is not None else 0

Complexity: O(1) bookkeeping per call under one lock, plus one Event per in-flight key. Memory is O(number of keys currently in flight); nothing is retained after completion.

Tests:

import threading
import time


def wait_until(cond, timeout=5.0):
    deadline = time.monotonic() + timeout
    while not cond():
        assert time.monotonic() < deadline, "condition not reached"
        time.sleep(0.001)


sf = SingleFlight()

# 1. Five concurrent callers for one key share a single load.
loads = []
release = threading.Event()

def load_price():
    loads.append("A1")
    assert release.wait(5)
    return {"sku": "A1", "price_inr": 499}

results = []
threads = [threading.Thread(target=lambda: results.append(sf.do("A1", load_price))) for _ in range(5)]
for t in threads:
    t.start()
wait_until(lambda: sf.pending("A1") == 5)

# 2. A different key is not blocked while A1 is in flight.
assert sf.do("B2", lambda: "b2-price") == "b2-price"

release.set()
for t in threads:
    t.join(5)
assert loads == ["A1"]
assert len(results) == 5 and all(r is results[0] for r in results)
assert sf.pending("A1") == 0

# 3. Nothing is cached: the next call runs again.
assert sf.do("A1", lambda: "fresh") == "fresh"

# 4. An error reaches every waiter, and is not cached either.
attempts = []
gate = threading.Event()

def flaky():
    attempts.append(1)
    assert gate.wait(5)
    raise TimeoutError("pricing service timed out")

errors = []

def caller():
    try:
        sf.do("C3", flaky)
    except TimeoutError as exc:
        errors.append(str(exc))

threads = [threading.Thread(target=caller) for _ in range(3)]
for t in threads:
    t.start()
wait_until(lambda: sf.pending("C3") == 3)
gate.set()
for t in threads:
    t.join(5)
assert len(attempts) == 1
assert errors == ["pricing service timed out"] * 3
assert sf.do("C3", lambda: "recovered") == "recovered"
assert sf.pending("C3") == 0

A common wrong solution (the tests catch it):

import threading


class _Call:
    def __init__(self):
        self.done = threading.Event()
        self.result = None
        self.error = None
        self.callers = 1


class SingleFlight:
    def __init__(self):
        self._lock = threading.Lock()
        self._calls = {}

    def do(self, key, fn):
        with self._lock:
            call = self._calls.get(key)
            leader = call is None
            if leader:
                call = self._calls[key] = _Call()
            else:
                call.callers += 1
        if not leader:
            call.done.wait()
            if call.error is not None:
                raise call.error
            return call.result
        try:
            call.result = fn()
        except Exception as exc:
            call.error = exc
            call.done.set()
            raise
        with self._lock:
            del self._calls[key]
        call.done.set()
        return call.result

    def pending(self, key):
        with self._lock:
            call = self._calls.get(key)
            return call.callers if call else 0

Alternatives and tradeoffs:

• concurrent.futures.Future per key instead of _Call + Event: the leader calls set_result / set_exception, followers call future.result(). Same design, less code; it re-raises the same shared exception object too, so the traceback caveat below still applies.
• Cache-level stampede protection: serve stale while one caller refreshes (stale-while-revalidate), add jitter to TTLs so hot keys don't expire together, or refresh ahead of expiry. These reduce how often you need single-flight; in a cluster they matter more than in-process coalescing.
• Distributed single-flight: in-process coalescing collapses 400 threads to one call per pod. With 50 pods you still send 50 calls. A short Redis lock (SET key token NX PX 2000) with followers polling or reading stale data gets it to one call per cluster, at the cost of a network round trip and lock-expiry edge cases.
• asyncio version: same shape with asyncio.Future; add care for cancellation (a cancelled leader must not leave followers waiting forever: shield the shared work or hand leadership over).

Follow-ups:

1. "Followers re-raise the leader's exception object. What's the subtle problem with that?" Every raise of the same exception object appends the current frames to its __traceback__, so tracebacks from different threads get stitched together and grow (you can see this in the broken solution's failure output), and the object is shared mutable state across threads. Options: re-raise a new exception chained from it (raise PricingUnavailable(key) from call.error) so each thread gets its own exception object and the leader's original stays intact as __cause__.
2. "The leader thread is killed by a timeout in the web framework while fn is running. What happens?" Python threads can't be killed, so fn keeps running and eventually completes, waking followers. The real risk is a hung fn: followers wait forever. Give fn its own timeout (HTTP client timeouts), and give followers a bounded done.wait(timeout) that raises a clear error so request threads are not pinned indefinitely.
3. "Should errors be coalesced too, or should each follower retry on its own?" Coalesce them: if 400 callers each retry immediately after the leader fails, you recreate the stampede on a dependency that's already failing. Combine it with a short negative cache or a circuit breaker so the next wave also backs off.

Scoring: Strong: Correct leader/follower design with lock held only for bookkeeping, try/finally cleanup so errors are never cached, followers get the same result or exception, different keys stay independent; discusses cluster-wide stampedes and traceback sharing. Acceptable: Works for the success path and coalesces correctly, but needs a hint for the failure path (cleanup in finally) or holds the lock slightly too long. Red flag: Wraps the whole fn call in one global lock (serialising every key), or uses lru_cache and calls it stampede protection.

SR-H-C02 · Circuit breaker for a flaky UPI switch, with a single half-open trial

Hard · Coding · Senior (5–8 years) · resilience, circuit breaker, state machines, threading, injectable clocks, API design · 25 min

Problem:

Your payments service calls a bank's UPI switch. When the switch degrades, every request waits for a 5 s timeout, worker threads pile up, and the whole service goes down with it. Implement a circuit breaker:

• closed (normal): calls go through. Count consecutive failures; a success resets the count. When the count reaches failure_threshold, the circuit opens.
• open: calls fail fast with CircuitOpenError without calling the function, until reset_timeout seconds have passed since it opened.
• half-open (after the timeout): exactly one trial call is let through. While that trial is running, other callers get CircuitOpenError. If the trial succeeds, the circuit closes (failure count back to 0). If it fails, the circuit opens again with a fresh timer.
• Only exceptions in trip_on count as failures. Anything else (e.g. ValueError for an invalid VPA, which is a 4xx from your side) propagates unchanged and does not count; if it happens during the half-open trial, it proves the dependency answered, so the circuit closes.
• state returns "closed", "open" or "half_open" (report "half_open" once the timeout has passed, even before a trial starts).
• Time comes from an injectable clock (default time.monotonic), so tests don't sleep.

Signature: class CircuitBreaker: def __init__(self, failure_threshold: int, reset_timeout: float, *, trip_on: tuple[type[BaseException], ...] = (Exception,), clock: Callable[[], float] = time.monotonic) -> None: ... @property def state(self) -> str: ... def call(self, fn: Callable[..., T], *args: Any, **kwargs: Any) -> T: ...

Constraints:

• Many threads share one breaker. Don't hold a lock while calling fn.
• No sleeping or background threads; state transitions happen lazily inside call / state.
• Standard library only.

Examples:

• threshold=3: fail, fail, success, fail, fail → state == 'closed' (failures must be consecutive)
• third consecutive ConnectionError, then call(up) 10 s later (reset_timeout=30) → CircuitOpenError; `up` is not called
• 30 s after opening: a trial call is running and another thread calls → the second caller gets CircuitOpenError

Hints:

1. Write the state machine on paper first: three states, and which event moves you between them.
2. Open → half-open can be computed lazily from clock() - opened_at; no timer thread needed.
3. To allow exactly one trial you need a flag that is set and checked under the same lock acquisition.
4. Split call into before (decide/reserve), the call itself (no lock), and after (record outcome).
5. A call that started while closed may fail after the circuit has already opened. Should it count again?
6. Test the 'one trial only' rule without threads: have the trial function itself try a second call.

Reference solution:

import threading
import time
from collections.abc import Callable
from typing import Any, TypeVar

T = TypeVar("T")


class CircuitOpenError(Exception):
    """Raised without calling the dependency while the circuit is open."""


class CircuitBreaker:
    """Closed -> open after `failure_threshold` consecutive failures.

    Open -> half-open once `reset_timeout` seconds have passed; half-open lets exactly one
    trial call through (others are rejected) and its outcome closes or re-opens the circuit.
    Only exceptions in `trip_on` count as failures; anything else is the caller's problem
    (e.g. a 4xx validation error) and proves the dependency is up.
    """

    def __init__(self, failure_threshold: int, reset_timeout: float, *,
                 trip_on: tuple[type[BaseException], ...] = (Exception,),
                 clock: Callable[[], float] = time.monotonic) -> None:
        if failure_threshold < 1 or reset_timeout <= 0:
            raise ValueError("failure_threshold must be >= 1 and reset_timeout > 0")
        self._threshold = failure_threshold
        self._reset_timeout = reset_timeout
        self._trip_on = trip_on
        self._clock = clock
        self._lock = threading.Lock()
        self._state = "closed"
        self._failures = 0
        self._opened_at = 0.0
        self._trial_in_flight = False

    @property
    def state(self) -> str:
        with self._lock:
            if self._state == "open" and self._clock() - self._opened_at >= self._reset_timeout:
                return "half_open"
            return self._state

    def _before_call(self) -> bool:
        """Return True if this call is the half-open trial; raise if rejected."""
        with self._lock:
            if self._state == "closed":
                return False
            if self._state == "open":
                if self._clock() - self._opened_at < self._reset_timeout:
                    raise CircuitOpenError("circuit open")
                self._state = "half_open"
            if self._trial_in_flight:
                raise CircuitOpenError("circuit half-open: trial call in progress")
            self._trial_in_flight = True
            return True

    def _on_success(self, trial: bool) -> None:
        with self._lock:
            if trial:
                self._trial_in_flight = False
                self._state = "closed"
            if self._state == "closed":
                self._failures = 0

    def _on_failure(self, trial: bool) -> None:
        with self._lock:
            if trial:
                self._trial_in_flight = False
                self._trip()
                return
            if self._state != "closed":
                return                      # a slow call that started before the trip
            self._failures += 1
            if self._failures >= self._threshold:
                self._trip()

    def _trip(self) -> None:
        self._state = "open"
        self._opened_at = self._clock()
        self._failures = 0

    def call(self, fn: Callable[..., T], *args: Any, **kwargs: Any) -> T:
        trial = self._before_call()
        try:
            result = fn(*args, **kwargs)    # never hold the lock while calling out
        except self._trip_on:
            self._on_failure(trial)
            raise
        except BaseException:
            if trial:                       # e.g. ValueError from bad input: dependency answered
                self._on_success(trial)
            raise
        self._on_success(trial)
        return result

Complexity: O(1) time per call; O(1) memory. One short critical section before and one after each call.

Tests:

class FakeClock:
    def __init__(self):
        self.now = 1000.0
    def __call__(self):
        return self.now

clock = FakeClock()
cb = CircuitBreaker(failure_threshold=3, reset_timeout=30, trip_on=(ConnectionError, TimeoutError), clock=clock)
calls = []

def down():
    calls.append("down")
    raise ConnectionError("UPI switch unreachable")

def up(x=1):
    calls.append("up")
    return x

def expect(exc_type, fn, *args):
    try:
        cb.call(fn, *args)
    except exc_type:
        return
    raise AssertionError(f"expected {exc_type.__name__}")

# Consecutive failures only: a success in between resets the count.
expect(ConnectionError, down)
expect(ConnectionError, down)
assert cb.call(up, 7) == 7
expect(ConnectionError, down)
expect(ConnectionError, down)
assert cb.state == "closed"

# Exceptions outside trip_on (bad input) don't count.
def bad_request():
    raise ValueError("invalid VPA")
expect(ValueError, bad_request)
expect(ValueError, bad_request)
assert cb.state == "closed"

expect(ConnectionError, down)            # third consecutive failure -> open
assert cb.state == "open"

# Open: fail fast without calling the dependency.
n = len(calls)
expect(CircuitOpenError, up)
assert len(calls) == n

clock.now += 29.9
expect(CircuitOpenError, up)
clock.now += 0.1
assert cb.state == "half_open"

# Half-open: exactly one trial; a concurrent caller is rejected while it runs.
def trial_that_sees_a_second_caller():
    expect(CircuitOpenError, up)
    raise TimeoutError("still slow")
expect(TimeoutError, trial_that_sees_a_second_caller)
assert cb.state == "open"                 # failed trial re-opens with a fresh timer
clock.now += 10
expect(CircuitOpenError, up)

clock.now += 20
inner = []
def good_trial():
    expect(CircuitOpenError, up)
    inner.append("rejected")
    return "ok"
assert cb.call(good_trial) == "ok"
assert inner == ["rejected"]
assert cb.state == "closed"

# After closing, the failure count starts from zero again.
expect(ConnectionError, down)
expect(ConnectionError, down)
assert cb.state == "closed"
assert cb.call(up, 3) == 3

A common wrong solution (the tests catch it):

import time


class CircuitOpenError(Exception):
    pass


class CircuitBreaker:
    def __init__(self, failure_threshold, reset_timeout, *, trip_on=(Exception,), clock=time.monotonic):
        self._threshold = failure_threshold
        self._reset_timeout = reset_timeout
        self._trip_on = trip_on
        self._clock = clock
        self._failures = 0
        self._opened_at = None

    @property
    def state(self):
        if self._opened_at is None:
            return "closed"
        if self._clock() - self._opened_at >= self._reset_timeout:
            return "half_open"
        return "open"

    def call(self, fn, *args, **kwargs):
        if self.state == "open":
            raise CircuitOpenError("circuit open")
        try:
            result = fn(*args, **kwargs)
        except self._trip_on:
            self._failures += 1
            if self._failures >= self._threshold:
                self._opened_at = self._clock()
            raise
        self._failures = 0
        self._opened_at = None
        return result

Alternatives and tradeoffs:

• Failure-rate window instead of consecutive failures: open when more than 50% of the last N calls (or the last 10 seconds, with a minimum volume) failed. More robust when a dependency fails intermittently; this is what resilience4j and Envoy outlier detection do.
• Slow calls as failures: count calls over a latency threshold, because a dependency that answers in 4.9 s hurts as much as one that errors.
• Library: pybreaker or aiobreaker; in a service mesh, the sidecar (Envoy/Istio) can do it per upstream host. Then the in-process version only adds value for logic the mesh can't see (e.g. bank-specific error codes).
• asyncio version: same state machine; the half-open flag works the same, and asyncio.CancelledError during the trial should release the trial slot without counting as a failure.

Follow-ups:

1. "You run 40 pods. Should the breaker state be shared through Redis?" Usually not. Local breakers converge quickly because every pod sees the same failures, and they keep working when Redis is down. Shared state adds a network hop to every call and a new failure mode. Share only if traffic per pod is too low for a local breaker to ever see enough failures; then use a failure-rate window over shared counters.
2. "What should the API return to the client when the circuit is open?" Fail fast with a clear, retryable error (HTTP 503 with Retry-After) or degrade gracefully (queue the payout, show "bank is slow, we'll confirm by SMS"). Never pretend success. Emit a metric and a log line on every state transition so on-call sees the breaker open before customers complain.
3. "How do retries interact with the breaker?" Put the breaker outside the retry loop for user-facing paths (or make retries check the breaker) so that an open circuit stops retries immediately; and count one failure per logical call, not per retry attempt, or the breaker trips three times faster than you configured.

Scoring: Strong: Clean three-state machine with an injectable clock, single half-open trial enforced under a lock, no lock held during the call, correct handling of non-tripping exceptions; discusses failure-rate windows, metrics on transitions and the retry interaction. Acceptable: Closed/open/half-open work in the simple case but the half-open phase lets every caller through, or the failure count isn't reset properly; fixes it when the test shows it. Red flag: Uses time.sleep or a background thread to "close" the circuit, holds a lock across the remote call, or counts every exception (including bad input) as a dependency failure.

SR-H-C03 · CI gate: is this event-schema change safe for existing consumers?

Hard · Coding · Senior (5–8 years) · data contracts, schema evolution, backward compatibility, recursion, CI checks · 25 min

Problem:

The orders team publishes order_placed events consumed by 12 downstream jobs (finance, fraud, the data lake). Schemas live in the repo as dicts. You're adding a CI check that blocks a pull request if the new schema can break consumers built against the old one.

Rule: every record the producer can emit under new must also be valid under old. Consumers ignore unknown fields, so adding a field is never breaking.

A schema maps field name → spec: {"type": "int" | "float" | "str" | "bool" | "object", "required": bool (default True), "nullable": bool (default False), "enum": [...] (optional), "fields": {...} (only for "object")}.

Return a sorted list of messages, with nested paths joined by dots:

Type compatibility: an int is a valid float (JSON consumers accept 5 where 5.0 is expected), so float → int is safe and int → float is breaking. bool is not an int, whatever Python's isinstance says. Removing an optional field is safe. Narrowing (optional → required, shrinking an enum, float → int) is safe.

Signature: def breaking_changes(old: Schema, new: Schema) -> list[str]:

Constraints:

• Schemas are well-formed (validate them separately); nesting depth ≤ 10; up to a few hundred fields.
• Pure function: don't mutate the inputs.

Examples:

• old qty int, new qty float → ['qty: type int -> float']
• old amount float, new amount int → []
• new method enum adds 'UPI_LITE' → ["method: enum values added ['UPI_LITE']"]
• address.pincode removed (required) → ['address.pincode: removed']

Hints:

1. Walk the fields of old: a field that exists only in new can never break an old consumer.
2. Write the type rule as data: for each new type, which old types accept all its values?
3. Recurse into object fields with a path prefix, but only when the type check passed.
4. Collect messages and sort once at the top level.

Reference solution:

from typing import Any

Field = dict[str, Any]
Schema = dict[str, Field]

# new type -> old types that accept every value of the new type
_ACCEPTED_BY = {
    "int": {"int", "float"},      # every int is a valid float for a JSON consumer
    "float": {"float"},
    "str": {"str"},
    "bool": {"bool"},             # bool is NOT an int here, even though Python says it is
    "object": {"object"},
}


def breaking_changes(old: Schema, new: Schema) -> list[str]:
    """List the ways records valid under `new` can break consumers built for `old`.

    The rule: every record the producer can emit under `new` must also be valid under
    `old`. Consumers ignore unknown fields, so adding a field is never breaking.
    Returns the messages sorted.
    """
    return sorted(_check(old, new, ""))


def _check(old: Schema, new: Schema, prefix: str) -> list[str]:
    problems: list[str] = []
    for name in sorted(old):
        path = f"{prefix}{name}"
        o = old[name]
        n = new.get(name)
        o_required = o.get("required", True)
        if n is None:
            if o_required:
                problems.append(f"{path}: removed")
            continue
        if o_required and not n.get("required", True):
            problems.append(f"{path}: became optional")
        if n.get("nullable", False) and not o.get("nullable", False):
            problems.append(f"{path}: became nullable")
        if o["type"] not in _ACCEPTED_BY[n["type"]]:
            problems.append(f"{path}: type {o['type']} -> {n['type']}")
            continue                  # nested/enum checks are meaningless after a type change
        if "enum" in o:
            if "enum" not in n:
                problems.append(f"{path}: enum constraint dropped")
            else:
                added = sorted(set(n["enum"]) - set(o["enum"]))
                if added:
                    problems.append(f"{path}: enum values added {added}")
        if o["type"] == "object":
            problems += _check(o["fields"], n["fields"], f"{path}.")
    return problems

Complexity: O(F log F) for F fields in total across all levels (one visit per field, plus sorting the messages).

Tests:

v1 = {
    "order_id": {"type": "str"},
    "amount": {"type": "float"},
    "qty": {"type": "int"},
    "coupon": {"type": "str", "required": False},
    "method": {"type": "str", "enum": ["UPI", "CARD", "NETBANKING"]},
    "address": {"type": "object", "fields": {
        "pincode": {"type": "str"},
        "landmark": {"type": "str", "required": False},
    }},
}

# Identical schema, and purely additive changes, are compatible.
assert breaking_changes(v1, v1) == []
added = dict(v1, gst_rate={"type": "float"}, channel={"type": "str", "required": False})
assert breaking_changes(v1, added) == []

# Narrowing is safe: float -> int, optional -> required, enum shrinks.
narrowed = dict(v1, amount={"type": "int"}, coupon={"type": "str"},
                method={"type": "str", "enum": ["UPI", "CARD"]})
assert breaking_changes(v1, narrowed) == []

# Dropping an OPTIONAL field is safe (top level and nested); consumers already handle its absence.
no_coupon = {k: v for k, v in v1.items() if k != "coupon"}
no_coupon["address"] = {"type": "object", "fields": {"pincode": {"type": "str"}}}
assert breaking_changes(v1, no_coupon) == []

# Widening and removal break consumers.
v2 = {
    "order_id": {"type": "str", "required": False},
    "amount": {"type": "float", "nullable": True},
    "qty": {"type": "float"},
    "method": {"type": "str", "enum": ["UPI", "CARD", "NETBANKING", "UPI_LITE"]},
    "address": {"type": "object", "fields": {
        "pincode": {"type": "int"},
    }},
}
assert breaking_changes(v1, v2) == [
    "address.pincode: type str -> int",
    "amount: became nullable",
    "method: enum values added ['UPI_LITE']",
    "order_id: became optional",
    "qty: type int -> float",
], breaking_changes(v1, v2)

# Required field removed, enum dropped, bool is not an int, nested removal.
v3 = {
    "amount": {"type": "float"},
    "qty": {"type": "bool"},
    "method": {"type": "str"},
    "coupon": {"type": "str", "required": False, "nullable": True},
    "address": {"type": "object", "fields": {"landmark": {"type": "str"}}},
}
assert breaking_changes(v1, v3) == [
    "address.pincode: removed",
    "coupon: became nullable",
    "method: enum constraint dropped",
    "order_id: removed",
    "qty: type int -> bool",
], breaking_changes(v1, v3)

# A type change on an object is reported once, without recursing into it.
assert breaking_changes(v1, dict(v1, address={"type": "str"})) == ["address: type object -> str"]

A common wrong solution (the tests catch it):

def breaking_changes(old, new, _prefix=""):
    problems = []
    for name in sorted(old):
        path = f"{_prefix}{name}"
        o, n = old[name], new.get(name)
        if n is None:
            problems.append(f"{path}: removed")
            continue
        if o.get("required", True) and not n.get("required", True):
            problems.append(f"{path}: became optional")
        if n.get("nullable", False) and not o.get("nullable", False):
            problems.append(f"{path}: became nullable")
        if o["type"] != n["type"]:
            problems.append(f"{path}: type {o['type']} -> {n['type']}")
            continue
        if "enum" in o and "enum" in n:
            added = sorted(set(n["enum"]) - set(o["enum"]))
            if added:
                problems.append(f"{path}: enum values added {added}")
        if o["type"] == "object":
            problems += breaking_changes(o["fields"], n["fields"], f"{path}.")
    return problems

Alternatives and tradeoffs:

• Use the real tooling: Avro / Protobuf schema registries (Confluent, Buf) implement BACKWARD / FORWARD / FULL compatibility modes, and JSON Schema diff tools exist. Writing this yourself is reasonable only for an in-house dict/Pydantic-based contract; the important part is choosing the direction of compatibility.
• Generate the schema from Pydantic models (Model.model_json_schema()) and diff those, so the contract can't drift from the code that produces events.
• Contract tests with consumer examples (consumer-driven contracts, Pact-style): each consumer commits example payloads it relies on and CI validates them against the new producer schema. Catches semantic changes the structural diff misses (e.g. amount switching from rupees to paise).

Follow-ups:

1. "This checks one direction. When do you need the other one too?" If consumers can be upgraded before producers (or both read historical data from the lake), old records must also be readable by the new schema: that's the reverse check, breaking_changes(new, old). "Full" compatibility requires both, which effectively allows only adding/removing optional fields. Pick per topic and write it down in the contract.
2. "The check passes, but finance's job broke because amount changed from rupees to paise. What now?" Structure is not semantics. Units, currency, time zone and meaning must be part of the contract (field names like amount_paise, descriptions, or a unit annotation), changes to meaning need a new field or a new event version, and consumer-driven contract tests or reconciliation checks (totals vs the ledger) catch what a schema diff can't.
3. "How would you roll out a breaking change that is genuinely needed?" Expand/contract: publish a new field (or order_placed.v2 topic) alongside the old one, migrate consumers one by one with a tracked list and a deadline, monitor who still reads the old field, then stop producing it. The CI gate should allow an explicit, reviewed override that links to that migration plan.

Scoring: Strong: Defines compatibility as "new values ⊆ old values" before coding, gets int/float/bool and optional-removal right, recurses with paths, and discusses direction (backward vs forward), semantics and rollout. Acceptable: Handles removal, nullability and enums but treats every type change as breaking, or flags harmless optional-field removal; corrects it after discussing the rule. Red flag: Treats "any change" as breaking or "only removals" as breaking without reasoning about which records consumers will see; can't explain why adding an enum value can break a consumer.

SR-H-D01 · Serving a fraud model at checkout, and shipping v8 without breaking payments

Hard · Design · Senior (5–8 years) · model serving, FastAPI, latency budgets, feature stores, shadow and canary releases, fallbacks, observability · 40 min

Scenario:

A payments company scores every UPI and card transaction with a fraud model before authorising it. Today the model (a gradient-boosted tree, ~40 MB, scikit-learn/LightGBM) runs inside the Django monolith: it's loaded at import time in every Gunicorn worker, features are computed with four SQL queries per request, and a deploy of the monolith is the only way to ship a new model.

You own building a separate scoring service in Python. Requirements:

• Peak 3,000 requests/s (festive sales), normal ~600/s.
• The checkout path gives fraud scoring a p99 budget of 50 ms end to end, including feature lookup.
• Data science wants to ship fraud-v8 next month, and new models roughly every two weeks after that.
• If scoring is unavailable, payments must not stop.

Clarifying questions:

• What does the caller need back: a score, a decision (allow / step-up OTP / block), reasons? (Score plus decision plus top reason codes; the decision thresholds belong to the risk team and change more often than the model.)
• Which features are needed, and how fresh? (About 60 features: ~45 are aggregates such as "transactions by this VPA in the last 24 h", 15 come from the request itself. Aggregates may be a few seconds stale.)
• How is the model's inference cost? (Measured: ~1.5 ms per row single-threaded, ~4 ms for a batch of 32.)
• What's the fallback policy when scoring times out: allow, allow under an amount, or rules-only? (Risk team: a simple rules engine plus allow under ₹2,000; above that, step-up authentication.)
• Are there audit or regulatory needs? (Yes: every decision must be explainable later with the exact model version and the feature values used.)

Reference design:

Shape of the service. - A stateless FastAPI service, POST /score, with a Pydantic v2 request model (strict types, amounts in paise, explicit schema_version). Run several Uvicorn worker processes per pod (one per vCPU), because inference is CPU-bound: inside the async def endpoint, run the model call through a small thread pool only if the library releases the GIL (LightGBM's predict does), otherwise call it in a sync def endpoint or use processes. Measure rather than assume. - Load the model at startup (lifespan handler), not at import and not per request; fail readiness until the model and its feature schema are loaded and a smoke prediction succeeds.

Latency budget (p99 50 ms): network + parsing ~5 ms, feature lookup ≤ 15 ms, inference ≤ 10 ms, slack ~20 ms. - Features: the four SQL queries go. A streaming job (Kafka/Flink or a Python consumer) maintains per-VPA and per-device aggregates in Redis (or a managed online feature store). The service does one pipelined MGET/hash read with a 15 ms timeout. Missing features are explicit (None plus an "is_missing" flag the model was trained with), never silently zero. - The same feature code for training and serving: feature definitions live in one versioned Python package used by both the offline pipeline and the online service, to avoid train/serve skew. - Inference: a single-row tree prediction is cheap; micro-batching (collect for ≤ 2 ms or 32 rows) only pays off for heavier models or GPUs. Start without it and add it when the profile says so.

Model lifecycle. - Models are artifacts in a registry (MLflow or a versioned bucket) with metadata: version, feature list and schema, training data window, offline metrics, checksum/signature. The service loads by version pin from config, verifies the checksum, and never unpickles untrusted files (prefer a native/ONNX format). - Shadow first: v8 scores every request asynchronously (off the request path, sampled if needed) and both scores are logged; compare distributions, latency and the decision-change rate by segment (amount band, merchant category) for a week. - Canary: route 1% → 10% → 50% of decisions to v8 by a stable hash of the customer ID (so a user doesn't flip between models), with automatic rollback on latency or decision-rate guardrails. - Thresholds are config owned by the risk team, versioned separately from the model, and logged with each decision.

Failure handling. The caller (payments) enforces a hard 50 ms timeout and applies the fallback policy; the scoring service also degrades internally (feature timeout → score with missing-feature flags; model error → rules engine). A circuit breaker in the caller stops waiting on a dead service.

Observability and audit. Per-request structured log or event: request ID, model version, threshold version, feature vector hash (full features to an audit topic), score, decision, latency breakdown. Metrics: p50/p99 per stage, fallback rate, missing-feature rate per feature, score distribution per model version (drift alarms).

Trade-offs:

• In-process model in the monolith vs separate service: in-process has no network hop, but couples model releases to monolith deploys and loads the model into every web worker. A service adds ~2–5 ms and an operational surface, and buys independent releases, scaling and observability.
• Online feature store vs computing features on request: on-request is fresh and simple but blows the latency budget and loads the OLTP database. Precomputed aggregates are fast but slightly stale and need a streaming pipeline to own and monitor.
• Python service vs a dedicated inference server (Triton, BentoML, KServe): Python/FastAPI is flexible and matches the team's skills; a dedicated server gives batching, multi-model hosting and GPU management. For tree models on CPU, Python is enough.
• Fail-open vs fail-closed: failing open protects revenue and user experience; failing closed protects against fraud. The amount-based hybrid is a business decision; the engineer's job is to make it explicit, configurable and measured.
• Shadow cost: running two models doubles inference CPU during the shadow period; sample if needed.

Deep-dive probes:

• "p99 is 140 ms at 3k RPS but p50 is 9 ms. Walk me through finding the cause." Look for: per-stage timing histograms; checking the Redis client's pool size and timeouts; event-loop lag (a CPU-bound predict inside async def); GC pauses; uneven load across workers; py-spy on a hot pod; tail amplification from fan-out reads.
• "v8 offline AUC is better, but in canary the block rate for small merchants jumped 3×." Look for: segment-level guardrails, train/serve skew checks (log online features and re-score offline), a missing-feature default that differs between training and serving, and rollback before root cause.
• "How do you load a new model version without restarting pods?" Look for: load and validate the new model in the background, then swap a single reference atomically; keep the old one until in-flight requests finish; or simply roll pods (simpler, often better) with readiness gates.
• "Six months later, a customer disputes a block. What can you show?" Look for: an immutable audit record with model version, threshold version and feature values, plus the ability to re-score that record with the archived model.

Rubric: Strong: Builds an explicit latency budget and designs to it (precomputed features, one round trip, startup loading, CPU-aware workers); separates model, thresholds and fallback policy; describes shadow → canary with segment guardrails and rollback; covers train/serve skew, audit and per-stage observability. Acceptable: Proposes a reasonable FastAPI service with startup model loading and a cache for features, and some rollout plan, but leaves the latency budget, fallback policy or skew handling vague until prompted. Red flag: Keeps the four SQL queries on the request path, loads the model per request, ships v8 with a straight cutover, or has no answer for "the scoring service is down".

SR-H-D02 · Sending 2 crore sale notifications in 30 minutes without double-sending or delaying OTPs

Hard · Design · Senior (5–8 years) · job queues, rate limiting, idempotency, priorities, retries, backpressure, observability, compliance · 40 min

Scenario:

An e-commerce company runs a "Big Diwali Sale" that starts at 8:00 PM IST. Marketing wants 2 crore (20 million) push notifications and 50 lakh SMS delivered between 8:00 and 8:30 PM. The same notification platform also sends OTPs and order updates all day, and OTP delivery must stay under 5 s even during the blast.

Today a cron job loops over users in Python and calls the SMS and push providers synchronously. Last year it took 3 hours, some users got the same SMS four times, OTPs were delayed by 10 minutes and the SMS provider throttled the account. You own the redesign. The team is Python (Celery or similar is available, plus Postgres, Redis and Kafka).

Clarifying questions:

• What are the provider limits? (SMS gateway: 1,000 messages/s per account across two providers; push (FCM/APNs): effectively ~10k/s per sender with batching up to 500 per request.)
• Is the audience fixed at 7:00 PM, or can it change during the send? (Segment frozen at 7:30 PM; opt-outs must be honoured until the moment of sending.)
• What are the compliance rules? (TRAI: promotional SMS only to non-DND numbers, during permitted hours, with the registered template; opt-outs honoured immediately.)
• What does "delivered" mean for marketing: accepted by the provider, or confirmed on the device via delivery receipts? (They report on provider DLRs.)
• Can a duplicate ever be acceptable? (For marketing, rare duplicates are tolerable but not 4×; for OTPs a duplicate is fine, a delay is not.)

Reference design:

Separate the paths. OTPs/transactional messages and campaigns never share a queue or worker pool. Transactional traffic gets its own high-priority queue, dedicated workers and a reserved slice of each provider's rate limit (e.g. 200 of the 1,000 SMS/s). Campaign traffic can only use what's left.

Campaign pipeline. 1. Plan (7:30 PM): freeze the audience into a campaign_recipients table (campaign_id, user_id, channel, status), filtering DND, opt-outs and invalid tokens up front. Split into shards of ~1,000 recipients; one job per shard, enqueued onto the campaign queue (Celery/RQ/Arq on Redis, or Kafka partitions). 2. Send (8:00 PM): shard workers load their recipients, re-check opt-out at send time (cheap set lookup), render templates, and call providers in batches (push: 500 per FCM call) through an async HTTP client (httpx with asyncio) since this is I/O-bound: one process per core, each with hundreds of concurrent requests bounded by a semaphore. 3. Rate limiting: a distributed token bucket in Redis per provider and channel (atomic Lua script), shared by all workers; workers acquire tokens in chunks to reduce Redis round trips. Respect 429 / throttling responses with backoff, and feed them back into the limiter. 4. Idempotency (the 4× SMS bug): each message has a deterministic ID (campaign_id, user_id, channel). Before sending, the worker claims it with a state transition pending → sending (UPDATE ... WHERE status='pending'); after the provider accepts it, sending → sent with the provider message ID. Retries of a shard skip sent rows. Where the provider supports a client reference / idempotency key, pass the message ID so the provider dedupes too. A crash between the provider call and the sent write can still cause one duplicate: that's at-least-once, made rare and bounded rather than impossible. 5. Retries: retry only retryable failures (timeouts, 5xx, 429) with jittered exponential backoff and a cap; permanent failures (invalid number, unregistered token) mark the recipient and clean the token table. After the campaign window, stop: a sale notification at 11 PM is worse than none.

Throughput math. 50 lakh SMS at ~800/s available ≈ 104 minutes: the SMS goal is impossible with these limits. Say so early: negotiate a temporary limit increase with both providers, prioritise SMS to users without push, or start SMS earlier. Push: 2 crore at ~10k/s ≈ 33 minutes, feasible with batching. The engineer should do this arithmetic before designing workers.

Observability: per campaign and channel, metrics for queued / sending / sent / failed counts, send rate vs the limiter's rate, provider latency and error codes, OTP end-to-end latency (the SLO that matters most), and a live dashboard with a kill switch to pause a campaign instantly (workers check a Redis flag per batch).

Trade-offs:

• Celery/RQ jobs vs Kafka partitions: a task queue is simple and gives retries and visibility per job; Kafka gives ordered, replayable partitions and higher throughput but you build retry/dead-letter handling. For 20k shard jobs, a task queue is enough.
• Pre-render vs render at send time: pre-rendering moves work out of the peak but stores 2.5 crore payloads; rendering at send time is cheap with simple templates and allows last-minute fixes.
• Exactly-once is not available across an external SMS provider. The design aims for at-least-once with a claim step that makes duplicates rare; the alternative (send at most once, mark before sending) loses messages on crash. For marketing, prefer rare duplicates; state it explicitly.
• Threads vs asyncio for sending: both work for I/O; asyncio scales to more concurrent requests per process with less memory, threads are simpler with blocking SDKs. Avoid mixing a blocking provider SDK into an event loop.

Deep-dive probes:

• "At 8:05 PM OTP latency climbs to 40 s. What happened and how do you stop it now?" Look for: shared rate limit or shared workers between paths, provider throttling the whole account; immediate action: pause the campaign via the kill switch, then confirm OTP latency recovers; long-term: reserved capacity and separate sender IDs/accounts.
• "Half the workers crash mid-campaign. What state is everything in, and how do you resume?" Look for: shard jobs are idempotent, sending rows older than N minutes are reconciled against provider DLRs/message IDs before retrying, and resume re-enqueues only unfinished shards.
• "How do you test this before Diwali?" Look for: a load test against a provider sandbox or a fake provider that enforces the same limits and injects 429s and timeouts; a dry run to an internal segment; chaos tests that kill workers; and asserting no duplicate message IDs.
• "A user opted out at 8:10 PM but got an SMS at 8:12 PM." Look for: opt-out check at send time from a fast store, propagation delay from the preferences service, and a measurable bound on that delay.

Rubric: Strong: Does the throughput arithmetic and challenges the SMS target; isolates transactional traffic with reserved capacity; designs a shared token bucket, shard jobs and a claim-based idempotency scheme with an honest at-least-once statement; covers retries, compliance, kill switch and OTP SLO monitoring. Acceptable: Moves to a queue with workers and some rate limiting and dedupe, but shares capacity with OTPs or never checks whether the provider limits make the goal feasible. Red flag: "Use more threads" or "use asyncio" as the whole answer; no rate limiting or idempotency; claims exactly-once delivery through an SMS provider.

SR-HT-O01 · An early return from async for leaves the DB connection checked out

Hard & Tricky · Output · Senior (5–8 years) · asyncio, async generators, resource cleanup, aclosing, connection pools, CPython-specific · 10 min

Question: A FastAPI service streams rows from Postgres through an async generator. Under load, the pool reports connections "in use" that no request is using, and occasionally runs dry. The pool is simulated with a counter. What does this print? (CPython 3.12, standard asyncio event loop.)

import asyncio
from contextlib import aclosing

pool = {"in_use": 0}

async def stream_rows(query):
    pool["in_use"] += 1
    print(f"acquire {query}")
    try:
        for i in range(1, 100):
            await asyncio.sleep(0)      # fetch the next row from the DB
            yield i
    finally:
        pool["in_use"] -= 1
        print(f"release {query}")

def stream_rows_sync(query):
    pool["in_use"] += 1
    print(f"acquire {query}")
    try:
        yield from range(1, 100)
    finally:
        pool["in_use"] -= 1
        print(f"release {query}")

def first_match_sync(query, target):
    for row in stream_rows_sync(query):
        if row == target:
            return row

async def first_match(query, target):
    async for row in stream_rows(query):
        if row == target:
            return row

async def first_match_closing(query, target):
    async with aclosing(stream_rows(query)) as rows:
        async for row in rows:
            if row == target:
                return row

async def main():
    print("found", first_match_sync("q1", 3), "in_use", pool["in_use"])
    print("found", await first_match("q2", 3), "in_use", pool["in_use"])
    await asyncio.sleep(0.01)
    print("after sleep, in_use", pool["in_use"])
    print("found", await first_match_closing("q3", 3), "in_use", pool["in_use"])

asyncio.run(main())

Expected output:

acquire q1
release q1
found 3 in_use 0
acquire q2
found 3 in_use 1
release q2
after sleep, in_use 0
acquire q3
release q3
found 3 in_use 0

The trap: release q2 printed right after acquire q2, before found 3 in_use 0, "just like the sync version, because returning out of the loop closes the generator". That's true for sync generators and false for async ones. The other common miss is the opposite: assuming q2 is never released at all.

Explanation:

Sync generator (q1). When first_match_sync returns, the last reference to the generator disappears; CPython frees it immediately, and freeing a suspended generator calls close(), which raises GeneratorExit at the yield and runs finally synchronously. Release happens before the print arguments are evaluated.

Async generator (q2). Its finally may need to await (a real driver awaits conn.release()), and that can't happen inside a deallocator. So when an unfinished async generator is garbage-collected, CPython calls the event loop's finalizer hook (sys.set_asyncgen_hooks), and asyncio's hook schedules agen.aclose() as a new task for a later loop iteration. first_match returns with the connection still checked out (in_use 1); the release runs only once the loop gets control back, during the sleep.

Under load that "later" stretches: every early exit (a return, a break, an exception, a client disconnect) leaves a connection held until the loop gets round to the cleanup task, and if anything else still references the generator (a traceback, a list, a debugger) it isn't finalized at all until that reference goes. On a busy pool this shows up as phantom in-use connections and pool exhaustion.

contextlib.aclosing (q3) (3.10+) awaits aclose() when the async with block exits, so cleanup is deterministic and happens before first_match_closing returns.

Takeaway: whenever an async generator holds a resource, consume it under async with aclosing(...), or make the resource's owner a real async context manager around the loop.

Follow-ups:

1. "What happens if the event loop closes before the scheduled aclose() runs?" asyncio.run calls loop.shutdown_asyncgens() before closing, which aclose()s every async generator the loop still tracks, so cleanup happens at shutdown at the latest. With a hand-managed loop that skips this, you get "Task was destroyed but it is pending" or an async generator that is never finalized, and the finally block never runs.
2. "How does this interact with FastAPI's StreamingResponse when the client disconnects?" The response stops iterating your generator and the generator is closed through the same paths (explicit aclose by the server, or finalization). Put resource ownership in a dependency with yield or an async with inside the generator around the whole stream, and test the disconnect path explicitly.
3. "Is the sync behaviour (q1) guaranteed by the language?" No: immediate finalization comes from CPython's reference counting. On PyPy a generator abandoned mid-way is closed only when the GC runs. Portable code uses contextlib.closing(gen) or a with block for sync generators that hold resources, too.

Scoring: Strong: Exact order; contrasts refcount-driven close() for sync generators with the finalizer hook scheduling aclose() as a task; prescribes aclosing; links it to pool exhaustion. Acceptable: Knows the async case is different and that aclosing fixes it, but misplaces release q2. Red flag: Assumes finally in a generator always runs immediately on return/break, or proposes a bigger pool as the fix.

SR-HT-O02 · One task's Decimal precision leaks into GST for every request

Hard & Tricky · Output · Senior (5–8 years) · decimal, contextvars, asyncio, money, shared mutable state · 10 min

Question: A pricing service computes GST and FX quotes with Decimal inside asyncio tasks. After a release that added an FX endpoint, a few invoices show amounts like 1.456E+4. The FX author says "I only changed the precision in my own task, and each task gets its own copy of the context". What does this print?

import asyncio
import decimal
from decimal import Decimal

decimal.getcontext()      # app startup touched the context (e.g. a config parser)

async def fx_quote(amount):
    ctx = decimal.getcontext()
    ctx.prec = 4                      # "only this task needs 4 significant digits"
    ctx.rounding = decimal.ROUND_DOWN
    await asyncio.sleep(0)
    return amount * Decimal("83.4567")

async def gst(amount):
    await asyncio.sleep(0)
    return amount * Decimal("1.18")

async def fx_quote_local(amount):
    with decimal.localcontext(prec=4, rounding=decimal.ROUND_DOWN):
        await asyncio.sleep(0)
        return amount * Decimal("83.4567")

async def main():
    print(await asyncio.gather(gst(Decimal("12345.67")), fx_quote_local(Decimal("10"))))
    print(await asyncio.gather(gst(Decimal("12345.67")), fx_quote(Decimal("10"))))
    print(Decimal("12345.67") * Decimal("1.18"), decimal.getcontext().prec)

asyncio.run(main())

Expected output:

[Decimal('14567.8906'), Decimal('834.5')]
[Decimal('1.456E+4'), Decimal('834.5')]
1.456E+4 4

The trap: Line 2 as [Decimal('14567.8906'), Decimal('834.5')] and line 3 as 14567.8906 28, on the reasoning that every task runs in a copy of the context, so changing precision in fx_quote stays inside fx_quote. The copy is real, but it is a shallow copy.

Explanation:

Since Python 3.7 the decimal context lives in a ContextVar, and each asyncio task runs in a copy of the contextvars context taken when the task is created. Two details make that copy useless here:

1. The copy is shallow. It copies the mapping of variables to values, not the values. If the decimal context object already exists in the parent (any Decimal operation creates it, even building Decimal("12345.67") as an argument; the startup line just makes it explicit), every task's copy points at the same mutable decimal.Context object.
2. ctx.prec = 4 mutates that shared object; it doesn't .set() a new value in the task's copy. So gst, which runs after fx_quote mutated it (the sleep(0) lets that happen), multiplies at 4 significant digits with ROUND_DOWN: 14567.8906 becomes 1.456E+4. The change also leaks back into main and every later request.

decimal.localcontext(...) creates a new context and sets it in the current task's context var for the duration of the with block, then restores the previous one. That's isolated per task, so the first gather is correct.

The same trap applies to any ContextVar holding a mutable object (a dict of request tags, a list of spans): mutate it in a child task and the parent and siblings see it; var.set(new_obj) stays local.

Takeaway: never mutate the result of decimal.getcontext() in request code; use localcontext(), or pass an explicit context (ctx.multiply(a, b), amount.quantize(Decimal("0.01"), rounding=...)). For money, quantize to paise explicitly rather than relying on precision.

Follow-ups:

1. "Why does the 28-digit default not matter for typical rupee amounts, and what should define rounding instead?" prec is significant digits for intermediate results, and 28 digits is far more than any invoice needs. What matters is the final rounding rule: quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) (or what the tax rules say) at defined points, such as per line item vs per invoice. That's a business rule to make explicit in code and tests, not a global setting.
2. "Would the same bug happen with threads instead of asyncio tasks?" No, not in this form. A new thread starts with an empty contextvars context, so its first getcontext() creates a fresh default decimal.Context for that thread, and prec = 4 there affects only that thread. (Thread pools reuse threads, so the change would still leak into later jobs on the same worker thread.) Tasks are the surprising case because they inherit the parent's context object.
3. "How would you catch this class of bug in CI?" A test that runs the FX endpoint and the GST computation concurrently and asserts exact GST output, plus a lint rule or grep that bans getcontext(). attribute assignment outside startup. Property-based tests over amounts catch precision-dependent outputs that a few hand-picked examples miss.

Scoring: Strong: Exact output; explains the context var holds a shared mutable Context, that tasks copy the mapping shallowly, the difference between mutating and localcontext()/set, and the leak into main. Acceptable: Spots the leak on line 2 but believes main is unaffected, or can't say why the copy doesn't protect it. Red flag: Says Decimal precision is "per task" or "per coroutine", or proposes setting prec back after the await as the fix.

SR-HT-O03 · A retry loop that eats the timeout meant to stop it

Hard & Tricky · Output · Senior (5–8 years) · asyncio, cancellation, CancelledError, asyncio.timeout, retries, BaseException · 10 min

Question: A payout service wraps a bank call in a retry helper and gives the whole operation a 150 ms budget with asyncio.timeout. Incident review: payouts "timed out" in the dashboard but the service kept calling the bank well past the budget, and the handler logged "gave up" instead of a timeout. What does this print? (Each bank call takes 100 ms and then fails.)

import asyncio

attempts = []

async def call_bank(i):
    attempts.append(i)
    await asyncio.sleep(0.1)
    raise ConnectionError("bank 503")

async def with_retries(n):
    for i in range(n):
        try:
            return await call_bank(i)
        except BaseException as exc:          # "never let a retry loop crash"
            print(f"attempt {i}: {type(exc).__name__}")
    return "gave up"

async def main():
    try:
        async with asyncio.timeout(0.15):
            result = await with_retries(3)
        print("result:", result)
    except TimeoutError:
        print("timed out")
    print("attempts:", attempts)
    task = asyncio.current_task()
    print("cancelling:", task.cancelling())

asyncio.run(main())

Expected output:

attempt 0: ConnectionError
attempt 1: CancelledError
attempt 2: ConnectionError
result: gave up
attempts: [0, 1, 2]
cancelling: 0

The trap: attempt 0: ConnectionError, then timed out and attempts: [0, 1]: "the timeout fires at 150 ms, so the second attempt is interrupted and TimeoutError comes out of the async with". That is what happens with except Exception; with except BaseException the timeout is silently defeated.

Explanation:

How asyncio.timeout works: when the deadline passes, it calls task.cancel() on the current task. That raises CancelledError at whatever await the task is suspended on, here attempt 1's sleep (t ≈ 150 ms, 50 ms into the second call). On the way out, __aexit__ checks two things: that the exception leaving the block is CancelledError, and that it isn't a cancellation requested by someone else (it calls task.uncancel() and compares the count). Only then does it convert the cancellation into TimeoutError.

CancelledError derives from BaseException (since 3.8) precisely so that except Exception doesn't swallow it. The retry loop catches BaseException, so it treats the cancellation like any other failed attempt, logs it, and starts attempt 2: another real request to the bank after the budget has expired. When the block finally exits normally with "gave up", __aexit__ sees no exception, so it raises nothing; the deadline has expired, but it has also undone its cancel request (uncancel()), which is why cancelling() is back to 0.

Net effect in production: timeouts don't bound latency, cancelled requests (client disconnects, shutdown) keep hammering a struggling dependency, and graceful shutdown hangs, because the same mechanism also stops TaskGroup and asyncio.run from cancelling the task.

With except Exception the output would be attempt 0: ConnectionError, timed out, attempts: [0, 1].

Takeaway: retry only specific, retryable exceptions; if you must catch broadly, re-raise asyncio.CancelledError first (except asyncio.CancelledError: raise). Never convert cancellation into a normal return.

Follow-ups:

1. "You need cleanup (e.g. release a lock, record a metric) when the call is cancelled. How do you write it?" Use try/finally, or except asyncio.CancelledError: cleanup(); raise. The cleanup must be fast and must not await anything that can hang; if it has to await, bound it, and still re-raise. Never return from a handler for CancelledError.
2. "Is it ever legitimate to suppress a CancelledError?" Rarely: when your code itself cancelled a child task and is awaiting it to confirm it stopped (task.cancel(); with suppress(CancelledError): await task). Even then, check asyncio.current_task().cancelling() so you don't swallow a cancellation aimed at you, which is exactly the bookkeeping asyncio.timeout and TaskGroup do with uncancel().
3. "How would the retry helper respect an overall deadline even without cancellation?" Pass a deadline (loop.time() + budget) down, compute the remaining time before each attempt, skip the retry if the remaining budget is smaller than a minimum useful attempt, and use asyncio.timeout_at(deadline) per attempt. Add jittered backoff, and retry only idempotent operations or ones with an idempotency key.

Scoring: Strong: Exact output including cancelling: 0; explains that timeout cancels the task, needs CancelledError to reach __aexit__, why it's a BaseException, and the production consequences (extra calls, hung shutdown). Acceptable: Recognises the swallowed cancellation and gets most lines, misses the third attempt or the cancelling() value. Red flag: Expects timed out, or recommends except BaseException / bare except in retry loops as defensive coding.

SR-HT-O04 · Vectorised per-merchant counters that undercount

Hard & Tricky · Output · Senior (5–8 years) · numpy, fancy indexing, ufunc.at, bincount, float32, aggregation correctness, performance · 10 min

Question: A fraud feature job replaced a slow Python loop with NumPy to aggregate UPI transactions per merchant in each micro-batch. The numbers are now "a bit low", and a daily counter stored as float32 has stopped moving. What does this print?

import numpy as np

# merchant index of each UPI transaction in one micro-batch
merchant_idx = np.array([0, 2, 2, 1, 2, 0])
amount_inr = np.array([100.0, 250.0, 250.0, 75.0, 40.0, 10.0])

txn_count = np.zeros(3, dtype=np.int64)
txn_count[merchant_idx] += 1
volume = np.zeros(3)
volume[merchant_idx] += amount_inr
print(txn_count, volume)

txn_count = np.zeros(3, dtype=np.int64)
np.add.at(txn_count, merchant_idx, 1)
print(txn_count, np.bincount(merchant_idx, weights=amount_inr, minlength=3))

daily = np.zeros(1, dtype=np.float32)
daily[0] = 16_777_216            # 2**24 transactions so far today
for _ in range(1000):
    daily[0] += 1
print(daily[0], np.float32(16_777_216) + np.float32(1) == np.float32(16_777_216))

Expected output:

[1 1 1] [10. 75. 40.]
[2 1 3] [110.  75. 540.]
1.6777216e+07 True

The trap: [2 1 3] [110. 75. 540.] on the first line, because a[idx] += v "obviously" adds once per index the way the Python loop did. And 16778216.0 on the last line, because adding 1 a thousand times to a float must add 1000.

Explanation:

Mechanism 1: augmented assignment with fancy indexing is buffered, not accumulated. a[idx] += v is executed as a.__setitem__(idx, a.__getitem__(idx) + v): gather all indexed values into a temporary, add, then scatter back. With duplicate indices, each duplicate writes into the same slot, so only one of the additions survives. Merchant 2 appears three times: every copy computes 0 + 1 (count) or 0 + amount, and in practice the last one written wins, 40.0; merchant 0 gets 10.0, not 110.0. (NumPy's docs say which duplicate wins is not guaranteed; either way, the sum is lost.) There is no error and no warning; the result just looks plausible.

The fixes are unbuffered operations: np.add.at(a, idx, v) applies the add once per index occurrence (and has been much faster since NumPy 1.25), and np.bincount(idx, weights=..., minlength=n) is the fastest way to build per-group counts and sums for integer keys. pandas.groupby().sum() is the other usual choice.

Mechanism 2: float32 has a 24-bit significand. Above 2²⁴ = 16,777,216 the gap between adjacent float32 values is 2, so 16777216 + 1 rounds back to 16777216 (round-half-to-even) every time: the counter freezes silently. Big totals in float32 lose paise long before that, too.

Takeaway: test vectorised rewrites against the slow loop on data with duplicate keys; use np.add.at, bincount or groupby for scatter-adds; keep counters in int64 and money in integer paise or Decimal, never float32.

Follow-ups:

1. "How would you have caught this before release?" Differential testing: keep the old loop as an oracle and compare on random batches (seeded, with heavy key repetition) using hypothesis or a fixed generator, plus a reconciliation check in production: the sum of per-merchant counts must equal the batch length, and the sum of volumes must equal amount_inr.sum().
2. "Which is faster here, np.add.at or np.bincount, and when can't you use bincount?" bincount is typically faster: it is a single specialised pass. It only works for non-negative integer keys and sums (or counts); for other ufuncs (np.maximum.at, np.minimum.at), multi-dimensional targets, or negative keys you need ufunc.at or a groupby. Measure with realistic sizes before choosing.
3. "The counter has to be float for a downstream library. What then?" Accumulate in int64 (or float64, which is exact for integers up to 2⁵³) and convert at the boundary. If you must sum many floats, use math.fsum or NumPy's pairwise sum in float64, never a running float32 total.

Scoring: Strong: All three lines exactly; explains gather-add-scatter and why duplicates collapse to one write, names np.add.at/bincount, explains the 2²⁴ significand limit, and proposes a reconciliation invariant. Acceptable: Knows duplicate indices are a problem with += but can't predict the exact values, or misses the float32 freeze. Red flag: Assumes the vectorised version is equivalent to the loop by construction and would ship it without a comparison test.

SR-HT-C01 · Dynamic micro-batching for a GPU model behind an async API

Hard & Tricky · Coding · Senior (5–8 years) · asyncio, futures, cancellation, model serving, batching, latency vs throughput · 30 min

Problem:

A FastAPI service scores transactions with a fraud model on a GPU. One item per forward pass wastes the GPU; a batch of 32 costs almost the same as a batch of 1. Requests arrive one at a time, so you'll build a micro-batcher:

• await batcher.submit(item) returns that item's prediction.
• A batch is dispatched when it reaches max_batch items, or max_wait seconds after the first item of the batch arrived, whichever comes first. Dispatch calls await predict_batch(items), which returns a list of results in the same order.
• Each caller receives its own result. If predict_batch raises, every caller in that batch gets the exception; later batches are unaffected. If it returns the wrong number of results, every caller in the batch gets a ValueError (never route results by position when the lengths disagree).
• Cancellation (client disconnects, request timeouts) must be safe:
• a caller cancelled before its batch is dispatched is dropped from the batch (don't spend GPU time on it);
• a caller cancelled while its batch is running must not affect the other callers in that batch.
• await batcher.close(): stop accepting new items (submit raises RuntimeError), dispatch whatever is queued, and wait for in-flight batches to finish.

Signature: class MicroBatcher(Generic[X, Y]): def __init__(self, predict_batch: Callable[[list[X]], Awaitable[list[Y]]], max_batch: int, max_wait: float) -> None: ... async def submit(self, item: X) -> Y: ... async def close(self) -> None: ...

Constraints:

• Single event loop; everything runs on it (no threads). predict_batch may take tens of milliseconds and several batches may be in flight at once.
• Standard library only.

Examples:

• max_batch=3: three concurrent submit() calls → one predict_batch call with all three, results routed back in order
• max_batch=100, max_wait=0.01: two submit() calls → one predict_batch call ~10 ms after the first item
• max_batch=3, the first caller is cancelled while the batch runs → callers 2 and 3 still get their results

Hints:

1. Give each caller an asyncio.Future and have submit simply await it; the batch task resolves the futures.
2. loop.call_later gives you the max-wait timer; cancel it when the batch fills up first.
3. When the awaiting task is cancelled, the future it awaits is cancelled too. What does set_result on a cancelled future do?
4. Run each batch in its own task, and keep a strong reference to it (the event loop only keeps weak references).
5. Write the cancellation tests first; they are the part most implementations get wrong.

Reference solution:

import asyncio
from collections.abc import Awaitable, Callable
from typing import Generic, TypeVar

X = TypeVar("X")
Y = TypeVar("Y")


class MicroBatcher(Generic[X, Y]):
    """Group concurrent single-item requests into batched model calls.

    A batch is dispatched when it reaches `max_batch` items, or `max_wait` seconds after
    its first item arrived, whichever comes first. Each caller gets its own result, or the
    batch's exception. A caller that is cancelled must not affect anyone else.
    """

    def __init__(self, predict_batch: Callable[[list[X]], Awaitable[list[Y]]],
                 max_batch: int, max_wait: float) -> None:
        if max_batch < 1 or max_wait < 0:
            raise ValueError("max_batch must be >= 1 and max_wait >= 0")
        self._predict = predict_batch
        self._max_batch = max_batch
        self._max_wait = max_wait
        self._pending: list[tuple[X, asyncio.Future[Y]]] = []
        self._timer: asyncio.TimerHandle | None = None
        self._inflight: set[asyncio.Task[None]] = set()
        self._closed = False

    async def submit(self, item: X) -> Y:
        if self._closed:
            raise RuntimeError("batcher is closed")
        loop = asyncio.get_running_loop()
        fut: asyncio.Future[Y] = loop.create_future()
        self._pending.append((item, fut))
        if len(self._pending) >= self._max_batch:
            self._flush()
        elif self._timer is None:
            self._timer = loop.call_later(self._max_wait, self._flush)
        return await fut              # cancelling the caller cancels `fut`

    def _flush(self) -> None:
        if self._timer is not None:
            self._timer.cancel()
            self._timer = None
        # Drop callers that gave up before dispatch: no wasted GPU time on them.
        batch = [(item, fut) for item, fut in self._pending if not fut.done()]
        self._pending = []
        if not batch:
            return
        task = asyncio.get_running_loop().create_task(self._run(batch))
        self._inflight.add(task)              # keep a strong reference until it finishes
        task.add_done_callback(self._inflight.discard)

    async def _run(self, batch: list[tuple[X, asyncio.Future[Y]]]) -> None:
        items = [item for item, _ in batch]
        try:
            results = await self._predict(items)
            if len(results) != len(items):
                raise ValueError(f"predict_batch returned {len(results)} results for {len(items)} items")
        except Exception as exc:
            for _, fut in batch:
                if not fut.done():
                    fut.set_exception(exc)
            return
        for (_, fut), result in zip(batch, results):
            if not fut.done():                # a caller may have been cancelled mid-flight
                fut.set_result(result)

    async def close(self) -> None:
        """Stop accepting work, dispatch what is queued, and wait for in-flight batches."""
        self._closed = True
        self._flush()
        if self._inflight:
            await asyncio.gather(*self._inflight)

Complexity: O(1) amortised per submit plus O(b) per dispatched batch of size b. Memory: O(queued + in-flight items). Added latency per request is at most max_wait plus the batch's model time.

Tests:

import asyncio

calls = []

async def score(items):
    calls.append(list(items))
    await asyncio.sleep(0.02)                   # simulated GPU forward pass
    if "boom" in items:
        raise RuntimeError("CUDA out of memory")
    return [f"{x}:{len(x)}" for x in items]

async def bad_length(items):
    return ["only-one"]

async def main():
    # 1. Size-triggered dispatch, results routed back in order.
    b = MicroBatcher(score, max_batch=3, max_wait=10)
    out = await asyncio.wait_for(asyncio.gather(*(b.submit(x) for x in ["ab", "c", "def"])), 1)
    assert out == ["ab:2", "c:1", "def:3"] and calls == [["ab", "c", "def"]]

    # 2. Time-triggered dispatch for a partial batch.
    calls.clear()
    b = MicroBatcher(score, max_batch=100, max_wait=0.01)
    out = await asyncio.wait_for(asyncio.gather(b.submit("x"), b.submit("yy")), 1)
    assert out == ["x:1", "yy:2"] and calls == [["x", "yy"]]

    # 3. A failing batch fails each of its callers; the next batch is unaffected.
    calls.clear()
    b = MicroBatcher(score, max_batch=2, max_wait=10)
    res = await asyncio.wait_for(asyncio.gather(b.submit("ok"), b.submit("boom"), return_exceptions=True), 1)
    assert [type(r).__name__ for r in res] == ["RuntimeError", "RuntimeError"]
    assert await asyncio.wait_for(asyncio.gather(b.submit("p"), b.submit("q")), 1) == ["p:1", "q:1"]

    # 4. A caller cancelled BEFORE dispatch is dropped from the batch.
    calls.clear()
    b = MicroBatcher(score, max_batch=10, max_wait=0.03)
    tasks = [asyncio.create_task(b.submit(x)) for x in ["a", "bb", "ccc"]]
    await asyncio.sleep(0)
    tasks[1].cancel()
    res = await asyncio.wait_for(asyncio.gather(*tasks, return_exceptions=True), 1)
    assert res[0] == "a:1" and res[2] == "ccc:3"
    assert isinstance(res[1], asyncio.CancelledError)
    assert calls == [["a", "ccc"]]

    # 5. A caller cancelled WHILE its batch is running doesn't break the others.
    calls.clear()
    b = MicroBatcher(score, max_batch=3, max_wait=10)
    tasks = [asyncio.create_task(b.submit(x)) for x in ["first", "second", "third"]]
    await asyncio.sleep(0.005)                  # batch dispatched, model still running
    assert calls == [["first", "second", "third"]]
    tasks[0].cancel()
    res = await asyncio.wait_for(asyncio.gather(*tasks, return_exceptions=True), 1)
    assert isinstance(res[0], asyncio.CancelledError)
    assert res[1:] == ["second:6", "third:5"]

    # 6. A result-count mismatch is an error for everyone, not a silent misroute.
    b = MicroBatcher(bad_length, max_batch=2, max_wait=10)
    res = await asyncio.wait_for(asyncio.gather(b.submit("u"), b.submit("v"), return_exceptions=True), 1)
    assert all(isinstance(r, ValueError) for r in res)

    # 7. close() flushes queued work and refuses new work.
    calls.clear()
    b = MicroBatcher(score, max_batch=10, max_wait=10)
    t = asyncio.create_task(b.submit("late"))
    await asyncio.sleep(0)
    await asyncio.wait_for(b.close(), 1)
    assert await t == "late:4"
    try:
        await b.submit("after-close")
        raise AssertionError("submit after close must fail")
    except RuntimeError:
        pass

asyncio.run(main())

A common wrong solution (the tests catch it):

import asyncio


class MicroBatcher:
    def __init__(self, predict_batch, max_batch, max_wait):
        self._predict = predict_batch
        self._max_batch = max_batch
        self._max_wait = max_wait
        self._pending = []
        self._timer = None
        self._inflight = set()
        self._closed = False

    async def submit(self, item):
        if self._closed:
            raise RuntimeError("batcher is closed")
        loop = asyncio.get_running_loop()
        fut = loop.create_future()
        self._pending.append((item, fut))
        if len(self._pending) >= self._max_batch:
            self._flush()
        elif self._timer is None:
            self._timer = loop.call_later(self._max_wait, self._flush)
        return await fut

    def _flush(self):
        if self._timer is not None:
            self._timer.cancel()
            self._timer = None
        batch, self._pending = self._pending, []
        if not batch:
            return
        task = asyncio.get_running_loop().create_task(self._run(batch))
        self._inflight.add(task)
        task.add_done_callback(self._inflight.discard)

    async def _run(self, batch):
        items = [item for item, _ in batch]
        try:
            results = await self._predict(items)
            if len(results) != len(items):
                raise ValueError("length mismatch")
        except Exception as exc:
            for _, fut in batch:
                fut.set_exception(exc)
            return
        for (_, fut), result in zip(batch, results):
            fut.set_result(result)

    async def close(self):
        self._closed = True
        self._flush()
        if self._inflight:
            await asyncio.gather(*self._inflight)

Alternatives and tradeoffs:

• A single consumer task reading from an asyncio.Queue: pull the first item, then keep pulling with asyncio.wait_for(queue.get(), remaining) until full or timed out. Equivalent behaviour; one long-lived task to supervise and restart, instead of timer callbacks.
• Bound concurrency of in-flight batches with a semaphore (one or two per GPU) and apply backpressure to submit (reject with 503 when the queue is too deep) instead of letting latency grow without limit.
• Serving frameworks (Triton dynamic batching, Ray Serve @serve.batch, TorchServe) implement this with per-model configs. Building it yourself makes sense for a small service; the hard parts are the same either way: cancellation, error isolation and latency SLOs.
• CPU-bound models: predict_batch must not block the loop. Run it with asyncio.to_thread or in a process pool, otherwise the timer and every other request stall while the model runs.

Follow-ups:

1. "One poisoned input crashes the whole batch every time. How do you stop it taking 31 other requests down?" On a batch failure, bisect: retry the two halves separately (or fall back to batch size 1) so only the offending item fails, and record the failing input for debugging. Validate inputs (shape, NaN, length) before batching so most poison never reaches the model.
2. "How do you choose max_batch and max_wait?" From the latency budget and a load test. max_wait adds directly to p50 latency under low traffic, so keep it a small fraction of the SLO (e.g. 2–5 ms of a 50 ms budget). max_batch comes from GPU memory and the throughput curve: increase it until throughput stops improving or p99 breaks the SLO. Export batch size and queue wait as histograms to tune it in production.
3. "The model is updated (fraud-v7 → v8) while requests are queued. What could go wrong?" A batch could mix items scored by different versions, or a hot swap could happen mid-batch. Bind the model version when a batch is dispatched, return it with each result (for audit and debugging), and swap models by creating a new batcher and draining the old one with close().

Scoring: Strong: Future-per-caller design with size and time triggers, filters cancelled futures before dispatch, guards set_result/set_exception with done(), keeps task references, validates result length; explains the latency/throughput trade-off and how to tune and observe it. Acceptable: Batching by size and time works and errors propagate, but needs a hint to handle cancelled callers (InvalidStateError) or to keep a reference to batch tasks. Red flag: Calls the model once per request inside submit, blocks the loop with a synchronous model call, or can't explain what happens to other callers when one is cancelled.

SR-HT-C02 · Incremental sync that doesn't lose late-committed rows

Hard & Tricky · Coding · Senior (5–8 years) · data pipelines, incremental loads, watermarks, idempotency, late data, backfills · 25 min

Problem:

Every 15 minutes a job copies changed rows from the orders table (OLTP) into the analytics store. It keeps a watermark: the largest updated_at it has seen. Finance found orders that exist in OLTP but never reached analytics, and nobody can reproduce it.

The cause: a row's updated_at is set when its transaction starts, but the row becomes visible only when the transaction commits. A transaction that starts at t=105 and commits after the job's run at t=105 is invisible to that run, and later runs that read "rows newer than the watermark" skip it forever.

Implement one run of the job: sync(source, target, state, lookback=0):

• source: the rows visible right now ({"id", "version", "updated_at", ...}); target: dict id → row; state["watermark"]: None before the first run.
• Read rows with updated_at >= watermark - lookback (all rows on the first run). The comparison is inclusive: rows that share the watermark's timestamp must not be skipped. lookback (seconds) re-reads a window before the watermark to catch commits that land late with an older timestamp.
• Apply a row only if its version is newer than the version already in target (or the id is new), so re-reads, overlapping windows, reruns and replays of old data are no-ops. Store a copy of the row.
• New watermark = max updated_at among the rows read, but never move it backwards; unchanged if nothing was read.
• Return the number of rows applied (inserted or updated).

Signature: def sync(source: list[Row], target: dict[str, Row], state: SyncState, lookback: int = 0) -> int:

Constraints:

• updated_at is an integer number of seconds; version increases by 1 on every update of a row.
• The same id may appear more than once in one window (several updates between runs), in any order.
• lookback >= 0; raise ValueError otherwise.

Examples:

• watermark 105; a row with updated_at=105 commits after the last run → the next run applies it (>= not >)
• watermark 105; late row with updated_at=101; lookback=10 → applied; watermark stays 105
• the same run executed twice → second run returns 0 and changes nothing

Hints:

1. Draw a timeline of two transactions and two job runs. Which rows can each run see?
2. > vs >= on the watermark: which one loses rows, and which one re-reads rows? Which of those is safe?
3. Re-reading is only safe if applying a row twice is a no-op. What makes it a no-op?
4. What should happen to the watermark when the only rows read are old ones from the lookback window?
5. If one window contains versions 2 and 3 of the same id, in either order, version 3 must win.

Reference solution:

from typing import Any, TypedDict

Row = dict[str, Any]          # {"id": str, "version": int, "updated_at": int, ...}


class SyncState(TypedDict):
    watermark: int | None


def sync(source: list[Row], target: dict[str, Row], state: SyncState, lookback: int = 0) -> int:
    """Copy changed rows from `source` into `target`; return how many rows were applied.

    - Reads rows with updated_at >= watermark - lookback (inclusive: rows that share the
      watermark's timestamp but committed after the last run must not be skipped).
    - Applies a row only if its version is newer than what target holds, so replays,
      overlapping windows and backfills from old snapshots are no-ops.
    - Advances the watermark to the max updated_at seen, and never moves it backwards.
    """
    if lookback < 0:
        raise ValueError("lookback must be >= 0")
    wm = state["watermark"]
    since = None if wm is None else wm - lookback
    changed = [r for r in source if since is None or r["updated_at"] >= since]

    applied = 0
    # Lowest version first, so if the window holds two versions of an id, the newest wins.
    for row in sorted(changed, key=lambda r: (r["id"], r["version"])):
        current = target.get(row["id"])
        if current is None or row["version"] > current["version"]:
            target[row["id"]] = dict(row)     # copy: target must not alias source rows
            applied += 1

    if changed:
        newest = max(r["updated_at"] for r in changed)
        state["watermark"] = newest if wm is None else max(wm, newest)
    return applied

Complexity: O(n log n) per run for n rows in the window (the sort); O(n) extra memory for the window.

Tests:

target = {}
state = {"watermark": None}
source = [
    {"id": "ORD-1", "version": 1, "updated_at": 100, "status": "PLACED"},
    {"id": "ORD-2", "version": 1, "updated_at": 105, "status": "PLACED"},
]

# Initial load.
assert sync(source, target, state) == 2
assert state["watermark"] == 105

# Rerunning the same job is a no-op (idempotent).
assert sync(source, target, state) == 0
assert state["watermark"] == 105

# A row whose transaction began at t=105 commits after the run: same timestamp as the watermark.
source.append({"id": "ORD-3", "version": 1, "updated_at": 105, "status": "PLACED"})
assert sync(source, target, state) == 1
assert "ORD-3" in target

# Late commit with an OLDER timestamp: only a lookback window catches it.
source.append({"id": "ORD-4", "version": 1, "updated_at": 101, "status": "PLACED"})
assert sync(source, target, state, lookback=0) == 0
assert sync(source, target, state, lookback=10) == 1
assert state["watermark"] == 105                      # lookback must not drag the watermark back

# Update to an existing row; the source row object is not aliased into target.
upd = {"id": "ORD-1", "version": 2, "updated_at": 120, "status": "SHIPPED"}
source.append(upd)
assert sync(source, target, state, lookback=10) == 1
assert target["ORD-1"]["status"] == "SHIPPED" and state["watermark"] == 120
upd["status"] = "MUTATED-IN-SOURCE"
assert target["ORD-1"]["status"] == "SHIPPED"

# Backfill from an old snapshot replays stale versions with NEW timestamps: must not regress.
stale_backfill = [{"id": "ORD-1", "version": 1, "updated_at": 130, "status": "PLACED"}]
assert sync(stale_backfill, target, state) == 0
assert target["ORD-1"]["status"] == "SHIPPED"
assert state["watermark"] == 130

# Two versions of one id in a single window: the newest wins regardless of source order.
burst = [
    {"id": "ORD-5", "version": 3, "updated_at": 140, "status": "DELIVERED"},
    {"id": "ORD-5", "version": 2, "updated_at": 135, "status": "SHIPPED"},
]
sync(burst, target, state)
assert target["ORD-5"]["status"] == "DELIVERED" and target["ORD-5"]["version"] == 3

# Empty window leaves the watermark alone.
assert sync([], target, state) == 0 and state["watermark"] == 140

A common wrong solution (the tests catch it):

def sync(source, target, state, lookback=0):
    wm = state["watermark"]
    since = None if wm is None else wm - lookback
    changed = [r for r in source if since is None or r["updated_at"] > since]
    applied = 0
    for row in sorted(changed, key=lambda r: (r["id"], r["version"])):
        current = target.get(row["id"])
        if current is None or row["version"] > current["version"]:
            target[row["id"]] = dict(row)
            applied += 1
    if changed:
        newest = max(r["updated_at"] for r in changed)
        state["watermark"] = newest if wm is None else max(wm, newest)
    return applied

Alternatives and tradeoffs:

• Change data capture (CDC) from the database log (Debezium on the Postgres WAL / MySQL binlog) orders changes by commit, so there is no late-commit gap at all. It's the robust fix for high-value tables; watermark polling with lookback is the pragmatic fix when CDC isn't available.
• Watermark on a commit-ordered column: a sequence assigned at commit (e.g. a trigger-maintained change log, or Postgres xmin with care about wraparound) instead of updated_at from the application clock.
• Upper bound on the window: only read rows with updated_at <= now - safety_lag, so in-flight transactions have time to commit; combine with a lookback for the stragglers.
• Upsert in the warehouse: MERGE ... WHEN MATCHED AND src.version > tgt.version gives the same idempotency in SQL; the logic here is what that statement must express.

Follow-ups:

1. "How big should lookback be, and how would you know it's big enough?" At least the longest transaction duration plus clock skew between app servers and the DB, with margin (for example the p99.9 transaction time × 2). Verify it continuously: a daily reconciliation job that compares counts and checksums per day between OLTP and analytics, and alert on any row whose updated_at is older than watermark - lookback when it first appears (that's a row the lookback would have missed).
2. "Rows can also be hard-deleted in OLTP. What does this design miss, and how do you handle it?" A hard delete leaves nothing to read, so the target keeps the row forever. Options: soft deletes (a deleted_at column, which flows through the same versioned upsert as a tombstone), CDC which emits delete events, or a periodic anti-join reconciliation of ids. Keep tombstones (id + version) in the target so an older replay can't resurrect a deleted row.
3. "Where should the watermark be stored, and when is it updated?" In durable state (a table in the target or a checkpoint store) updated in the same transaction as the upserts, or strictly after they succeed. Updating it first and then crashing skips data; updating it after the upserts and crashing re-reads data, which is safe because applying is idempotent.

Scoring: Strong: Explains the start-vs-commit visibility gap before coding; uses >= plus lookback; makes application idempotent with a version guard; keeps the watermark monotonic; mentions CDC, reconciliation and where the watermark is persisted. Acceptable: Writes a correct version-guarded upsert and monotonic watermark, but uses > or no lookback until the tie and late-commit cases are pointed out. Red flag: Sets the watermark to the job's wall-clock start time, overwrites rows without a version check, or believes that "it runs every 15 minutes" means no rows can be missed.

SR-HT-C03 · Cache keys that are stable across processes and mean what the call means

Hard & Tricky · Coding · Senior (5–8 years) · caching, inspect.signature, canonical serialisation, hashing, determinism, API design · 30 min

Problem:

Your team is moving a @cached decorator from an in-process dict to Redis, shared by 30 Gunicorn workers on 8 hosts. The old decorator used hash((args, tuple(kwargs.items()))) as the key. In Redis the hit rate is close to zero, and after a price change in Bengaluru, invalidation missed some entries and stale prices were served for hours.

Write cache_key(fn, version, *args, **kwargs) -> str that returns a hex SHA-256 key such that:

• Stable across processes and restarts: no hash(), no id(), no dependence on set or dict iteration order.
• Same meaning, same key: positional vs keyword arguments, and omitted defaults vs explicitly passed defaults, give the same key (f("A1"), f("A1", 1), f(sku="A1", qty=1) when qty=1 is the default).
• Different meaning, different key: 1, 1.0, True and "1" are all different keys (Python considers the first three equal, which is how the old cache mixed them up). Lists and tuples with the same items are the same; order inside a list matters; dict key order and set order don't.
• Equal values share a key: Decimal("1.50") and Decimal("1.5"); aware datetimes that are the same instant in different time zones (10:00 IST == 04:30 UTC). A date is not the same as its ISO string.
• Reject what can't be keyed safely with TypeError: unknown types (arbitrary objects), NaN, naive datetimes. A call that fn itself would reject (too many arguments) also raises TypeError.
• The key includes the function's module and qualified name, and version (bumped when the function's logic changes).

Signature: def cache_key(fn: Callable[..., Any], version: str, /, *args: Any, **kwargs: Any) -> str:

Constraints:

• Supported value types: None, bool, int, float, Decimal, str, date, aware datetime, and list, tuple, set, frozenset, dict nested arbitrarily of those.
• Standard library only. The key must be a 64-character lowercase hex string.

Examples:

• cache_key(price_quote, 'v1', 'A1', 2) vs cache_key(price_quote, 'v1', sku='A1', qty=2) → equal
• cache_key(price_quote, 'v1', 'A1', 1) vs cache_key(price_quote, 'v1', 'A1', True) → different
• cache_key(price_quote, 'v1', 'A1', object()) → TypeError

Hints:

1. inspect.signature(fn).bind(*args, **kwargs) followed by apply_defaults() normalises how arguments were passed.
2. Turn every value into a canonical JSON-able structure first, then hash json.dumps(..., sort_keys=True).
3. Tag each value with its type, so 1, 1.0 and True can't collide. Check bool before int and datetime before date: why?
4. Sets and dict keys can mix types, so sorted() on raw values may raise. Sort by the canonical encoding instead.
5. How will you prove stability across processes in a test? String hashing changes with PYTHONHASHSEED.

Reference solution:

import datetime as dt
import hashlib
import inspect
import json
from collections.abc import Callable
from decimal import Decimal
from typing import Any


def _canon(value: Any) -> Any:
    """Map a value to a JSON-able structure that is equal iff the values should share a key.

    Every scalar is tagged with its type, so 1, 1.0, True and "1" never collide.
    """
    if value is None:
        return ["none"]
    if isinstance(value, bool):                    # before int: bool is a subclass of int
        return ["bool", value]
    if isinstance(value, int):
        return ["int", str(value)]                 # str: JSON consumers may lose big ints
    if isinstance(value, float):
        if value != value:
            raise TypeError("NaN is not a usable cache key")
        return ["float", repr(value)]
    if isinstance(value, Decimal):
        return ["decimal", str(value.normalize())]  # Decimal("1.50") and Decimal("1.5") are equal
    if isinstance(value, str):
        return ["str", value]
    if isinstance(value, dt.datetime):             # before date: datetime is a subclass of date
        if value.tzinfo is None:
            raise TypeError("naive datetime is ambiguous across servers; pass an aware one")
        return ["datetime", value.astimezone(dt.timezone.utc).isoformat()]
    if isinstance(value, dt.date):
        return ["date", value.isoformat()]
    if isinstance(value, (list, tuple)):
        return ["seq", [_canon(v) for v in value]]
    if isinstance(value, (set, frozenset)):
        items = [_canon(v) for v in value]
        return ["set", sorted(items, key=lambda c: json.dumps(c, sort_keys=True))]
    if isinstance(value, dict):
        pairs = [[_canon(k), _canon(v)] for k, v in value.items()]
        return ["map", sorted(pairs, key=lambda p: json.dumps(p[0], sort_keys=True))]
    raise TypeError(f"unsupported cache key type: {type(value).__name__}")


def cache_key(fn: Callable[..., Any], version: str, /, *args: Any, **kwargs: Any) -> str:
    """Stable key for fn(*args, **kwargs): the same across processes, hosts and restarts.

    Calls that mean the same thing (positional vs keyword, defaults omitted or passed)
    get the same key. Bump `version` when fn's logic changes, to orphan old entries.
    """
    bound = inspect.signature(fn).bind(*args, **kwargs)   # TypeError on a bad call, like fn would
    bound.apply_defaults()
    payload = {
        "fn": f"{fn.__module__}.{fn.__qualname__}",
        "v": version,
        "args": [[name, _canon(val)] for name, val in bound.arguments.items()],
    }
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()

Complexity: O(n log n) in the total number of values n (sorting set elements and dict keys at each level), plus O(size) for serialising and hashing.

Tests:

import datetime as dt
import os
import subprocess
import sys
from decimal import Decimal

IST = dt.timezone(dt.timedelta(hours=5, minutes=30))

def price_quote(sku, qty=1, *, city="Bengaluru", filters=None):
    return None

SAMPLE = {"filters": {"tags": {"veg", "dairy", "paneer", 7}, "brand": "Amul"}}
if os.environ.get("CACHE_KEY_PROBE"):          # child process: print the key and stop
    print(cache_key(price_quote, "v1", "A1", 2, **SAMPLE))
    sys.exit(0)

def other(sku, qty=1, *, city="Bengaluru", filters=None):
    return None

k = cache_key(price_quote, "v1", "A1", 2)
assert len(k) == 64 and int(k, 16) >= 0

# Same meaning, same key: positional vs keyword, defaults omitted vs explicit.
assert cache_key(price_quote, "v1", "A1", 2) == cache_key(price_quote, "v1", sku="A1", qty=2)
assert cache_key(price_quote, "v1", "A1") == cache_key(price_quote, "v1", "A1", 1, city="Bengaluru")
assert cache_key(price_quote, "v1", "A1", 2, city="Pune") != k

# Order of dict keys and set elements doesn't matter; mixed-type sets don't crash.
f1 = {"brand": "Amul", "tags": {"dairy", "veg", 7}}
f2 = {"tags": {7, "veg", "dairy"}, "brand": "Amul"}
assert cache_key(price_quote, "v1", "A1", filters=f1) == cache_key(price_quote, "v1", "A1", filters=f2)

# Values that compare equal in Python but mean different things get different keys.
assert cache_key(price_quote, "v1", "A1", 1) != cache_key(price_quote, "v1", "A1", True)
assert cache_key(price_quote, "v1", "A1", 1) != cache_key(price_quote, "v1", "A1", 1.0)
assert cache_key(price_quote, "v1", "A1", 1) != cache_key(price_quote, "v1", "A1", "1")
assert cache_key(price_quote, "v1", "A1", [1, 2]) == cache_key(price_quote, "v1", "A1", (1, 2))
assert cache_key(price_quote, "v1", "A1", [1, 2]) != cache_key(price_quote, "v1", "A1", [2, 1])

# Equal amounts and equal instants share a key.
assert cache_key(price_quote, "v1", "A1", Decimal("1.50")) == cache_key(price_quote, "v1", "A1", Decimal("1.5"))
t_ist = dt.datetime(2026, 10, 4, 10, 0, tzinfo=IST)
t_utc = dt.datetime(2026, 10, 4, 4, 30, tzinfo=dt.timezone.utc)
assert cache_key(price_quote, "v1", "A1", t_ist) == cache_key(price_quote, "v1", "A1", t_utc)
assert cache_key(price_quote, "v1", "A1", dt.date(2026, 10, 4)) != cache_key(price_quote, "v1", "A1", "2026-10-04")

# Function identity and version are part of the key.
assert cache_key(other, "v1", "A1", 2) != k
assert cache_key(price_quote, "v2", "A1", 2) != k

# Stable across processes: string hashing (and so set order) differs per hash seed.
here = cache_key(price_quote, "v1", "A1", 2, **SAMPLE)
for seed in ("1", "2", "3"):
    out = subprocess.run([sys.executable, __file__], env={"CACHE_KEY_PROBE": "1", "PYTHONHASHSEED": seed},
                         capture_output=True, text=True, check=True).stdout.strip()
    assert out == here, (seed, out, here)

# Reject what can't be keyed safely, and calls fn itself would reject.
for bad in (object(), float("nan"), dt.datetime(2026, 10, 4, 10, 0)):
    try:
        cache_key(price_quote, "v1", "A1", bad)
        raise AssertionError(f"{bad!r} should be rejected")
    except TypeError:
        pass
try:
    cache_key(price_quote, "v1", "A1", 2, 3)
    raise AssertionError("too many positional args must raise")
except TypeError:
    pass

A common wrong solution (the tests catch it):

import hashlib
import json


def cache_key(fn, version, /, *args, **kwargs):
    payload = {"fn": f"{fn.__module__}.{fn.__qualname__}", "v": version,
               "args": [repr(a) for a in args],
               "kwargs": {k: repr(v) for k, v in kwargs.items()}}
    blob = json.dumps(payload, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()

Alternatives and tradeoffs:

• Explicit key functions: @cached(key=lambda sku, qty=1, **_: f"price:{sku}:{qty}"). Most robust and readable for hot paths; the generic canonicaliser is for a shared decorator used by many teams.
• pickle.dumps of the arguments: not stable (depends on protocol, set order and object internals), and it invites unpickling later. Avoid.
• Readable keys: f"{namespace}:{fn}:{version}:{sha256[:16]}" keeps keys debuggable in Redis (SCAN price_quote:*) and lets you invalidate a namespace by bumping version.
• Pydantic models as arguments: canonicalise via model.model_dump(mode="json"), which gives a stable JSON-able structure for nested models.

Follow-ups:

1. "Why did the old hash()-based key give a near-zero hit rate in Redis?" String and bytes hashes are salted per process (PYTHONHASHSEED), so each Gunicorn worker computed different keys for the same call. In a per-process dict that's invisible; in a shared cache, each worker only ever hits its own entries, and they are lost on every restart.
2. "How could invalidation miss entries for Bengaluru?" Keyword arguments that equal their default were sometimes passed and sometimes omitted, and the key used kwargs.items() without binding to the signature, so the code that built keys and the code that invalidated them disagreed about which entry belonged to city="Bengaluru". Normalising through the signature, and invalidating by the same function, removes that class of bug.
3. "The function's logic changes in a deploy, but old entries stay in Redis for 24 h. What do you do?" Bump version (or derive it from a hash of the function's source or the package version) so new code reads new keys and old entries expire naturally. During a rolling deploy both versions run side by side, and each must only read the keys it understands, which versioned keys guarantee.

Scoring: Strong: Binds arguments to the signature with defaults, builds a type-tagged canonical form (bool before int, datetime before date, normalised Decimal and UTC instants), sorts sets/dicts by canonical encoding, hashes with SHA-256, and tests stability across hash seeds; explains why hash() failed. Acceptable: Uses JSON with sort_keys and SHA-256 and binds the signature, but misses type collisions (1/True/1.0) or set ordering until tested. Red flag: Uses hash(), repr() or str() of the arguments as the key, or argues that key collisions in a cache are harmless.

SR-HT-D01 · An internal Python SDK for the payments platform, used by 40 teams

Hard & Tricky · Design · Senior (5–8 years) · library design, packaging, versioning, dependency management, sync and async APIs, retries, observability, deprecation · 45 min

Scenario:

Forty teams call the internal payments platform (create payment, refund, payout, status) over HTTP. Each team wrote its own client: some retry POST /payments on timeout (causing double charges), some don't set timeouts, three pin requests to old versions, and every API change breaks someone. You're asked to design payments-sdk, a Python package every team will use.

Constraints you learn quickly: - Consumers range from Django 3.2 monoliths on Python 3.9 to FastAPI services on 3.12, Airflow DAGs, and notebooks. - Half the services are async (FastAPI, aiohttp), half are sync. - Teams upgrade dependencies irregularly; some services are touched twice a year. - The platform API itself evolves (v1 → v2 adds idempotency_key and amounts in paise).

Clarifying questions:

• Which Python versions must be supported, and for how long? (3.9 to 3.13 today; 3.9 support must have an end date.)
• Is there a machine-readable API spec (OpenAPI)? Who owns it? (Yes, owned by the platform team; it's mostly accurate.)
• What do consumers need beyond calls: webhooks verification, pagination helpers, test fakes? (Webhook signature verification and a test fake are top requests.)
• How are services authenticated, and who manages credentials? (Service tokens from a secrets manager, rotated monthly.)
• How will you know who uses which version and which methods? (Nobody knows today; that's part of the ask.)

Reference design:

Dependencies: be a good guest in 40 dependency trees. The SDK's dependency choices become everyone's constraints. Depend on one HTTP library (httpx, which supports both sync and async) with a wide version range (httpx>=0.24,<1), and Pydantic only if you support both v1 and v2 or can require v2 everywhere (that's often the real blocker; plain dataclasses for the public models avoid the issue). No pins (==) in a library; pins belong in applications' lock files. Zero import-time side effects: no network calls, no logging configuration, no global monkeypatching.

API shape. - Two clients with the same surface: PaymentsClient (sync) and AsyncPaymentsClient, generated from one implementation (shared request-building and response-parsing code; only the transport differs) so they can't drift. Both are context managers that own a connection pool; construct once per process, not per call. - Typed request/response models, money as integer paise (or Decimal) in a Money type with currency, py.typed shipped so mypy/pyright users get the types. - Explicit, safe defaults: connect/read timeouts always set; retries only for idempotent operations, or for POST with an idempotency key that the SDK generates if the caller doesn't pass one, reused across retries of that call (this fixes the double-charge bug structurally). Jittered backoff, a retry budget, and respect for Retry-After. - A small exception hierarchy: PaymentsError → ApiError(status, code, request_id), RateLimited, Timeout, ValidationError. Never leak httpx exceptions through the public API, so the transport can change. - Keyword-only arguments for everything except the obvious first one, so adding parameters never breaks callers.

Versioning and evolution. - Semantic versioning with a written policy: what counts as breaking (removing/renaming, changing defaults, exception types), deprecation for at least two minor releases with DeprecationWarning(stacklevel=2) and a changelog entry, and an announced removal version. - The SDK targets the platform's v2 API while keeping v1-shaped convenience methods as deprecated shims where possible, so teams upgrade the SDK first and change their code later. - CI matrix across supported Python versions and the min/max of each dependency range (for example nox sessions that install with uv pip install --resolution lowest-direct), plus a contract test suite run against the platform's staging API on every platform release.

Testing support for consumers. Ship payments_sdk.testing with a FakePaymentsClient (in-memory, same interface, configurable failures) and pytest fixtures, so teams stop mocking httpx internals, and their tests break when the SDK's interface changes rather than silently passing.

Observability and ownership. Every request sends a User-Agent like payments-sdk/2.3.1 python/3.12 service/<name>; the platform can then answer "who still uses 1.x / the deprecated refund path". Optional OpenTelemetry spans and a standard logger name (payments_sdk) with a NullHandler, never configuring logging for the host app. Request IDs from responses are attached to exceptions.

Distribution. Published to the internal package index (Artifactory/CodeArtifact) as a wheel with a pure-Python build; release automation from tags; a migration guide per major version; an owner team with an SLA for issues.

Trade-offs:

• Generated client (from OpenAPI) vs hand-written: generation keeps it in sync with the API and is cheap to maintain, but generated code is often unidiomatic (bad names, no retries/idempotency, weak errors). A common middle ground: generate the low-level layer, hand-write the thin public layer that teams use.
• One client with sync and async methods vs two clients: one class with create() and acreate() is easy to discover but doubles the surface on one object; two clients keep each simple. Either way, share the core logic.
• Pydantic models vs dataclasses: Pydantic gives validation and JSON handling but forces a major version onto every consumer; dataclasses plus a small validation layer keep the dependency tree light.
• Wide dependency ranges vs tested pins: wide ranges avoid resolver conflicts across 40 apps but increase the test matrix and the risk of an untested combination. Test min and max of each range in CI.
• Centralised SDK vs letting teams call HTTP directly: the SDK is a product with an owner and a support cost; its value is that correctness (timeouts, retries, idempotency, webhook verification) is fixed once.

Deep-dive probes:

• "Team X can't install payments-sdk 2.0 because it requires Pydantic 2 and they're on Pydantic 1 via an old FastAPI." Look for: understanding that a library's dependency is a constraint on the whole app; options are avoiding the dependency in the public surface, supporting both via a compatibility layer, or keeping 1.x on a maintenance branch with security fixes for a fixed period.
• "A teammate wants the SDK to auto-retry create_payment on any timeout. Review that." Look for: a timeout means the outcome is unknown; retry is only safe with an idempotency key honoured by the server; the key must be stable across retries of the same logical call, not regenerated per attempt; and the SDK should surface "unknown outcome" distinctly when retries are exhausted.
• "How do you remove a deprecated method that 6 teams still call?" Look for: usage telemetry by service from the User-Agent/metrics, direct outreach and PRs to those teams, CI warnings-as-errors for the deprecated path, a dated removal in a major version, and keeping the old major alive for security fixes for a defined period.
• "What happens if the SDK creates a new client (and connection pool) per call inside a Celery task?" Look for: no connection reuse, TLS handshakes per call, file descriptor exhaustion under load; document and enforce one client per process (module-level lazy singleton or dependency injection), and be careful with clients created before a fork (pre-fork servers, Celery prefork) because sockets don't survive fork safely.

Rubric: Strong: Treats the SDK as a product whose dependencies constrain 40 apps (wide ranges, no pins, no import side effects); designs safe defaults (timeouts, idempotency keys reused across retries, clear exception hierarchy); a sync and async surface sharing one core; a written semver and deprecation policy with telemetry to drive removals; and a test fake for consumers. Acceptable: Proposes a typed client with retries, timeouts and semver, but misses dependency-conflict risks, generates new idempotency keys per retry, or ignores the sync/async split until prompted. Red flag: Pins exact dependency versions in the library, retries POSTs blindly on timeout, configures logging or does network calls at import time, or has no plan for evolving the API without breaking teams.

SR-HT-D02 · A 50-minute, flaky CI for the payments monolith

Hard & Tricky · Design · Senior (5–8 years) · testing strategy, pytest, flaky tests, test isolation, contract tests, CI performance, determinism · 40 min

Scenario: You join as the senior engineer on a payments monolith (Django, Postgres, Celery, Redis; ~300k lines of Python). The test suite has 14,000 tests and takes 50 minutes in CI. About 1 in 4 runs fails on a test that passes on retry, so developers hit "re-run" by habit. Last month, a real bug in refund rounding shipped because its test had been marked @pytest.mark.skip("flaky") six months earlier. Product wants faster releases; the CTO asks you for a plan to make CI fast and trustworthy within a quarter, without stopping feature work.

Clarifying questions:

• What's in the 14,000 tests? (A rough census: ~9,000 Django TestCase tests hitting Postgres, ~3,000 pure unit tests, ~1,500 "integration" tests calling Redis/Celery and stubbed external services, ~500 Selenium tests.)
• Is there data on which tests are slow and which are flaky? (CI keeps JUnit XML for 90 days; nobody has analysed it.)
• How do tests run today? (Single job, pytest serially, one shared Postgres; some tests depend on fixtures loaded by earlier tests.)
• What do the external integrations look like? (Bank and UPI gateways, SMS, a tax service; tests use a mix of mock.patch and a shared sandbox environment.)
• What are the release mechanics? (Weekly release train; a failed main build blocks everyone.)

Reference design:

1. Measure first (week 1). Parse the JUnit XML history: per-test duration, failure rate, and flake rate (failed then passed on the same commit). Typically a small set of tests accounts for most of the time and most of the flakes. Publish a dashboard; it's the baseline for every later claim.

2. Make flakes visible and owned, not ignored. - Policy: a flaky test is a bug. It goes into quarantine (a marker that runs it in a separate, non-blocking job that still reports results) with an owner and a ticket with a deadline. skip without a ticket is banned by a lint rule; quarantine has a size budget. - Fix the root causes, which in Python suites are usually a handful of patterns: - Time: tests using datetime.now(), date.today() or IST/UTC day boundaries (fails between 00:00 and 05:30 IST). Inject a clock or freeze time (time-machine), and add a CI job that runs with TZ=Asia/Kolkata and one with TZ=UTC. - Order dependence and shared state: tests that rely on data from earlier tests, module-level caches, lru_cache, settings mutated without restore, singletons. Detect with pytest-randomly (random order with a printed seed) and fix by making fixtures own their data. - Randomness and hash order: unseeded random, factories generating colliding unique values, assertions on set order. Seed per test (the seed is printed so failures replay). - Async and Celery timing: sleep(1) then assert. Replace with eager Celery in unit tests, and polling with a deadline (or explicit synchronisation) in integration tests. - External sandboxes: replace calls to shared sandboxes in the main suite with fakes/recorded responses; keep a small, separate, non-blocking suite against real sandboxes.

3. Make it fast. - Parallelise: pytest-xdist with one Postgres database per worker (pytest-django creates a test_<db>_gw0, gw1, ... database per xdist worker; Django's own --parallel clones one per process), then shard across CI machines by historical duration. This alone usually takes 50 minutes to under 10 on enough runners, but only after order dependence is fixed, which is why step 2 comes first. - Cheaper DB tests: use transactions rolled back per test (TestCase) instead of truncation (TransactionTestCase) wherever possible; build fixtures with factories instead of loading large JSON fixtures per test. - Test selection on PRs: run tests affected by changed files (coverage-based mapping or import graph) plus a smoke set, with the full suite on main and nightly. Selection must be conservative: when in doubt, run more. - Move tests down the pyramid: much of the 9,000 DB-backed tests check pure logic (fees, rounding, state machines). Extract that logic into pure functions/modules tested without the DB, and add property-based tests (Hypothesis) for money and rounding invariants, which is exactly where the refund bug lived.

4. Contracts at the boundaries. For bank/UPI/SMS/tax integrations, define the contract once: recorded, versioned example payloads (including error responses) validated against Pydantic models, a fake that implements the contract and is used by unit tests, and a scheduled contract test against the real sandbox that alerts when the provider changes. create_autospec (not bare Mock) where mocks remain, so signature drift fails tests.

5. Guardrails so it stays fixed. CI budget alerts (p50/p90 duration), a flake-rate SLO per week, required ownership metadata for slow tests, and a rule that new tests can't be added to quarantine directly.

Trade-offs:

• Retries on failure (pytest-rerunfailures): they make CI green quickly but hide real concurrency and timing bugs, including ones that happen in production. Acceptable only as a temporary, reported measure for quarantined tests, never globally.
• Test selection vs full runs: selection cuts PR time drastically but can miss indirect effects (settings, migrations, shared fixtures); keep full runs on main and before release.
• Fakes vs mocks vs real services: fakes cost effort to build and must be kept honest by contract tests; mocks are cheap but couple tests to implementation details; real sandboxes are realistic but slow and flaky. Use each at its own layer.
• Rewriting tests vs fixing infrastructure: parallelism and per-worker DBs give the biggest speed-up for the least code change, but only work once isolation is fixed; rewriting DB tests as unit tests is slower work with lasting benefit. Sequence: measure → isolation → parallelism → pyramid.
• Feature work vs test investment: a fixed budget (e.g. 20% of the team for the quarter) with visible metrics makes the trade-off explicit to product.

Deep-dive probes:

• "A test fails only when run with xdist, never alone. How do you debug it?" Look for: reproduce with the printed --randomly-seed, bisect the tests that ran earlier in the same worker (run that worker's test list in order and halve it until the failure needs only one predecessor), look for shared DB rows, module-level state, environment variables or files in a shared temp directory, and fix ownership of the state rather than pinning the order.
• "How would a property-based test have caught the refund rounding bug?" Look for: invariants such as "sum of partial refunds never exceeds the captured amount", "refund of the full amount returns exactly the captured paise", tested with Hypothesis over amounts, splits and GST rates, with shrinking producing a minimal failing case.
• "Developers say test selection let a broken migration reach main." Look for: always running migration and settings-related checks, mapping non-Python files (migrations, templates, settings) to test sets, and treating main's full run as the gate for the release train.
• "How do you convince the team to stop pressing re-run?" Look for: making flakes cheap to report (auto-detection and auto-quarantine with an owner), publishing the flake dashboard, and leading by fixing the top 10 flakes first to show the payoff.

Rubric: Strong: Starts from data (durations and flake rates), names the concrete Python flake sources (time, order, shared state, randomness, async timing, external sandboxes) and their fixes, sequences isolation before parallelism, reshapes the pyramid with pure logic and property-based tests for money, adds contract tests at integration boundaries, and puts guardrails and ownership in place. Acceptable: Proposes xdist, sharding and fixing flaky tests, but without a measurement step, without addressing why parallel runs will expose order dependence, or without a quarantine and ownership policy. Red flag: Recommends global automatic retries or skipping flaky tests, buying bigger CI machines as the main plan, or deleting slow tests without understanding what they protect.

SR-DE-C01 · Watermark-based dedup of an out-of-order event stream in bounded memory

Hard · Coding · Senior (5–8 years) · streaming, deduplication, watermarks, late data, bounded state, heapq, at-least-once delivery · 30 min

Problem:

A ride-hailing platform streams trip events from driver apps through Kafka into the warehouse. Delivery is at-least-once (retries after flaky mobile networks, consumer restarts), and events arrive out of order by up to a minute. Downstream billing needs every event_id emitted exactly once. The consumer runs for weeks, so it cannot remember every ID it has ever seen.

Implement WatermarkDeduper(allowed_lateness) with process(event) -> "emit" | "duplicate" | "late":

• The watermark is max event_time seen so far - allowed_lateness (None before the first event). Only events that are emitted can advance it.
• An event with event_time < watermark is "late": it goes to a side output for a reconciliation job, and is never emitted by this consumer.
• Otherwise, an event_id already emitted is a "duplicate"; a new one is "emit".
• Memory must stay bounded by the number of distinct events inside the lateness window, not by the length of the stream. Expose state_size (number of remembered IDs) and watermark.
• allowed_lateness < 0 raises ValueError.

A redelivered copy of an event carries the same event_time as the original.

Signature: class WatermarkDeduper: def __init__(self, allowed_lateness: int) -> None: ... def process(self, event: Event) -> Literal["emit", "duplicate", "late"]: ...

Constraints: event_time is integer epoch seconds (UTC). Up to ~20,000 events/s per partition, so process must be O(log n) or better. Standard library only. One deduper per partition; you may assume a single thread calls it.

Examples:

• d = WatermarkDeduper(60); d.process(Event('a', 1000)); d.process(Event('a', 1000)) → 'emit', then 'duplicate'
• d.process(Event('b', 990)); d.process(Event('c', 1100)) → 'emit' (out of order but within 60 s), 'emit'; watermark is now 1040
• d.process(Event('d', 1039)); d.process(Event('a', 1000)) → 'late', 'late' (the redelivered 'a' must not be emitted again, even though its ID was evicted)

Hints:

1. When is it safe to forget an ID? Think about what happens to a redelivered copy of it afterwards.
2. If you evict IDs older than the watermark, a copy of an evicted event is itself older than the watermark. Which check has to run first, so eviction can never cause a second emit?
3. To evict in O(log n), keep a min-heap of (event_time, event_id) next to the set of IDs and pop while the top is older than the watermark.

Reference solution:

import heapq
from dataclasses import dataclass
from typing import Literal

Verdict = Literal["emit", "duplicate", "late"]


@dataclass(frozen=True)
class Event:
    event_id: str
    event_time: int          # epoch seconds, UTC
    payload: str = ""


class WatermarkDeduper:
    """Exactly-once *output* for an at-least-once, out-of-order stream, in bounded memory.

    watermark = max event_time seen - allowed_lateness. Events older than the
    watermark are 'late' (routed to a side output); ids are remembered only while
    a copy of them could still be accepted.
    """

    def __init__(self, allowed_lateness: int) -> None:
        if allowed_lateness < 0:
            raise ValueError("allowed_lateness must be >= 0")
        self.allowed_lateness = allowed_lateness
        self._max_time: int | None = None
        self._seen: set[str] = set()
        self._expiry: list[tuple[int, str]] = []      # min-heap of (event_time, event_id)

    @property
    def watermark(self) -> int | None:
        return None if self._max_time is None else self._max_time - self.allowed_lateness

    @property
    def state_size(self) -> int:
        return len(self._seen)

    def process(self, event: Event) -> Verdict:
        wm = self.watermark
        if wm is not None and event.event_time < wm:
            return "late"                 # never consult state: it may have been evicted
        if event.event_id in self._seen:
            return "duplicate"
        self._seen.add(event.event_id)
        heapq.heappush(self._expiry, (event.event_time, event.event_id))
        if self._max_time is None or event.event_time > self._max_time:
            self._max_time = event.event_time
            self._evict()
        return "emit"

    def _evict(self) -> None:
        wm = self.watermark
        while self._expiry and self._expiry[0][0] < wm:
            _, event_id = heapq.heappop(self._expiry)
            self._seen.discard(event_id)

Complexity: process is O(log w) amortised, where w is the number of distinct events inside the lateness window (heap push, plus each ID is popped exactly once). Space is O(w): with 20,000 events/s and 60 s lateness, about 1.2 million IDs, roughly 200–250 MB of CPython strings, set slots and heap tuples (measured with tracemalloc), which is worth knowing before choosing in-process state.

Tests:

d = WatermarkDeduper(allowed_lateness=60)
assert d.watermark is None
assert d.process(Event("a", 1000)) == "emit"
assert d.process(Event("a", 1000)) == "duplicate"
assert d.process(Event("b", 990)) == "emit"            # out of order but within lateness
assert d.process(Event("c", 1100)) == "emit"           # watermark -> 1040
assert d.watermark == 1040
assert d.process(Event("d", 1039)) == "late"
assert d.process(Event("e", 1040)) == "emit"           # exactly at the watermark is on time
assert d.process(Event("e", 1040)) == "duplicate"
# a and b were evicted; a redelivery of 'a' must NOT be emitted a second time
assert d.state_size == 2, d.state_size
assert d.process(Event("a", 1000)) == "late"
# a late event must not move the watermark
assert d.watermark == 1040

# bounded memory on a long stream with redeliveries
d2 = WatermarkDeduper(allowed_lateness=30)
emitted = 0
max_state = 0
for t in range(10_000):
    for eid in (f"e{t}", f"e{t}", f"e{max(t - 5, 0)}"):    # dup + a redelivery 5 s later
        if d2.process(Event(eid, t if eid == f"e{t}" else max(t - 5, 0))) == "emit":
            emitted += 1
    max_state = max(max_state, d2.state_size)
assert emitted == 10_000, emitted
assert max_state <= 31, max_state

# redelivery long after eviction is late, not re-emitted
assert d2.process(Event("e10", 10)) == "late"

# zero lateness: anything older than the max is late
z = WatermarkDeduper(0)
assert [z.process(Event(i, t)) for i, t in [("x", 5), ("y", 4), ("z", 5), ("x", 5)]] == \
    ["emit", "late", "emit", "duplicate"]

try:
    WatermarkDeduper(-1)
except ValueError:
    pass
else:
    raise AssertionError("negative lateness accepted")

A common wrong solution (the tests catch it):

from dataclasses import dataclass
from typing import Literal

Verdict = Literal["emit", "duplicate", "late"]


@dataclass(frozen=True)
class Event:
    event_id: str
    event_time: int
    payload: str = ""


class WatermarkDeduper:
    def __init__(self, allowed_lateness: int) -> None:
        if allowed_lateness < 0:
            raise ValueError("allowed_lateness must be >= 0")
        self.allowed_lateness = allowed_lateness
        self._max_time: int | None = None
        self._seen: dict[str, int] = {}

    @property
    def watermark(self) -> int | None:
        return None if self._max_time is None else self._max_time - self.allowed_lateness

    @property
    def state_size(self) -> int:
        return len(self._seen)

    def process(self, event: Event) -> Verdict:
        if event.event_id in self._seen:
            return "duplicate"
        self._seen[event.event_id] = event.event_time
        if self._max_time is None or event.event_time > self._max_time:
            self._max_time = event.event_time
            wm = self.watermark
            self._seen = {k: t for k, t in self._seen.items() if t >= wm}   # bounded state
        return "emit"

Alternatives and tradeoffs:

• Rebuild the set on every watermark advance (as the broken version does): correct eviction but O(w) per event; it falls over at real rates.
• Time-bucketed sets (one set per second or minute, drop whole buckets past the watermark): O(1) per event and cheaper eviction, at the cost of coarser memory bounds. This is what most production implementations do.
• Bloom filter per bucket: far less memory, but false positives mean dropping real events, which billing can't accept without a reconciliation path.
• Push it to the sink: an idempotent MERGE/upsert on event_id in the warehouse makes the stream consumer's dedup a cost optimisation rather than a correctness requirement. Senior answers usually do both: bounded in-stream dedup plus an idempotent sink.
• Frameworks (Flink keyed state with TTL, Spark dropDuplicatesWithinWatermark) implement the same contract; the question is whether you understand what they guarantee.

Follow-ups:

1. "The consumer crashes and restarts from the last committed Kafka offset. What breaks, and how do you fix it?" In-memory state is lost, so events between the committed offset and the crash are re-read and re-emitted: duplicates downstream. Either checkpoint the deduper state atomically together with the offset (what Flink does), or rebuild it on startup by replaying allowed_lateness worth of input before emitting, or rely on an idempotent sink keyed on event_id.
2. "One driver's phone has a clock set to 2030. What happens to the watermark, and how do you defend against it?" Its event jumps max_time to 2030, the watermark leaps forward and every normal event becomes 'late'. Use server-side ingestion time for the watermark, or clamp event_time to ingest_time + small skew and quarantine outliers, and compute the watermark from a low percentile across partitions rather than a single max.
3. "Billing asks for an SLA: what fraction of events are 'late' and what happens to them?" Emit a metric for late counts and lateness distribution per partition, alert above a threshold, and land late events in a side table that a daily reconciliation job merges idempotently into billing. Tune allowed_lateness from the measured p99.9 delay, trading memory and output latency for fewer late events.

Scoring: Strong: Checks lateness before the ID lookup and explains why that ordering is what makes eviction safe; heap or bucketed eviction with bounded state; the watermark only moves forward on accepted events; discusses restart semantics and clock skew without being led. Acceptable: Correct dedup with a watermark and eviction, but evicts by rescanning the dict or needs the hint to see that an evicted ID can be re-emitted. Red flag: Keeps every ID forever, keeps the 'last N IDs' regardless of time, or says Kafka's exactly-once makes consumer-side dedup unnecessary.

SR-DE-C02 · A CI gate that flags breaking changes in a table's data contract

Medium · Coding · Senior (5–8 years) · data contracts, schema evolution, compatibility, recursion, dataclasses, CI gates · 25 min

Problem:

Your team owns the orders table that 14 downstream teams (finance, fraud, growth dashboards) read. Producers propose schema changes in pull requests; you want a CI check that blocks changes that would break existing consumers, and lets everything else through without a meeting.

Implement breaking_changes(current, proposed). A consumer was written against current and will now read data produced with proposed. Report, as messages sorted by field path:

• "<path>: removed" for a field in current that is missing from proposed.
• "<path>: type <cur> -> <new> is not safe" when values of the proposed type cannot always be read as the current type. A type is safely readable as another if it is the same, or reachable through these lossless steps (transitively): int16 → int32 → int64, int32 → float64, float32 → float64, date → timestamp.
• "<path>: became nullable" when a non-null field becomes nullable (consumers don't handle nulls they've never seen).
• Nested structs (a Field whose type is itself a schema dict) are checked recursively with dotted paths, e.g. address.pincode. Swapping a struct for a primitive or vice versa is "<path>: struct/primitive mismatch".

New fields, tightening nullable → non-null, and narrowing a type (e.g. int64 → int32) are not breaking. Don't mutate inputs.

Signature: def breaking_changes(current: dict[str, Field], proposed: dict[str, Field]) -> list[str]:

Constraints: Schemas have up to a few hundred fields and nesting depth up to 5. Standard library only. Output must be deterministic (it is posted as a PR comment and diffed between runs). If a field has both a type and a nullability problem, report the type message first.

Examples:

• current {'qty': int32}, proposed {'qty': int64} → ['qty: type int32 -> int64 is not safe'] # consumers declared int32
• current {'amount_paise': int64}, proposed {'amount_paise': int32, 'gst_paise': int64} → [] # narrowing and additions are fine
• current {'address': {pincode: string}}, proposed {'address': {pincode: int32}} → ['address.pincode: type string -> int32 is not safe']

Hints:

1. Whose reader matters? Phrase each rule as: can a value written with the proposed type be read as the current type?
2. Precompute the transitive closure of the safe-read pairs once, instead of walking the graph per field.
3. For nested structs, recurse with a path prefix and merge the results before one final sort.

Reference solution:

from __future__ import annotations

from dataclasses import dataclass
from typing import Union

Schema = dict[str, "Field"]


@dataclass(frozen=True)
class Field:
    type: Union[str, Schema]      # a primitive name, or a nested struct schema
    nullable: bool = False


# (narrow, wide): a value of `narrow` can always be read as `wide` without loss.
_SAFE_READS = {
    ("int16", "int32"), ("int32", "int64"), ("int32", "float64"),
    ("float32", "float64"), ("date", "timestamp"),
}


def _closure(pairs: set[tuple[str, str]]) -> set[tuple[str, str]]:
    closed = set(pairs)
    while True:
        extra = {(a, d) for a, b in closed for c, d in closed if b == c} - closed
        if not extra:
            return closed
        closed |= extra


SAFE_READS = _closure(_SAFE_READS)


def breaking_changes(current: Schema, proposed: Schema, _prefix: str = "") -> list[str]:
    """Changes in `proposed` that would break consumers written against `current`.

    Consumers read data produced with `proposed` using `current`'s expectations, so a
    proposed type must be readable *as* the current type, and a non-null column must
    stay non-null. Returns messages sorted by field path.
    """
    found: list[tuple[str, int, str]] = []
    for name, cur in current.items():
        path = f"{_prefix}{name}"
        new = proposed.get(name)
        if new is None:
            found.append((path, 0, f"{path}: removed"))
            continue
        cur_struct, new_struct = isinstance(cur.type, dict), isinstance(new.type, dict)
        if cur_struct and new_struct:
            for msg in breaking_changes(cur.type, new.type, f"{path}."):
                found.append((msg.split(":", 1)[0], 0, msg))
        elif cur_struct or new_struct:
            found.append((path, 0, f"{path}: struct/primitive mismatch"))
        elif cur.type != new.type and (new.type, cur.type) not in SAFE_READS:
            found.append((path, 0, f"{path}: type {cur.type} -> {new.type} is not safe"))
        if new.nullable and not cur.nullable:
            found.append((path, 1, f"{path}: became nullable"))
    return [msg for _, _, msg in sorted(found)]

Complexity: O(F) field comparisons for F fields across all nesting levels, plus O(V log V) to sort V violations. The closure of the safe-read table is computed once at import; it is tiny, so the naive fixed-point loop is fine.

Tests:

current = {
    "order_id": Field("int64"),
    "amount_paise": Field("int64"),
    "qty": Field("int32"),
    "coupon": Field("string", nullable=True),
    "placed_on": Field("timestamp"),
    "address": Field({"pincode": Field("string"), "city": Field("string", nullable=True)}),
}

assert breaking_changes(current, current) == []

# Additions, narrowing a type the consumer reads, and tightening nullability are all safe.
safe = dict(current,
            gst_paise=Field("int64"),                         # new column
            amount_paise=Field("int32"),                      # int32 read as int64: fine
            placed_on=Field("date"),                          # date read as timestamp: fine
            coupon=Field("string", nullable=False))           # never-null is still valid
assert breaking_changes(current, safe) == [], breaking_changes(current, safe)

# Transitive safe read: int16 values read as int64.
assert breaking_changes({"x": Field("int64")}, {"x": Field("int16")}) == []

# Widening what the producer writes is the classic mistake: consumers expect int32.
assert breaking_changes(current, dict(current, qty=Field("int64"))) == \
    ["qty: type int32 -> int64 is not safe"]
# int64 -> float64 loses precision above 2**53, so it is not a safe read either way.
assert breaking_changes({"id": Field("float64")}, {"id": Field("int64")}) == \
    ["id: type float64 -> int64 is not safe"]
assert breaking_changes({"id": Field("int64")}, {"id": Field("float64")}) == \
    ["id: type int64 -> float64 is not safe"]

bad = {
    "amount_paise": Field("float64", nullable=True),          # type and nullability
    "qty": Field("int32"),
    "coupon": Field("string", nullable=True),
    "placed_on": Field("string"),
    "address": Field({"pincode": Field("int32"), "city": Field("string", nullable=True),
                      "landmark": Field("string", nullable=True)}),
    "customer_ref": Field("string"),                          # order_id was "renamed"
}
assert breaking_changes(current, bad) == [
    "address.pincode: type string -> int32 is not safe",
    "amount_paise: type int64 -> float64 is not safe",
    "amount_paise: became nullable",
    "order_id: removed",
    "placed_on: type timestamp -> string is not safe",
], breaking_changes(current, bad)

# Nested removals and struct/primitive swaps.
assert breaking_changes(current, dict(current, address=Field({"pincode": Field("string")}))) == \
    ["address.city: removed"]
assert breaking_changes(current, dict(current, address=Field("string"))) == \
    ["address: struct/primitive mismatch"]
# Inputs are not mutated.
assert set(current) == {"order_id", "amount_paise", "qty", "coupon", "placed_on", "address"}

A common wrong solution (the tests catch it):

from __future__ import annotations

from dataclasses import dataclass
from typing import Union

Schema = dict[str, "Field"]


@dataclass(frozen=True)
class Field:
    type: Union[str, Schema]      # a primitive name, or a nested struct schema
    nullable: bool = False


# (narrow, wide): a value of `narrow` can always be read as `wide` without loss.
_SAFE_READS = {
    ("int16", "int32"), ("int32", "int64"), ("int32", "float64"),
    ("float32", "float64"), ("date", "timestamp"),
}


def _closure(pairs: set[tuple[str, str]]) -> set[tuple[str, str]]:
    closed = set(pairs)
    while True:
        extra = {(a, d) for a, b in closed for c, d in closed if b == c} - closed
        if not extra:
            return closed
        closed |= extra


SAFE_READS = _closure(_SAFE_READS)


def breaking_changes(current: Schema, proposed: Schema, _prefix: str = "") -> list[str]:
    """Changes in `proposed` that would break consumers written against `current`.

    Consumers read data produced with `proposed` using `current`'s expectations, so a
    proposed type must be readable *as* the current type, and a non-null column must
    stay non-null. Returns messages sorted by field path.
    """
    found: list[tuple[str, int, str]] = []
    for name, cur in current.items():
        path = f"{_prefix}{name}"
        new = proposed.get(name)
        if new is None:
            found.append((path, 0, f"{path}: removed"))
            continue
        cur_struct, new_struct = isinstance(cur.type, dict), isinstance(new.type, dict)
        if cur_struct and new_struct:
            for msg in breaking_changes(cur.type, new.type, f"{path}."):
                found.append((msg.split(":", 1)[0], 0, msg))
        elif cur_struct or new_struct:
            found.append((path, 0, f"{path}: struct/primitive mismatch"))
        elif cur.type != new.type and (cur.type, new.type) not in SAFE_READS:  # producer may widen
            found.append((path, 0, f"{path}: type {cur.type} -> {new.type} is not safe"))
        if new.nullable and not cur.nullable:
            found.append((path, 1, f"{path}: became nullable"))
    return [msg for _, _, msg in sorted(found)]

Alternatives and tradeoffs:

• Use the format's own rules: Avro and Protobuf have well-defined compatibility modes (Confluent Schema Registry's BACKWARD / FORWARD / FULL). If your contracts are Avro or Protobuf, enforce them in the registry rather than reimplementing; this checker is the right tool for warehouse/Parquet tables and pandas/pydantic contracts that have no registry.
• Contract tests from consumers (consumer-driven contracts): each consumer declares the fields and types it actually uses; a removal of an unused field is then not breaking. More precise, more process.
• Semantic checks the type system can't see (amount switching from rupees to paise, a timezone change) need a versioned contract with owners and a changelog; no schema diff catches them.

Follow-ups:

1. "This checker says the change is breaking. The producer team says they really need to change qty to int64. How do you ship it?" Expand–contract: add qty_v2 int64 alongside qty, dual-write (fail or clamp with an alert when a value overflows int32), migrate consumers with a deadline and usage tracking (query logs), then deprecate and remove qty in a later release. Or publish a new versioned table/topic and run both for a migration window.
2. "Why is int64 → float64 not in the safe list, when every database will happily cast it?" float64 has a 53-bit mantissa, so integers above 2**53 are rounded. For IDs and paise amounts this silently corrupts values (and can create false join matches). A cast that succeeds is not the same as a lossless read.
3. "Where else would you run this check besides CI?" At ingestion time: compare each incoming batch's inferred schema with the contract and quarantine the batch on a breaking diff, because producers don't always go through your CI (a vendor file, an upstream team's hotfix). And in a nightly job against the actual warehouse table to catch drift made by hand.

Scoring: Strong: Gets the direction right (proposed must be readable as current) and explains why from the consumer's point of view, precomputes the transitive closure, handles nesting with dotted paths, deterministic ordering, and connects int64→float64 to precision loss. Proposes expand–contract for the necessary breaking change. Acceptable: Correct flat-schema rules after clarifying the direction; nesting or transitivity added with a hint. Red flag: Treats any 'widening' as safe without asking who reads the data, or proposes skipping the check and 'telling consumers on Slack'.

SR-DE-D01 · Change data capture from sharded Postgres into the lakehouse

Hard · Design · Senior (5–8 years) · CDC, Debezium, Kafka, ordering, deletes, upserts, schema evolution, backfills, pyarrow, Iceberg · 45 min

Scenario:

A quick-commerce company in India runs its order system on 8 PostgreSQL shards (sharded by store_id). Analytics reads from a nightly pg_dump-and-load job, so the ops dashboard ("orders stuck in packing for more than 10 minutes, per dark store") is always yesterday's news. Leadership wants the orders and order_items tables in the lakehouse (Parquet on object storage, queried by Trino/Spark) with freshness under 5 minutes.

Numbers: ~4 million orders/day, each order updated ~8 times as it moves through statuses (placed → packed → picked up → delivered/cancelled), so ~35 million row changes/day for orders and ~3× that for items; peaks at 3× average between 19:00 and 22:00 IST. Rows are also hard-deleted by a GDPR erasure job. Developers add columns to these tables about twice a month.

Your team writes Python; there is a managed Kafka cluster. Design the pipeline from the Postgres write-ahead log to queryable tables, including the initial load and how you would rebuild a table after a bug.

Clarifying questions:

• What does 'current' mean for consumers: the latest state of each order, the full change history, or both? (Both: a current-state table for dashboards, a history table for the data science team's funnel analysis.)
• Are order IDs globally unique across shards, or only per shard? (Per shard; the key is (shard_id, order_id).)
• Can we run Debezium (Kafka Connect) against the shards, and do we have logical replication slots available? (Yes; DBAs are nervous about slot lag filling disks.)
• Which table format does the lakehouse support: plain Parquet, Iceberg, Delta, Hudi? (Iceberg is available via the catalogue; Trino reads it.)
• How must hard deletes appear downstream: removed from current state, and also scrubbed from history for GDPR? (Yes to both, within 30 days.)
• What freshness and correctness do the consumers need: is 5-minute eventual consistency OK, and is any number financial? (Dashboards: yes. Finance still uses the nightly reconciled batch.)
• Who owns the source schema, and is there any process when a column is renamed or its type changes?

Reference design:

1. Capture: Debezium per shard → Kafka. One connector per shard reads the logical replication slot (pgoutput) and publishes change events to cdc.orders and cdc.order_items, keyed by the primary key (shard_id, order_id), so all changes to one row land on one partition in commit order. Each event carries op (c/u/d/r for snapshot), before/after, the source LSN and transaction id, and commit timestamp. Use Avro or Protobuf with a schema registry; deletes produce a delete event plus a tombstone. Alert on replication-slot lag (bytes and seconds) so a stuck consumer doesn't fill the primary's disk.

2. Land raw, append-only (bronze). A Python consumer group (confluent-kafka) reads micro-batches of ~1–2 minutes, converts them to Arrow record batches with an explicit schema (decimals as decimal128 paise or int64, timestamps as UTC timestamp[us, tz=UTC], never inferred), and appends them to an Iceberg orders_changes table partitioned by commit date. Commit Kafka offsets only after the Iceberg commit succeeds; on crash, the batch is re-read, so bronze writes are made idempotent by storing (topic, partition, offset_range) in the Iceberg snapshot summary and skipping a batch whose range is already committed. This table is the replayable source of truth for everything downstream.

3. Apply to current state (silver). Every few minutes a job takes the new bronze rows, keeps the latest change per key by (LSN, event order) (never by wall-clock updated_at, which ties and skews across shards), and runs an Iceberg MERGE INTO orders_current (Spark or Trino; or pyiceberg with copy-on-write for small partitions): updates and inserts upsert, op = d deletes. Because the merge only applies a change if its LSN is newer than the stored one, replays and out-of-order batches are harmless: the apply is idempotent and order-insensitive.

4. Initial load and rebuilds. Debezium's incremental snapshot (signal table, chunked by PK) emits op = r rows interleaved with live changes, with watermarking so a snapshot row never overwrites a newer live change. Rebuilding silver after a bug = truncate silver and replay bronze with the fixed code (same job, different range); no need to touch production Postgres.

5. Schema evolution. Additive, nullable columns are allowed automatically: the registry accepts them, the Python consumer maps by field name, and Iceberg's schema evolution adds the column. Renames, type changes and drops are blocked by registry compatibility rules (and a CI check on the migration PRs), and go through expand–contract. Unknown fields never shift positions because nothing is positional.

6. GDPR deletes. Current state handles deletes via MERGE. For history, run a weekly job that rewrites affected bronze/history files with the erased keys removed (Iceberg row-level delete + rewrite_data_files + snapshot expiry), and keep an erasure log proving it happened.

7. Observability and SLOs. End-to-end freshness per table (now - max(commit_ts) in silver), slot lag, consumer lag per partition, MERGE duration, rows changed per batch vs a seasonal baseline, and a row-count reconciliation per shard: hourly count(*) and sum of amount_paise from a replica vs silver, with an alert on drift. Small-file compaction runs hourly on the hot partitions.

Python-specific: one consumer process per few partitions (the work is decode + Arrow conversion, which is CPU-bound, so scale with processes or more consumer instances, not threads); bound memory per micro-batch by bytes, not message count; poll in a loop that keeps calling poll() during long writes (or uses pause()/resume()) so the group doesn't rebalance on max.poll.interval.ms.

Trade-offs:

• Debezium + Kafka vs a managed CDC service (DMS, Datastream, Fivetran). Managed is faster to start and fewer moving parts; Debezium gives control over snapshots, ordering, schema registry and cost at 100M+ changes/day. With an existing Kafka team, Debezium; without one, managed.
• MERGE on every micro-batch vs merge-on-read. Copy-on-write MERGE gives fast reads but rewrites files every few minutes (write amplification, small files). Merge-on-read (Iceberg position/equality deletes) makes writes cheap and reads slower until compaction. For a 5-minute SLO with heavy updates, merge-on-read plus frequent compaction of hot partitions is the usual answer.
• Ordering by LSN vs updated_at. LSN is exact within a shard; across shards it doesn't matter because keys never move between shards. updated_at is set by application clocks and is not unique, so ties pick the wrong winner.
• Exactly-once Kafka transactions vs idempotent sinks. Kafka EOS doesn't cover an external sink like Iceberg; an idempotent apply keyed on LSN is simpler and survives replays from any point.
• One pipeline per table vs a generic framework. Two tables today; twenty next year. A small config-driven framework (table, key, partitioning, contract) pays off after the third table, not the first.

Deep-dive probes:

• "An order goes placed → cancelled → (support reverts) placed again, all within one micro-batch, and the batch arrives after a replay of an older batch." Look for: dedup to the latest change per key by LSN within the batch, and a MERGE condition WHEN MATCHED AND src.lsn > tgt.lsn, so the older replay can't regress state.
• "The consumer was down for 6 hours during the IPL final." Look for: Kafka retention well beyond the worst outage (e.g. 7 days), catch-up with larger batches and more consumer instances, slot lag protected because Debezium kept running, freshness SLO breach visible on the dashboard, and no special code path: catch-up is the same idempotent job.
• "A developer changed amount from numeric(10,2) to numeric(12,2) and another renamed status to order_status." Look for: widening the decimal is compatible if the Arrow/Iceberg type is widened too (registry permits it, Iceberg supports decimal precision widening); the rename is blocked and must become add-new, dual-write, backfill, deprecate.
• "Silver has 0.3% fewer orders than Postgres for one shard." Look for: per-shard reconciliation narrowing it down to time ranges, checking whether deletes or snapshot/stream watermark handling dropped rows, replaying bronze for that range, and a test fixture reproducing the interleaving.
• "The consumer's memory climbs until it is OOM-killed at peak." Look for: batches bounded by bytes, Arrow buffers released between batches, not accumulating Python dicts per message, and measuring with tracemalloc or by watching RSS per batch size.

Rubric: Strong: Log-based CDC keyed by primary key with LSN ordering; an immutable bronze layer that makes rebuilds a replay; an idempotent, order-insensitive MERGE with deletes; snapshot/stream interleaving handled; schema evolution gated by a registry with expand–contract for breaking changes; GDPR erasure in history; concrete freshness and reconciliation metrics. Explicit about merge-on-read vs copy-on-write and small files. Acceptable: Debezium → Kafka → periodic MERGE on the primary key with sensible monitoring, but orders changes by updated_at, or needs prompting on deletes, replay idempotency or the initial snapshot. Red flag: Polls WHERE updated_at > last_run on the primary (misses deletes, hammers the DB, loses same-timestamp rows), appends changes without a current-state table, or assumes exactly-once delivery end to end.

SR-DE-O01 · A left join turns 64-bit merchant IDs into floats and doubles a payout

Medium & Tricky · Output · Senior (5–8 years) · pandas merge, dtypes, int64 to float64, data contracts, join fan-out, reconciliation · 8 min

Question: A nightly refund-settlement job in a payments company enriches orders with refunds, then joins to the merchant table to find the settlement account. Merchant IDs are 64-bit integers generated by a Snowflake-style ID service. The job has run fine for months; on Monday finance reports one refund paid out to two different banks. What does this print?

import pandas as pd

orders = pd.DataFrame({"order_id": [101, 102, 103]})
refunds = pd.DataFrame({
    "order_id": [101, 103],
    "merchant_id": [9007199254740993, 42],
})
merchants = pd.DataFrame({
    "merchant_id": [9007199254740992, 9007199254740993, 42],
    "settlement_acct": ["HDFC-01", "ICICI-07", "SBI-11"],
})

report = orders.merge(refunds, on="order_id", how="left")
print(report["merchant_id"].dtype, report["merchant_id"].tolist())

payout = report.merge(merchants, on="merchant_id", how="left")
print(len(report), "->", len(payout))
print(payout[["order_id", "settlement_acct"]].to_string(index=False))

Expected output:

float64 [9007199254740992.0, nan, 42.0]
3 -> 4
 order_id settlement_acct
      101         HDFC-01
      101        ICICI-07
      102             NaN
      103          SBI-11

The trap: "int64 [9007199254740993, nan, 42], then 3 -> 3, with order 101 paid to ICICI-07." Candidates know a left join can introduce missing values but picture them as "an int column with a null in it". Others say float64 but assume the value survives (9007199254740993.0) because the number "looks" exact, and nobody expects a many-to-one lookup to add rows.

Explanation:

Step 1, the dtype change. Order 102 has no refund, so the left join must put a missing value in merchant_id. Classic NumPy int64 has no missing value, so pandas upcasts the whole column to float64 and uses NaN. A float64 has a 53-bit mantissa: every integer above 2⁵³ = 9,007,199,254,740,992 can no longer be represented exactly. 9007199254740993 (2⁵³ + 1) rounds to the nearest even representable value, 9007199254740992. Nothing warns; the column just holds a different number.

Step 2, the fan-out. Joining a float64 key against the int64 key in merchants, pandas brings both sides to a common dtype, so the merchant IDs …992 and …993 both become 9007199254740992.0. The refund for order 101 now matches two merchant rows, and the "lookup" join grows from 3 to 4 rows. Order 101 is paid to HDFC-01 (a merchant who never sold it) and to ICICI-07. Order 103 (ID 42) is fine because small integers are exact in float64, which is why the job "ran fine for months": it only broke once the ID service crossed 2⁵³.

Real-world incident: a settlement or reconciliation job doubles a payout, or silently attributes events to the wrong entity, after an upstream ID space grows. The same thing happens when IDs pass through JSON in JavaScript, through read_csv with a blank cell, or through fillna on a numeric column.

Fixes, in order of strength: - Make it a data contract: IDs are identifiers, not numbers. Carry them as string, or as nullable Int64 / int64[pyarrow], which keep missing values without converting to float (refunds.astype({"merchant_id": "Int64"}) gives [9007199254740993, <NA>, 42] after the join and the payout row count stays 3). - Assert row counts and key dtypes at every join boundary in the pipeline (assert len(payout) == len(report)), and fail the run rather than publish. - Reconcile totals (sum of refunds in == sum of payouts out) before money moves.

Takeaway: a missing value in an int64 column silently turns it into floats, and above 2⁵³ that rewrites your IDs; treat identifiers as strings or nullable integers and assert join cardinality.

Follow-ups:

1. "Would validate="many_to_one" on the second merge have caught this?" No. In pandas 2.2 it checks that the right keys are unique, and merchants.merchant_id is unique as int64 (…992 and …993 differ), so it passes; the collision only appears after the cross-dtype cast, and the result still has 4 rows. That is why a post-join row-count assertion is the guard you actually want, alongside validate.
2. "Where else in a Python data stack does this exact bug appear?" read_csv on a column with any blank cell (float64 by default), json.loads is fine in Python but the same payload in a browser or Node loses precision, Parquet written from a float-inferred column, df.loc[mask, 'id'] = np.nan, and pd.concat of a frame where the ID column is missing. Pandas' dtype_backend="numpy_nullable" / "pyarrow" on readers avoids most of them.
3. "You own this pipeline. What do you change so the next dtype drift fails loudly instead of paying money?" A schema contract checked at ingestion and before publish (column names, dtypes, nullability, key uniqueness), row-count and sum reconciliation between stages, an alert on any join whose output count differs from its left input on a declared many-to-one join, and a quarantine path. Plus a regression test with an ID above 2⁵³ in the fixture data.

Scoring: Strong: Exact output, including the rounding to …992, the 3 -> 4 fan-out and the double payout. Explains the 53-bit mantissa and why it only appeared once IDs grew. Proposes nullable/string IDs as a contract plus join-cardinality assertions, and knows validate= alone does not save you. Acceptable: Predicts float64 and NaN, but needs a prompt to see the precision loss; once told the value changed, reasons to the duplicate match and suggests Int64 or string IDs. Red flag: Expects int64 with NaN, or treats the 4th row as harmless and suggests drop_duplicates() after the join, which would hide which account was wrong.

SR-DS-C01 · Bootstrap CI for revenue per session, resampling users not sessions

Medium · Coding · Senior (5–8 years) · experimentation, bootstrap, ratio metrics, clustered data, numpy vectorisation, reproducibility · 25 min

Problem:

A food-delivery app runs A/B tests on its home feed. The headline metric is revenue per session = total revenue / total sessions in an arm. Randomisation is per user, and a user has many sessions. An analyst's notebook bootstraps the metric by resampling sessions, and its intervals look reassuringly tight; you suspect they are far too narrow.

Write bootstrap_ratio_ci(user_ids, numerator, denominator, *, n_boot=2000, alpha=0.05, seed=0, max_cells=5_000_000) that returns RatioCI(point, low, high):

• point is the ratio of sums over all rows (not the mean of per-row ratios).
• The CI is a percentile bootstrap where each replicate resamples users with replacement, carrying all of a user's rows with them.
• Reproducible: the same inputs and seed give exactly the same result, and the order of rows must not matter.
• Vectorised with NumPy; memory for the resampling indices must stay around max_cells integers regardless of n_boot.
• Raise ValueError for mismatched lengths, fewer than 2 users, a user with a non-positive total denominator, or invalid alpha/n_boot.

Signature: def bootstrap_ratio_ci(user_ids, numerator, denominator, *, n_boot: int = 2000, alpha: float = 0.05, seed: int = 0, max_cells: int = 5_000_000) -> RatioCI:

Constraints: Up to 2 million users and 30 million session rows per arm. NumPy only (no SciPy/statsmodels). Use np.random.default_rng(seed), never the global RNG.

Examples:

• users = ['asha']*100 + ['ravi']*100; revenue = [10.0]*100 + [0.0]*100; sessions = [1]*200 → RatioCI(point=5.0, low=0.0, high=10.0) # only 2 independent units of evidence
• same data, resampling sessions instead → roughly (4.3, 5.7): treats 200 correlated sessions as 200 independent observations

Hints:

1. What is the unit of randomisation in this experiment? That's what each bootstrap draw should sample.
2. Aggregate to one row per user first (np.unique(..., return_inverse=True) + np.bincount(..., weights=...)); then a replicate is just a draw of user indices.
3. Draw a (k, n_users) matrix of indices per chunk, with k chosen from max_cells, and compute num_u[idx].sum(1) / den_u[idx].sum(1).

Reference solution:

from typing import NamedTuple

import numpy as np


class RatioCI(NamedTuple):
    point: float
    low: float
    high: float


def bootstrap_ratio_ci(user_ids, numerator, denominator, *, n_boot: int = 2000,
                       alpha: float = 0.05, seed: int = 0,
                       max_cells: int = 5_000_000) -> RatioCI:
    """Percentile CI for sum(numerator) / sum(denominator), resampling USERS, not rows.

    Rows (sessions, orders) from one user are correlated; the user is the unit of
    randomisation, so the user is the unit of resampling.
    """
    user_ids = np.asarray(user_ids)
    num = np.asarray(numerator, dtype=float)
    den = np.asarray(denominator, dtype=float)
    if not (len(user_ids) == len(num) == len(den)):
        raise ValueError("user_ids, numerator and denominator must have the same length")
    if not 0 < alpha < 1 or n_boot < 1:
        raise ValueError("need 0 < alpha < 1 and n_boot >= 1")

    _, inv = np.unique(user_ids, return_inverse=True)
    num_u = np.bincount(inv, weights=num)
    den_u = np.bincount(inv, weights=den)
    n_users = len(num_u)
    if n_users < 2:
        raise ValueError("need at least 2 users to bootstrap")
    if np.any(den_u <= 0):
        raise ValueError("every user needs a positive denominator")

    rng = np.random.default_rng(seed)
    chunk = max(1, max_cells // n_users)          # bound memory: chunk x n_users indices
    ratios = np.empty(n_boot)
    for start in range(0, n_boot, chunk):
        k = min(chunk, n_boot - start)
        idx = rng.integers(0, n_users, size=(k, n_users))
        ratios[start:start + k] = num_u[idx].sum(axis=1) / den_u[idx].sum(axis=1)

    low, high = np.quantile(ratios, [alpha / 2, 1 - alpha / 2])
    return RatioCI(float(num_u.sum() / den_u.sum()), float(low), float(high))

Complexity: O(R) to aggregate R rows to U users, then O(n_boot × U) time for the replicates, done in chunks so peak extra memory is O(max_cells): the int64 index matrix plus the two gathered float arrays, about 120 MB at the default, plus O(n_boot) for the replicate ratios. At U = 2 million and n_boot = 2000 that's 4 billion index draws: minutes, not hours, and a good reason to move to the delta method (see alternatives) for routine dashboards.

Tests:

import numpy as np

# Two users, 100 sessions each: Asha spends 10 every session, Ravi never buys.
users = np.array(["asha"] * 100 + ["ravi"] * 100)
revenue = np.array([10.0] * 100 + [0.0] * 100)
sessions = np.ones(200)
ci = bootstrap_ratio_ci(users, revenue, sessions, n_boot=4000, seed=7)
assert ci.point == 5.0
# Resampling users: each replicate is {asha, asha}, {asha, ravi} or {ravi, ravi}.
assert (ci.low, ci.high) == (0.0, 10.0), ci

# Same seed -> identical result; row order does not matter.
again = bootstrap_ratio_ci(users, revenue, sessions, n_boot=4000, seed=7)
assert again == ci
perm = np.random.default_rng(1).permutation(200)
assert bootstrap_ratio_ci(users[perm], revenue[perm], sessions[perm], n_boot=4000, seed=7) == ci

# No between-user variation -> a zero-width interval at the point estimate.
flat = bootstrap_ratio_ci([1, 1, 2, 2, 3], [4, 4, 4, 4, 4], [2, 2, 2, 2, 2], n_boot=300)
assert flat == (2.0, 2.0, 2.0), flat

# Realistic sanity check, and chunking (max_cells forces many small chunks) gives the same
# answer as one big chunk because the RNG stream is consumed in the same order.
rng = np.random.default_rng(42)
n_sessions = rng.poisson(5, size=400) + 1
uid = np.repeat(np.arange(400), n_sessions)
rev = rng.gamma(2.0, 50.0, size=uid.size) * (rng.random(uid.size) < 0.3)
big = bootstrap_ratio_ci(uid, rev, np.ones(uid.size), n_boot=1000, seed=3)
small = bootstrap_ratio_ci(uid, rev, np.ones(uid.size), n_boot=1000, seed=3, max_cells=1000)
assert big.low < big.point < big.high
assert np.allclose(big, small)
# Ratio of sums, not mean of per-row ratios.
assert np.isclose(big.point, rev.sum() / uid.size)

for bad in [dict(user_ids=[1, 2], numerator=[1.0], denominator=[1.0, 1.0]),
            dict(user_ids=[1, 1], numerator=[1.0, 2.0], denominator=[1.0, 1.0]),
            dict(user_ids=[1, 2], numerator=[1.0, 2.0], denominator=[0.0, 1.0])]:
    try:
        bootstrap_ratio_ci(**bad)
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted {bad}")

A common wrong solution (the tests catch it):

from typing import NamedTuple

import numpy as np


class RatioCI(NamedTuple):
    point: float
    low: float
    high: float


def bootstrap_ratio_ci(user_ids, numerator, denominator, *, n_boot: int = 2000,
                       alpha: float = 0.05, seed: int = 0,
                       max_cells: int = 5_000_000) -> RatioCI:
    user_ids = np.asarray(user_ids)
    num = np.asarray(numerator, dtype=float)
    den = np.asarray(denominator, dtype=float)
    if not (len(user_ids) == len(num) == len(den)):
        raise ValueError("user_ids, numerator and denominator must have the same length")
    if len(np.unique(user_ids)) < 2 or np.any(den <= 0):
        raise ValueError("bad input")
    rng = np.random.default_rng(seed)
    n = len(num)
    chunk = max(1, max_cells // n)
    ratios = np.empty(n_boot)
    for start in range(0, n_boot, chunk):          # resample rows (sessions)
        k = min(chunk, n_boot - start)
        idx = rng.integers(0, n, size=(k, n))
        ratios[start:start + k] = num[idx].sum(axis=1) / den[idx].sum(axis=1)
    low, high = np.quantile(ratios, [alpha / 2, 1 - alpha / 2])
    return RatioCI(float(num.sum() / den.sum()), float(low), float(high))

Alternatives and tradeoffs:

• Delta method (linearisation of the ratio with user-level sums): a closed-form standard error in O(U), the standard choice for an experimentation platform's daily metrics. Bootstrap stays useful for metrics without a clean variance formula (quantiles, trimmed means) and to validate the delta method.
• Poisson bootstrap: give each user a Poisson(1) weight per replicate instead of drawing indices. Parallelises and streams (each user is processed independently), which is how large platforms bootstrap over distributed data.
• Mean of per-user ratios answers a different question (the average user's revenue per session); pick the estimand with the product team before choosing the method.
• Row-level (session) bootstrap is only valid if sessions are independent, which they almost never are when randomisation is per user.

Follow-ups:

1. "How would you show a sceptical PM that the analyst's session-level intervals are wrong, without a statistics lecture?" Run an A/A test (or simulate many A/A splits of historical data): with session-level CIs, far more than 5% of A/A comparisons come out 'significant'. With user-level resampling the false-positive rate is close to the nominal 5%. A calibration plot of A/A p-values makes it obvious.
2. "You need the CI of the difference between treatment and control. How do you extend this?" Resample users independently within each arm (the arms are independent samples), compute the ratio in each, and take the percentile interval of the difference (or the relative lift). Use one Generator per arm seeded from a SeedSequence.spawn so results are reproducible and streams don't overlap.
3. "A few whale users (bulk caterers) dominate revenue. What happens to the bootstrap and what do you do?" The replicate distribution becomes lumpy and the percentile CI unreliable with few whales. Options: cap/winsorise revenue per user at a pre-registered percentile, analyse them as a separate stratum, or use CUPED with pre-period revenue to remove much of the per-user variance. Decide before looking at results to avoid p-hacking.

Scoring: Strong: Resamples users and can explain why in terms of the unit of randomisation and correlated sessions; aggregates to per-user sums, vectorised replicates in memory-bounded chunks, seeded Generator, ratio of sums. Brings up the delta method and A/A validation without prompting. Acceptable: Correct user-level bootstrap with a Python loop over replicates or unbounded memory; explains the clustering issue once asked about the unit of randomisation. Red flag: Resamples sessions, averages per-session ratios, or uses np.random.seed global state in a library function.

SR-DS-C02 · CUPED variance reduction that survives an unlucky randomisation

Hard · Coding · Senior (5–8 years) · experimentation, CUPED, variance reduction, covariates, numpy, missing data, statistics in code · 30 min

Problem:

Your experimentation library reports A/B results for a grocery app. Tests on order value per user need weeks to reach significance because spend varies hugely between users. You will add CUPED: adjust each user's in-experiment metric y using the same metric from the pre-experiment period x.

Implement cuped(y, x, treated) -> CupedResult where:

• theta = cov(x, y) / var(x), computed on all users pooled (both arms), ddof=1.
• adjusted = y - theta * (x - mean(x)), with mean(x) also pooled.
• effect and se are the difference in means of adjusted (treatment − control) and its standard error sqrt(var_t/n_t + var_c/n_c); also return the unadjusted naive_effect and naive_se, plus theta and the adjusted array (input order).
• New users have no history: x is NaN for them. Impute with the pooled mean of observed x, so they are left unadjusted. Do not modify the caller's arrays.
• If x has zero variance, theta = 0.
• Raise ValueError for shape mismatches, NaN in y, or an arm with fewer than 2 users.

Signature: def cuped(y, x, treated) -> CupedResult:

Constraints: Up to 10 million users per experiment; NumPy only, fully vectorised (no Python loops over users). The result must be identical on reruns: no randomness inside the function.

Examples:

• 20,000 users; y = 0.8·x + noise + 2·treated; treated users happened to have x about 5 higher → naive_effect ≈ 6 (biased by the pre-period imbalance); effect ≈ 2; se well under half of naive_se; theta ≈ 0.8
• cuped([1, 2, 3, 4], [5, 5, 5, 5], [T, F, T, F]) → theta = 0.0 and effect == naive_effect == -1.0

Hints:

1. Why is it legitimate to use pre-period data at all? What property does x have that a post-period covariate wouldn't?
2. Think about what centring x within each arm would do to the difference in means. What happens to a chance pre-period imbalance?
3. np.array(x, dtype=float) makes a copy you can impute into; np.asarray may not.

Reference solution:

from dataclasses import dataclass

import numpy as np


@dataclass(frozen=True)
class CupedResult:
    effect: float              # CUPED difference in means (treatment - control)
    se: float
    naive_effect: float
    naive_se: float
    theta: float
    adjusted: np.ndarray       # per-user adjusted metric, same order as the input


def _diff_and_se(values: np.ndarray, treated: np.ndarray) -> tuple[float, float]:
    t, c = values[treated], values[~treated]
    se = np.sqrt(t.var(ddof=1) / t.size + c.var(ddof=1) / c.size)
    return float(t.mean() - c.mean()), float(se)


def cuped(y, x, treated) -> CupedResult:
    """CUPED: y_adj = y - theta * (x - mean(x)), with theta and mean(x) pooled over BOTH arms.

    x is the same metric in the pre-experiment period. NaN in x (new users with no
    history) is imputed with the pooled mean of observed x, i.e. those users are not adjusted.
    """
    y = np.asarray(y, dtype=float)
    x = np.array(x, dtype=float)              # copy: we impute in place below
    treated = np.asarray(treated, dtype=bool)
    if not (y.shape == x.shape == treated.shape) or y.ndim != 1:
        raise ValueError("y, x and treated must be 1-D arrays of the same length")
    if treated.sum() < 2 or (~treated).sum() < 2:
        raise ValueError("each arm needs at least 2 users")
    if np.isnan(y).any():
        raise ValueError("y must not contain NaN")

    observed = ~np.isnan(x)
    x_mean = x[observed].mean() if observed.any() else 0.0
    x[~observed] = x_mean

    var_x = x.var(ddof=1)
    theta = 0.0 if var_x == 0 else float(np.cov(x, y, ddof=1)[0, 1] / var_x)
    adjusted = y - theta * (x - x_mean)

    effect, se = _diff_and_se(adjusted, treated)
    naive_effect, naive_se = _diff_and_se(y, treated)
    return CupedResult(effect, se, naive_effect, naive_se, theta, adjusted)

Complexity: O(n) time and O(n) memory: a few passes over the arrays for means, variance and covariance, and one adjusted copy. No sorting, no resampling.

Tests:

import numpy as np

rng = np.random.default_rng(2026)
n = 20_000
treated = np.arange(n) % 2 == 0
# An unlucky randomisation: treated users happened to spend ~5 more before the test.
x = rng.normal(100, 30, n) + np.where(treated, 5.0, 0.0)
y = 0.8 * x + rng.normal(0, 10, n) + np.where(treated, 2.0, 0.0)   # true effect = 2.0

r = cuped(y, x, treated)
assert abs(r.naive_effect - 6.0) < 1.0, r.naive_effect      # biased by the imbalance
assert abs(r.effect - 2.0) < 0.5, r.effect                  # CUPED recovers ~2
assert r.se < 0.5 * r.naive_se, (r.se, r.naive_se)          # and is much more precise
assert abs(r.theta - 0.8) < 0.05, r.theta
assert np.isclose(r.adjusted.mean(), y.mean())              # pooled centring keeps the grand mean

# New users without history: imputed, not adjusted, and the caller's array is untouched.
x_missing = x.copy()
x_missing[:100] = np.nan
before = x_missing.copy()
r2 = cuped(y, x_missing, treated)
assert np.array_equal(np.isnan(x_missing), np.isnan(before))
assert np.allclose(r2.adjusted[:100], y[:100])
assert abs(r2.effect - 2.0) < 0.5

# A constant covariate carries no information: no adjustment at all.
r3 = cuped([1.0, 2.0, 3.0, 4.0], [5.0, 5.0, 5.0, 5.0], [True, False, True, False])
assert r3.theta == 0.0 and r3.effect == r3.naive_effect == -1.0

for args in [([1.0, 2.0], [1.0, 2.0], [True, True]),
             ([1.0, 2.0, 3.0], [1.0, 2.0], [True, False, True]),
             ([1.0, np.nan, 3.0, 4.0], [1.0, 2.0, 3.0, 4.0], [True, False, True, False])]:
    try:
        cuped(*args)
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted {args}")

A common wrong solution (the tests catch it):

from dataclasses import dataclass

import numpy as np


@dataclass(frozen=True)
class CupedResult:
    effect: float              # CUPED difference in means (treatment - control)
    se: float
    naive_effect: float
    naive_se: float
    theta: float
    adjusted: np.ndarray       # per-user adjusted metric, same order as the input


def _diff_and_se(values: np.ndarray, treated: np.ndarray) -> tuple[float, float]:
    t, c = values[treated], values[~treated]
    se = np.sqrt(t.var(ddof=1) / t.size + c.var(ddof=1) / c.size)
    return float(t.mean() - c.mean()), float(se)


def cuped(y, x, treated) -> CupedResult:
    """CUPED: y_adj = y - theta * (x - mean(x)), with theta and mean(x) pooled over BOTH arms.

    x is the same metric in the pre-experiment period. NaN in x (new users with no
    history) is imputed with the pooled mean of observed x, i.e. those users are not adjusted.
    """
    y = np.asarray(y, dtype=float)
    x = np.array(x, dtype=float)              # copy: we impute in place below
    treated = np.asarray(treated, dtype=bool)
    if not (y.shape == x.shape == treated.shape) or y.ndim != 1:
        raise ValueError("y, x and treated must be 1-D arrays of the same length")
    if treated.sum() < 2 or (~treated).sum() < 2:
        raise ValueError("each arm needs at least 2 users")
    if np.isnan(y).any():
        raise ValueError("y must not contain NaN")

    observed = ~np.isnan(x)
    x_mean = x[observed].mean() if observed.any() else 0.0
    x[~observed] = x_mean

    var_x = x.var(ddof=1)
    theta = 0.0 if var_x == 0 else float(np.cov(x, y, ddof=1)[0, 1] / var_x)
    arm_mean = np.where(treated, x[treated].mean(), x[~treated].mean())
    adjusted = y - theta * (x - arm_mean)          # centre within each arm

    effect, se = _diff_and_se(adjusted, treated)
    naive_effect, naive_se = _diff_and_se(y, treated)
    return CupedResult(effect, se, naive_effect, naive_se, theta, adjusted)

Alternatives and tradeoffs:

• Regression adjustment / ANCOVA: regress y on treated and x; the treatment coefficient is the CUPED estimate (with slightly different theta). It generalises to several covariates and interactions (treated × x), and gives a standard error directly; CUPED is the same idea in two NumPy lines that are easy to run at scale.
• Per-arm theta is sometimes used when the treatment changes the x–y relationship; pooled theta is the conventional default and keeps the estimator simple to explain.
• Stratification on pre-period spend buckets reduces variance too and handles non-linear relationships, at the cost of choosing buckets.
• CUPAC: use a model's prediction of y from pre-period features as the covariate; more variance reduction, but the model must use pre-treatment data only.

Follow-ups:

1. "A colleague 'improves' CUPED by centring x within each arm. The variance still drops. What's wrong?" Centring per arm subtracts each arm's own mean of x, so the arm means of adjusted equal the arm means of y: the effect estimate is exactly the naive one, chance imbalance included. Only the within-arm noise is removed, so the SE shrinks while the bias stays: a confidently wrong result. Pooled centring is what lets CUPED correct imbalance.
2. "Could you use 'number of sessions during the experiment' as the covariate? It correlates 0.9 with order value." No. A covariate measured after assignment can be affected by the treatment, which biases the estimate (you partly adjust away the effect). Only pre-treatment covariates, or ones provably unaffected by treatment, are safe.
3. "How do you validate this before it goes into the platform?" Run it on historical A/A splits: the effect should be centred on zero, the 95% CI should cover zero ~95% of the time, and the SE should drop by about 1 - corr(x, y)². Also inject a known synthetic lift into A/A data and check it's recovered without bias. Log theta and the variance reduction per experiment for monitoring.

Scoring: Strong: Pooled theta and pooled centring with a clear reason; vectorised; imputes without mutating input; can explain why per-arm centring re-introduces the imbalance and why post-treatment covariates are forbidden; proposes A/A validation. Acceptable: Correct formula and code, but treats NaN handling or the pooled-vs-per-arm choice as arbitrary until asked to reason about a chance imbalance. Red flag: Uses an in-experiment covariate, drops new users silently, or judges the method only by how much the variance went down.

SR-DS-D01 · An experiment-analysis library and daily pipeline for 40 concurrent A/B tests

Medium · Design · Senior (5–8 years) · experimentation, metric definitions, SRM, variance reduction, sequential testing, library design, reproducibility, testing statistics code · 40 min

Scenario:

A food-delivery app runs about 40 A/B tests at any time (ranking, pricing, UI). Today each data scientist analyses their own test in a notebook: they write their own SQL for "orders per user", pick their own outlier rules, and post a screenshot in Slack. Last quarter two analyses of the same test reached opposite conclusions, a pricing test was stopped early on day 3 "because it was significant", and nobody noticed that a test had assigned 52% of users to treatment instead of 50% because of a bug in the app's assignment code.

You are the senior DS asked to build the shared analysis layer: a Python library plus a daily pipeline that produces a standard scorecard for every running experiment, which DS can extend for deep dives. Assignment logs and events are in the warehouse (~25 million users, ~150 million events/day). Design it.

Clarifying questions:

• What is the unit of randomisation: user, device, session, or city (for marketplace tests where treatment spills over to riders and restaurants)? (User for most; city-level switchback tests for pricing.)
• Where do exposures come from: assignment at app open, or a logged exposure when the user actually sees the feature? (Both exist; exposure logging is inconsistent across teams.)
• Which decisions are made from the scorecard: ship/no-ship by product managers, or input to a DS who decides? How fast do they need results?
• Which metrics are company-wide guardrails (cancellations, delivery time, refunds) that every test must report?
• Is there an existing metrics layer (dbt, a semantic layer) or are metrics defined only in notebooks?
• What compute is available: warehouse SQL only, or Python jobs with enough memory to load per-user aggregates for a test (25M rows × a few columns)?

Reference design:

1. One definition of each metric, versioned. A metrics package where each metric is declared once as data: numerator, denominator, unit of analysis, filters, winsorisation/capping rule, and owner, e.g. orders_per_user = Ratio(num=orders.count, den=users.exposed, cap=p99.9). Definitions compile to warehouse SQL that produces per-unit aggregates (one row per user per experiment arm); statistics run in Python on those aggregates. Metric changes go through PR review and carry a version that every scorecard records.

2. Exposure and assignment as first-class inputs. The pipeline joins events only from each user's first exposure onward (pre-exposure events dilute the effect), restricted to the experiment window, and keeps the assignment table as the source of truth for arms. Users exposed to both arms (a known app bug class) are counted and reported, not silently dropped.

3. Trust checks before results. Every scorecard starts with: sample ratio mismatch (chi-square test of observed vs configured split, alarm at p < 0.001, results hidden behind a warning), pre-period balance on key metrics, exposure counts over time, and data-freshness checks. A failed SRM blocks the ship decision until explained; that would have caught the 52/48 bug on day one.

4. Statistics, chosen per metric type. Ratio metrics with user-level randomisation use the delta method on per-user sums (correct standard errors without resampling); CUPED with pre-period values of the same metric for variance reduction; cluster-robust standard errors or a switchback analysis for city-level tests. Multiple comparisons: one pre-registered primary metric; secondary and guardrail metrics are labelled as such, with a false-discovery-rate correction for exploratory slices.

5. Peeking, handled honestly. PMs will look daily, so the scorecard uses an always-valid / sequential method (e.g. mSPRT or group-sequential boundaries with alpha spending) for the primary metric, and shows a "not yet conclusive" state until the planned sample size or the boundary is reached. Day-3 "significance" is no longer a stopping rule.

6. Library design for DS users. A small, typed API: Experiment.load(id, as_of=...), scorecard(exp, metrics=...), analyse(per_unit_df, metric, method=...) returning a frozen result object (estimate, CI, p-value, method, metric version, data snapshot id). Pure functions over DataFrames/NumPy arrays so deep dives in notebooks use exactly the code the pipeline uses. Deterministic: seeds via np.random.default_rng where resampling is used, and results keyed by (experiment, metric version, code version, data snapshot) so any number can be reproduced.

7. Testing a statistics library. Unit tests against hand-computed cases; simulation tests in CI on synthetic A/A data (false-positive rate ≈ alpha, CI coverage ≈ 95%) and on injected lifts (bias ≈ 0, power as expected); regression tests that pin scorecard numbers for a frozen historical experiment.

8. Delivery. A daily Airflow/Dagster job runs SQL aggregates then the Python analysis per experiment (parallel per experiment, each fits in memory as per-user aggregates), writes results to a scorecards table, and a dashboard renders them. Alerts go to the experiment owner on SRM, guardrail regressions and data delays.

Trade-offs:

• Delta method vs bootstrap. Delta method is O(n) and closed-form, ideal for a daily job across 40 tests × 30 metrics; bootstrap is general (quantiles, trimmed means) but expensive. Default to delta method, keep bootstrap for special metrics and for validating the delta method.
• Sequential tests vs fixed-horizon. Sequential testing permits peeking but needs more data to reach the same power. Since people will peek, the honest choice is sequential for the primary metric and fixed-horizon for post-hoc analysis.
• Centralised definitions vs analyst freedom. A single metric definition prevents contradictory answers; but deep dives need flexibility. The library exposes the same primitives so custom analyses reuse vetted statistics even when the metric is new.
• SQL vs Python for the heavy lifting. Aggregating 150M events/day belongs in the warehouse; the statistics need a few columns per user and belong in Python where they can be unit-tested. Pushing statistics into SQL makes them hard to test.

Deep-dive probes:

• "A PM wants the result for 'users in Tier-2 cities on Android' because the overall result is flat." Look for: slicing as exploratory with FDR control or a pre-registered hypothesis, wider intervals for small segments, and a warning that a post-hoc slice is a hypothesis for the next test, not a ship decision.
• "The new ranking model increases orders per user but delivery time p90 worsens by 40 seconds." Look for: guardrails with pre-agreed thresholds, quantile metrics needing bootstrap or specialised methods, and an explicit decision framework (trade-off owner) rather than a statistics answer.
• "A pricing test randomised by user shows a huge lift, but riders were shared between arms." Look for: interference (SUTVA violation) in a marketplace, switchback or geo-randomised designs, and analysis with clustered standard errors at the randomisation unit.
• "Two years later, finance asks why the scorecard for test #418 says +2.1% and a re-run says +1.7%." Look for: versioned metric definitions, data snapshots or as-of queries, code version recorded in results, and late-arriving events (refunds) explained by an analysis cut-off date.

Rubric: Strong: Centralised, versioned metric definitions; analysis from first exposure on per-unit aggregates; SRM and balance checks gating results; delta method + CUPED with clustering at the randomisation unit; a principled answer to peeking; library API that notebooks and pipeline share; simulation-based tests (A/A false-positive rate, coverage). Recognises marketplace interference. Acceptable: Standard metrics, t-tests on per-user aggregates and a daily scorecard with SRM, but treats peeking or multiple comparisons lightly, or needs prompting on reproducibility and testing. Red flag: Analyses event-level rows as independent observations, lets each analyst define metrics ad hoc, ignores SRM, or treats the first day a p-value dips under 0.05 as the result.

SR-DS-O01 · Point-in-time joins with merge_asof leak a same-timestamp score

Medium & Tricky · Output · Senior (5–8 years) · pandas merge_asof, point-in-time correctness, label leakage, feature staleness, training-serving skew · 8 min

Question: You are reviewing the training-set builder for a credit-line model at a lending company. Each application is joined to the customer's bureau score "as of" the application time. The score table gets a new row whenever the bureau sends an update; the row for Asha at exactly 11:00 was written by the hard-inquiry pull that her own application triggered. At serving time the feature store only returns values older than 30 days as missing. What does this print?

import warnings

import pandas as pd

warnings.simplefilter("ignore", DeprecationWarning)  # pandas 2.2 + NumPy 2.x internal noise

apps = pd.DataFrame({
    "customer": ["asha", "ravi", "meera"],
    "applied_at": pd.to_datetime(["2026-03-10 11:00", "2026-03-12 09:00", "2026-03-05 10:00"]),
})
scores = pd.DataFrame({
    "customer": ["asha", "asha", "ravi", "ravi", "meera"],
    "scored_at": pd.to_datetime(["2026-03-01 00:00", "2026-03-10 11:00",
                                 "2026-03-02 00:00", "2026-03-14 00:00",
                                 "2026-01-02 00:00"]),
    "score": [720, 610, 680, 540, 700],
})

apps = apps.sort_values("applied_at")
scores = scores.sort_values("scored_at")

default = pd.merge_asof(apps, scores, left_on="applied_at", right_on="scored_at", by="customer")
print(default[["customer", "score"]].to_string(index=False))

strict = pd.merge_asof(apps, scores, left_on="applied_at", right_on="scored_at",
                       by="customer", allow_exact_matches=False,
                       tolerance=pd.Timedelta(days=30))
print(strict[["customer", "score"]].to_string(index=False))

try:
    pd.merge_asof(apps.sort_values("customer"), scores, left_on="applied_at",
                  right_on="scored_at", by="customer")
except ValueError as exc:
    print("ValueError:", exc)

Expected output:

customer  score
   meera    700
    asha    610
    ravi    680
customer  score
   meera    NaN
    asha  720.0
    ravi  680.0
ValueError: left keys must be sorted

The trap: "Asha gets 720 in the first table" (the candidate reads "as of" as strictly before), and "the rows come back in the original asha, ravi, meera order." Some also expect the last call to work because by="customer" "groups first, so the global sort shouldn't matter."

Explanation:

merge_asof is a nearest-key join. For each left row it takes the last right row (same by group) whose key is less than or equal to the left key: direction="backward" and allow_exact_matches=True are the defaults.

• Default join. Meera's application on 5 March picks up a score from 2 January, 62 days old. Asha's application at 11:00 matches the score written at exactly 11:00, which already contains the hard inquiry caused by this application: 610 instead of 720. Ravi correctly gets 680; the 14 March score is in the future and is excluded. The rows come back in the left frame's order, which is now sorted by applied_at.
• Strict join. allow_exact_matches=False makes the match strictly earlier, so Asha gets 720, which is what the decision engine actually saw. tolerance=30 days drops Meera's stale score to NaN, matching what the feature store returns at serving time. The column becomes float64 because of the NaN.
• Sort requirement. The on keys must be globally sorted in both frames, even with by. Sorting the left frame by customer raises ValueError: left keys must be sorted.

Why this matters at senior level: the model trained on the default join learns from a feature it will never have at decision time (Asha's post-inquiry 610) and from stale values that serving would have blanked (Meera's 700). Offline AUC looks better than production; the gap shows up weeks later as a training-serving skew incident. The fix is a written point-in-time rule shared by training and serving: features must be strictly before the decision timestamp, using the time the value became available (ingestion time), not the time it describes, and with the same staleness TTL as the online store.

Takeaway: merge_asof includes exact-time matches and any age of value by default; set allow_exact_matches=False and a tolerance that mirrors serving, and join on availability time.

Follow-ups:

1. "The bureau file for 1 March arrives on 3 March but is stamped scored_at = 2026-03-01. What goes wrong and how do you fix the join?" Applications on 1–2 March get a score that didn't exist yet: leakage through event time vs availability time. Store an available_at (ingestion timestamp) and join on that; keep scored_at as a feature if useful. Backfilled history must also carry the real arrival time, not the business date.
2. "How would you test that a training-set builder is point-in-time correct?" Property-style test: for every joined row assert available_at < decision_time and decision_time - available_at <= ttl; plus a fixture with a same-timestamp row and a future row like this one. Also compare a sample of training features against what the online store logged at decision time (feature logging) and alert on mismatch rate.
3. "This runs on 200 million applications. merge_asof needs a global sort; what are your options?" Partition by customer hash and sort within partitions (the by key keeps matches inside a partition), or do it in the warehouse with a window/ASOF JOIN (DuckDB, Snowflake, Spark's range joins). In pandas, process per month of applications with the score table filtered to [month_start - ttl, month_end) so each chunk fits in memory.

Scoring: Strong: All three blocks exactly, including row order, Asha's 610 and Meera's NaN. Names it label leakage plus training-serving skew, and fixes it with strict matches, a serving-equivalent TTL and availability timestamps, not just a flag change. Acceptable: Gets the default backward match and the sort error, misses that exact matches are included; once shown, explains why 610 is leakage and proposes allow_exact_matches=False. Red flag: Thinks an asof join is inherently leak-free, or proposes a random train/test split as the leakage fix.

SR-MLE-C01 · Async micro-batcher in front of a blocking model

Hard · Coding · Senior (5–8 years) · model serving, asyncio, dynamic batching, asyncio.to_thread, cancellation, failure isolation, graceful shutdown · 35 min

Problem:

A FastAPI service scores product-search queries with a ranking model. Each request carries one query, but the model's predict_batch is ~10× cheaper per item at batch size 32 than at batch size 1. Build the dynamic batching layer that sits between request handlers and the model.

MicroBatcher(predict_batch, *, max_batch_size=32, max_wait_s=0.005):

• await submit(item) returns that item's prediction. Concurrent submits are grouped into one predict_batch(list_of_items) call of up to max_batch_size items; a partial batch is sent once its first item has waited max_wait_s.
• predict_batch is a blocking function (NumPy / framework inference). It must not run on the event-loop thread.
• If predict_batch raises, or returns a different number of results than inputs, every caller in that batch gets an exception, and the batcher keeps serving later requests.
• A caller that is cancelled (client disconnected, request timeout) must not crash the worker or affect other callers; a cancelled caller's item should not be sent to the model if it hasn't been yet.
• await close() stops accepting new work (submit then raises RuntimeError), finishes everything already queued, and stops the background worker.
• Invalid settings (max_batch_size < 1, max_wait_s < 0) raise ValueError.

Signature: class MicroBatcher: def __init__(self, predict_batch: Callable[[list[Any]], Sequence[Any]], *, max_batch_size: int = 32, max_wait_s: float = 0.005) -> None: ... async def submit(self, item: Any) -> Any: ... async def close(self) -> None: ...

Constraints: Python 3.12, standard library asyncio only. One batcher per model per process; it is created inside the running event loop (e.g. in FastAPI's lifespan). Results must go back to the right caller, in the right order.

Examples:

• 20 concurrent submit(i) with max_batch_size=8, predict_batch = lambda xs: [2*x for x in xs] → results [0, 2, ..., 38]; predict_batch called with batches of 8, 8, 4
• a single submit(21) with max_wait_s=0.02 → 42, after about 20 ms, as a batch of 1
• a batch containing an input that makes predict_batch raise ValueError → every caller in that batch gets the ValueError; the next submit works normally

Hints:

1. Pair each item with an asyncio.Future the caller awaits; a single background task owns the queue and resolves the futures.
2. Collect a batch as: block for the first item, then drain with get_nowait(), then wait for more only until a deadline computed from the first item's arrival.
3. What happens to the futures still pending if predict_batch raises inside the worker task and nothing catches it? And what does set_result do on a future whose caller was cancelled?

Reference solution:

import asyncio
from collections.abc import Callable, Sequence
from typing import Any


class MicroBatcher:
    """Coalesce concurrent single-item predictions into batched model calls.

    `predict_batch` is a *blocking* function (NumPy / framework inference) and runs in a
    worker thread so the event loop keeps accepting requests.
    """

    def __init__(self, predict_batch: Callable[[list[Any]], Sequence[Any]], *,
                 max_batch_size: int = 32, max_wait_s: float = 0.005) -> None:
        if max_batch_size < 1 or max_wait_s < 0:
            raise ValueError("max_batch_size >= 1 and max_wait_s >= 0 required")
        self._predict = predict_batch
        self._max_batch = max_batch_size
        self._max_wait = max_wait_s
        self._queue: asyncio.Queue[tuple[Any, asyncio.Future]] = asyncio.Queue()
        self._worker: asyncio.Task | None = None
        self._closed = False

    async def submit(self, item: Any) -> Any:
        if self._closed:
            raise RuntimeError("batcher is closed")
        if self._worker is None:
            self._worker = asyncio.create_task(self._run(), name="micro-batcher")
        fut = asyncio.get_running_loop().create_future()
        self._queue.put_nowait((item, fut))
        return await fut

    async def close(self) -> None:
        """Stop accepting work, finish everything already queued, stop the worker."""
        self._closed = True
        if self._worker is not None:
            await self._queue.join()
            self._worker.cancel()
            try:
                await self._worker
            except asyncio.CancelledError:
                pass

    async def _collect(self) -> list[tuple[Any, asyncio.Future]]:
        loop = asyncio.get_running_loop()
        batch = [await self._queue.get()]
        deadline = loop.time() + self._max_wait
        while len(batch) < self._max_batch:
            try:
                batch.append(self._queue.get_nowait())
                continue
            except asyncio.QueueEmpty:
                pass
            remaining = deadline - loop.time()
            if remaining <= 0:
                break
            try:
                async with asyncio.timeout(remaining):
                    batch.append(await self._queue.get())
            except TimeoutError:
                break
        return batch

    async def _run(self) -> None:
        while True:
            batch = await self._collect()
            try:
                live = [(item, fut) for item, fut in batch if not fut.done()]  # skip cancelled callers
                if live:
                    await self._dispatch(live)
            finally:
                for _ in batch:
                    self._queue.task_done()

    async def _dispatch(self, live: list[tuple[Any, asyncio.Future]]) -> None:
        items = [item for item, _ in live]
        try:
            results = await asyncio.to_thread(self._predict, items)
            if len(results) != len(items):
                raise ValueError(f"predict_batch returned {len(results)} results for {len(items)} inputs")
        except Exception as exc:                      # one bad batch must not kill the worker
            for _, fut in live:
                if not fut.done():
                    fut.set_exception(exc)
            return
        for (_, fut), result in zip(live, results):
            if not fut.done():
                fut.set_result(result)

Complexity: Per request: O(1) queue operations plus its share of one model call. Added latency is at most max_wait_s plus queueing behind in-flight batches; throughput is bounded by predict_batch time per batch, because this design runs one batch at a time per batcher. Memory is O(queued requests), which is unbounded here: see the follow-up on backpressure.

Tests:

import asyncio

batches: list[list[int]] = []


def double(xs: list[int]) -> list[int]:
    batches.append(list(xs))
    if -1 in xs:
        raise ValueError("bad feature vector")
    if -2 in xs:
        return [0]                                    # wrong length
    return [x * 2 for x in xs]


async def main() -> None:
    b = MicroBatcher(double, max_batch_size=8, max_wait_s=0.02)

    # 20 concurrent requests -> 3 model calls, every caller gets its own answer.
    async with asyncio.timeout(2):
        results = await asyncio.gather(*(b.submit(i) for i in range(20)))
    assert results == [i * 2 for i in range(20)], results
    assert [len(x) for x in batches] == [8, 8, 4], batches

    # A lone request is not held forever: it goes out after max_wait as a batch of 1.
    batches.clear()
    async with asyncio.timeout(2):
        assert await b.submit(21) == 42
    assert batches == [[21]]

    # A failing batch fails every caller in it, and the worker survives.
    async with asyncio.timeout(2):
        out = await asyncio.gather(b.submit(1), b.submit(-1), b.submit(3), return_exceptions=True)
    assert all(isinstance(o, ValueError) for o in out), out
    async with asyncio.timeout(2):
        out = await asyncio.gather(b.submit(5), b.submit(-2), return_exceptions=True)
    assert all(isinstance(o, ValueError) and "results for 2 inputs" in str(o) for o in out), out
    async with asyncio.timeout(2):
        assert await b.submit(7) == 14

    # A caller that gives up (client disconnect) doesn't break the others.
    batches.clear()
    gone = asyncio.create_task(b.submit(100))
    stay = asyncio.create_task(b.submit(200))
    await asyncio.sleep(0)                            # both are queued now
    gone.cancel()
    async with asyncio.timeout(2):
        assert await stay == 400
    assert batches == [[200]], batches

    # close() drains queued work, then refuses new work.
    pending = [asyncio.create_task(b.submit(i)) for i in (30, 31, 32)]
    await asyncio.sleep(0)
    async with asyncio.timeout(2):
        await b.close()
    assert [p.result() for p in pending] == [60, 62, 64]
    try:
        await b.submit(1)
    except RuntimeError:
        pass
    else:
        raise AssertionError("submit after close accepted")


asyncio.run(main())

for bad in [dict(max_batch_size=0), dict(max_wait_s=-1)]:
    try:
        MicroBatcher(double, **bad)
    except ValueError:
        pass
    else:
        raise AssertionError(bad)

A common wrong solution (the tests catch it):

import asyncio
from collections.abc import Callable, Sequence
from typing import Any


class MicroBatcher:
    """Coalesce concurrent single-item predictions into batched model calls.

    `predict_batch` is a *blocking* function (NumPy / framework inference) and runs in a
    worker thread so the event loop keeps accepting requests.
    """

    def __init__(self, predict_batch: Callable[[list[Any]], Sequence[Any]], *,
                 max_batch_size: int = 32, max_wait_s: float = 0.005) -> None:
        if max_batch_size < 1 or max_wait_s < 0:
            raise ValueError("max_batch_size >= 1 and max_wait_s >= 0 required")
        self._predict = predict_batch
        self._max_batch = max_batch_size
        self._max_wait = max_wait_s
        self._queue: asyncio.Queue[tuple[Any, asyncio.Future]] = asyncio.Queue()
        self._worker: asyncio.Task | None = None
        self._closed = False

    async def submit(self, item: Any) -> Any:
        if self._closed:
            raise RuntimeError("batcher is closed")
        if self._worker is None:
            self._worker = asyncio.create_task(self._run(), name="micro-batcher")
        fut = asyncio.get_running_loop().create_future()
        self._queue.put_nowait((item, fut))
        return await fut

    async def close(self) -> None:
        """Stop accepting work, finish everything already queued, stop the worker."""
        self._closed = True
        if self._worker is not None:
            await self._queue.join()
            self._worker.cancel()
            try:
                await self._worker
            except asyncio.CancelledError:
                pass

    async def _collect(self) -> list[tuple[Any, asyncio.Future]]:
        loop = asyncio.get_running_loop()
        batch = [await self._queue.get()]
        deadline = loop.time() + self._max_wait
        while len(batch) < self._max_batch:
            try:
                batch.append(self._queue.get_nowait())
                continue
            except asyncio.QueueEmpty:
                pass
            remaining = deadline - loop.time()
            if remaining <= 0:
                break
            try:
                async with asyncio.timeout(remaining):
                    batch.append(await self._queue.get())
            except TimeoutError:
                break
        return batch

    async def _run(self) -> None:
        while True:
            batch = await self._collect()
            try:
                live = [(item, fut) for item, fut in batch if not fut.done()]  # skip cancelled callers
                if live:
                    await self._dispatch(live)
            finally:
                for _ in batch:
                    self._queue.task_done()

    async def _dispatch(self, live: list[tuple[Any, asyncio.Future]]) -> None:
        items = [item for item, _ in live]
        results = await asyncio.to_thread(self._predict, items)
        if len(results) != len(items):
            raise ValueError(f"predict_batch returned {len(results)} results for {len(items)} inputs")
        for (_, fut), result in zip(live, results):
            if not fut.done():
                fut.set_result(result)

Alternatives and tradeoffs:

• Serving frameworks (Triton, TorchServe, Ray Serve's @serve.batch, BentoML adaptive batching) implement this with tuned schedulers; know what they do before choosing to hand-roll. Hand-rolling makes sense when the model is a NumPy/LightGBM call inside an existing FastAPI service.
• Several batches in flight: dispatch each batch as its own task (bounded by a semaphore) to overlap model calls when the model releases the GIL or runs on a GPU stream; this improves throughput at the cost of ordering and memory.
• Process pool instead of to_thread when inference is pure-Python and holds the GIL; costs pickling of inputs and outputs per batch.
• Adaptive max_wait: wait less under low load (latency) and fill batches under high load (throughput), e.g. dispatch immediately when the model is idle.

Follow-ups:

1. "Traffic spikes to 3× what the model can process. What happens with this design, and what would you add?" The queue grows without bound: latency climbs for everyone, memory grows, and requests time out after the model has already spent work on them. Add a bounded queue (or a max-queued counter) and reject with 429/503 when full, propagate a per-request deadline and drop items whose deadline already passed before batching, and autoscale on queue depth rather than CPU.
2. "How do you choose max_batch_size and max_wait_s for a p99 latency SLO of 50 ms?" Measure model latency as a function of batch size (it's usually flat-ish then linear), then load-test: p99 ≈ max_wait + queueing + model time at the typical batch size. Pick the batch size at the knee of the throughput curve, and max_wait as a small fraction of the budget (a few ms). Export batch-size and wait-time histograms so you can see whether batches actually fill in production.
3. "Every caller in a failed batch got the error, but only one input was bad. Can you do better?" On failure, bisect: retry the batch split in halves (or item by item) to isolate the poison input, fail only that caller, and log it with a request ID. Cap the retries so a model outage doesn't multiply load. Validating inputs (shape, dtype, NaN) before batching prevents most of these.

Scoring: Strong: Future-per-request with a single owner task, deadline-based batch collection, model call in a thread, failure fan-out that keeps the worker alive, guards against cancelled futures, clean close() that drains. Raises backpressure and SLO-based tuning unprompted. Acceptable: Batches correctly and returns results to the right callers, but the worker dies on a model exception or crashes on a cancelled caller; fixes it once the failure scenario is described. Red flag: Calls the blocking model directly on the event loop, batches with a fixed sleep per request, or loses the mapping between inputs and callers.

SR-MLE-C02 · A PSI drift monitor that doesn't go blind outside the training range

Medium · Coding · Senior (5–8 years) · drift monitoring, PSI, numpy searchsorted, quantile binning, missing values, model observability · 25 min

Problem:

A credit-risk model in production uses monthly_income (rupees) as a feature. Last quarter an upstream team started sending the value in paise for one partner channel; the model's approval rate shifted for two weeks before anyone noticed, even though a drift dashboard existed. Its PSI stayed under 0.01 the whole time.

Write PsiMonitor(reference, n_bins=10, eps=1e-4) with psi(live) -> float:

• Bins come from the reference (training) data: the interior quantile cut points at 1/n_bins, 2/n_bins, …; duplicate cut points (heavily tied data) collapse, so n_bins_effective may be smaller than n_bins.
• The outermost bins are open-ended: any live value below the first cut or above the last cut still lands in a bin.
• Missing values (NaN) get their own bin, in both reference and live data, so a null spike is drift.
• Proportions are clipped below at eps before taking logs, so an empty bin can't produce inf.
• PSI = Σ (actual − expected) · ln(actual / expected) over all bins.
• Raise ValueError for an all-missing reference, n_bins < 2, or an empty live sample.

Signature: class PsiMonitor: def __init__(self, reference, n_bins: int = 10, eps: float = 1e-4) -> None: ... @property def n_bins_effective(self) -> int: ... def psi(self, live) -> float: ...

Constraints: NumPy only, vectorised: the live sample can be several million rows per hour, across ~300 features. The reference is fitted once at training time and shipped with the model artifact.

Examples:

• reference ~ N(50,000, 15,000); live from the same distribution → psi < 0.02
• live = half normal values, half 10× the training max (rupees sent as paise) → psi > 1.0 (a min/max histogram would silently drop the second half and report ≈ 0.001)
• live with 30% NaN, rest normal → psi > 0.25

Hints:

1. np.histogram(values, bins=edges) ignores values outside [edges[0], edges[-1]]. What would that do here?
2. Store only the interior cut points and use np.searchsorted(cuts, values, side='right'): indices 0..len(cuts) cover the whole real line.
3. Count missing values separately and append them as one more bin before normalising by the total sample size.

Reference solution:

import numpy as np


class PsiMonitor:
    """Population Stability Index of a live feature sample against its training reference.

    Bins are quantile cut points of the reference, open-ended at both ends, plus a
    dedicated bin for missing values, so out-of-range values and null spikes count.
    """

    def __init__(self, reference, n_bins: int = 10, eps: float = 1e-4) -> None:
        ref = np.asarray(reference, dtype=float)
        observed = ref[~np.isnan(ref)]
        if observed.size == 0:
            raise ValueError("reference has no non-missing values")
        if n_bins < 2:
            raise ValueError("n_bins must be >= 2")
        qs = np.linspace(0, 1, n_bins + 1)[1:-1]
        self.cuts = np.unique(np.quantile(observed, qs))   # ties collapse duplicate cuts
        self.eps = eps
        self.expected = self._proportions(ref)

    @property
    def n_bins_effective(self) -> int:
        return len(self.cuts) + 1                           # excluding the missing bin

    def _proportions(self, values: np.ndarray) -> np.ndarray:
        if values.size == 0:
            raise ValueError("no values to compare")
        missing = np.isnan(values)
        idx = np.searchsorted(self.cuts, values[~missing], side="right")
        counts = np.bincount(idx, minlength=len(self.cuts) + 1).astype(float)
        counts = np.append(counts, missing.sum())
        return np.maximum(counts / values.size, self.eps)   # empty bins would make log() blow up

    def psi(self, live) -> float:
        actual = self._proportions(np.asarray(live, dtype=float))
        return float(np.sum((actual - self.expected) * np.log(actual / self.expected)))

Complexity: Fitting: O(n log n) for the reference quantiles, done once. Each psi call is O(m log b) for m live values and b bins (searchsorted) plus O(m) for bincount, with O(b) state. Millions of rows per feature take milliseconds.

Tests:

import numpy as np

rng = np.random.default_rng(11)
train = rng.normal(50_000, 15_000, 50_000)          # monthly income feature, in rupees
mon = PsiMonitor(train, n_bins=10)
assert mon.n_bins_effective == 10
assert mon.psi(train) == 0.0
assert mon.psi(rng.normal(50_000, 15_000, 20_000)) < 0.02       # same distribution
assert mon.psi(rng.normal(65_000, 15_000, 20_000)) > 0.25       # one-sigma shift

# Values beyond the training range must count, not vanish.
same = rng.normal(50_000, 15_000, 10_000)
beyond = np.full(10_000, train.max() * 10)          # upstream switched rupees -> paise
assert mon.psi(np.concatenate([same, beyond])) > 1.0
assert mon.psi(np.concatenate([same, -beyond])) > 1.0

# A null spike is drift even if the non-null values look normal.
with_nulls = rng.normal(50_000, 15_000, 10_000)
with_nulls[:3_000] = np.nan
assert mon.psi(with_nulls) > 0.25

# Heavily tied reference (70% zeros, e.g. 'loans in last 90 days'): cuts collapse, still works.
tied = np.concatenate([np.zeros(7_000), rng.integers(1, 6, 3_000).astype(float)])
tmon = PsiMonitor(tied, n_bins=10)
assert tmon.n_bins_effective < 10
assert tmon.psi(np.concatenate([np.zeros(700), rng.integers(1, 6, 300).astype(float)])) < 0.05
assert tmon.psi(np.concatenate([np.zeros(300), rng.integers(1, 6, 700).astype(float)])) > 0.5

# Inputs are plain lists too; bad inputs raise.
assert PsiMonitor([1, 2, 3, 4], n_bins=2).psi([1, 2, 3, 4]) == 0.0
for make in (lambda: PsiMonitor([np.nan, np.nan]), lambda: PsiMonitor([1.0], n_bins=1),
             lambda: mon.psi([])):
    try:
        make()
    except ValueError:
        pass
    else:
        raise AssertionError("bad input accepted")

A common wrong solution (the tests catch it):

import numpy as np


class PsiMonitor:
    def __init__(self, reference, n_bins: int = 10, eps: float = 1e-4) -> None:
        ref = np.asarray(reference, dtype=float)
        ref = ref[~np.isnan(ref)]
        if ref.size == 0:
            raise ValueError("reference has no non-missing values")
        if n_bins < 2:
            raise ValueError("n_bins must be >= 2")
        self.edges = np.unique(np.quantile(ref, np.linspace(0, 1, n_bins + 1)))
        self.eps = eps
        self.expected = self._proportions(ref)

    @property
    def n_bins_effective(self) -> int:
        return len(self.edges) - 1

    def _proportions(self, values: np.ndarray) -> np.ndarray:
        values = values[~np.isnan(values)]
        if values.size == 0:
            raise ValueError("no values to compare")
        counts, _ = np.histogram(values, bins=self.edges)
        return np.maximum(counts / counts.sum(), self.eps)

    def psi(self, live) -> float:
        actual = self._proportions(np.asarray(live, dtype=float))
        return float(np.sum((actual - self.expected) * np.log(actual / self.expected)))

Alternatives and tradeoffs:

• KS statistic / Wasserstein distance: better for continuous features and don't depend on binning, but don't decompose per bin and have no "missing" notion; PSI's per-bin contributions explain where it moved.
• Jensen–Shannon divergence: symmetric and bounded, so thresholds are easier to set across features.
• Fixed-width bins are easier to explain but waste bins on skewed features like income; quantile bins put equal reference mass in each.
• Monitoring the model output and simple stats (null rate, min/max, mean, share of values outside the training range) catches unit changes immediately and cheaply; PSI is one signal among several.

Follow-ups:

1. "Your team sets a global alert at PSI > 0.2 on all 300 features and gets paged every night. What do you change?" PSI depends on sample size and binning; with small hourly samples, noise alone gives values near the threshold. Compare windows of enough data (or compute a bootstrap baseline of PSI between reference halves), tier features by importance, alert on sustained drift over N windows, and add cheap hard checks (null rate, out-of-range share, unit sanity) that are near-zero noise. Route to the feature owner with the per-bin breakdown.
2. "Why ship the reference cut points with the model rather than recomputing them from the training table when monitoring?" The training table changes (backfills, late data, retention), so recomputed bins would drift with it and stop describing what the model was trained on. Fitting once and versioning the profile with the model artifact keeps the comparison reproducible, and lets you compare model versions trained on different data.
3. "PSI fired on monthly_income, but the model's score distribution is unchanged. Do you retrain?" Not automatically. First find the cause: a pipeline bug (units, joins, defaults) is fixed upstream, not by retraining on corrupted data. If it's a real population shift, check the feature's importance and the model's performance on labelled recent data (or proxies) before retraining. Drift is a trigger for investigation, not a retraining button.

Scoring: Strong: Open-ended bins via searchsorted on interior cuts, a missing bin, eps clipping, ties handled with unique; explains why the min/max histogram went blind and links the incident to unit changes. Discusses alert noise and versioned reference profiles. Acceptable: Correct PSI with quantile bins, but needs the hint to handle out-of-range values or NaN; fixes them once pointed out. Red flag: Uses np.histogram over the reference range without noticing dropped values, drops NaN before computing, or computes bins from the live data (so every window looks like the reference).

SR-MLE-D01 · Real-time fraud scoring for UPI payments in under 50 ms

Hard · Design · Senior (5–8 years) · model serving, latency budgets, feature store, online/offline consistency, process model, canary and shadow deploys, fallbacks, ML monitoring · 45 min

Scenario:

A payments app in India must score every UPI payment for fraud before it is sent to the bank switch. The payments service calls your scoring API synchronously and will wait at most 60 ms; your budget is p99 ≤ 50 ms end to end. Traffic averages 800 requests/s and peaks at 3,000 requests/s on salary days and during festival sales.

The model is a gradient-boosted tree ensemble (~400 MB in memory) trained daily in Python on 90 days of history. It uses ~120 features: request fields (amount, payee VPA, device), aggregates ("payments by this sender in the last 10 minutes", "distinct payees in 24 h", "is the payee newly registered") and a few slower signals (device reputation from a vendor). Today the model runs as a nightly batch that flags accounts after the fact; fraud ops wants real-time blocking, with a rule-based fallback.

Design the online scoring system in Python: serving process, features, deployment of new models, and what happens when things fail.

Clarifying questions:

• What's the decision contract: a score only, or allow / step-up (extra auth) / block? Who owns thresholds, ML or fraud ops? (Score plus a decision from thresholds owned by fraud ops, configurable without a deploy.)
• What must happen if scoring is slow or down: fail open (allow the payment) or fail closed? (Fail open with rules, for amounts under ₹10,000; above that, step-up authentication.)
• How fresh must the aggregate features be? Is a 10-minute window that lags by 30 s acceptable? (Yes for most; 'payments in the last 60 s' must be near-real-time.)
• Where do the training features come from, and can we log the exact features used online? (Training reads a warehouse; there is no feature logging today.)
• Do we get labels, and how late? (Chargebacks and fraud reports arrive 2–45 days later.)
• Regulatory constraints: do we need to explain a block decision to the customer or the regulator? Data residency? (Reason codes required; data stays in India.)

Reference design:

Latency budget first. 50 ms p99 split roughly: 5 ms network and parsing, 15 ms feature fetch (one round-trip, in parallel), 10 ms model inference, 5 ms decision and logging, ~15 ms headroom. Everything that can't fit (vendor device reputation at 80 ms p99) is fetched asynchronously and cached per device, or precomputed, never awaited on the critical path.

1. Serving process. FastAPI on Uvicorn, several worker processes per pod (the GBM inference releases the GIL in the native library, but request parsing and feature assembly are Python and CPU-bound, so processes, not threads, give parallelism). Load the model before fork (Gunicorn preload_app) so workers share its pages copy-on-write, and freeze it with gc.freeze() after loading to stop the GC from touching (and copying) those pages. Endpoints are async and do I/O with httpx/Redis async clients; inference is fast enough (~1–3 ms for one row) to run inline, or it goes through asyncio.to_thread if it exceeds ~5 ms. Pods sized for peak at ≤ 60% CPU so p99 doesn't collapse under queueing; load-shed with 503 above a concurrency limit (the caller then applies the fallback).

2. Features: one definition, two paths. A feature store pattern: aggregates are computed by a streaming job (Kafka → Flink/Bytewax or a Python consumer) that maintains per-sender and per-payee windows in Redis (sorted sets or bucketed counters with TTL), written with the same feature definitions used to build training data. Online fetch is a single pipelined Redis call for all keys. Request-time features (amount, hour in IST, amount vs sender's 30-day median) are computed in a shared Python module imported by both training and serving, so the transform code is literally the same.

3. Training-serving consistency. Log the exact feature vector and model version for every scored request (to Kafka → warehouse). Train the next model on these logged features joined with later labels, where possible; and run a daily job comparing logged online features to recomputed offline ones (mismatch rate per feature, alert above 0.5%). This catches skew that would otherwise surface as an unexplained AUC drop.

4. Model lifecycle. The daily training job registers a model with its feature list, schema version, training data window, and offline metrics. Promotion: automatic checks (AUC/recall at fixed false-positive rate vs the current model on the same holdout, calibration, latency benchmark on the serving image), then shadow (scores logged, not used) for 24 h, then a canary (5% of traffic decides with the new model) with automatic rollback on error rate, latency, or block-rate shift beyond bounds. Models are loaded by version at startup; rollback is a config change, and two versions can be resident for shadowing.

5. Failure modes. Timeouts on every dependency (Redis 10 ms, total internal deadline 45 ms). If Redis is slow or down, score with request-time features only using a fallback model trained on that subset, plus rules; mark the decision degraded for monitoring. Circuit breaker on the vendor signal. If the service is down entirely, the payments service applies the agreed fail-open/step-up policy, so this service can never block all payments.

6. Monitoring. Latency per stage (histograms, not averages), error and fallback rates, score distribution and block rate per hour vs the same hour last week (a sudden shift is usually a bug, not fraud), feature null rates and PSI against the training reference, and delayed-label metrics (precision/recall once chargebacks arrive) on a weekly dashboard. Reason codes per decision (top contributing features from the tree model) for the support team and regulator.

Trade-offs:

• Python + FastAPI vs a dedicated inference server (Triton, a compiled model in Go/Rust). Python keeps feature code shared with training and is fast enough at 3,000 RPS with ~10 pods; a compiled model server cuts per-request CPU, but the feature assembly is the hard part, and duplicating it in another language risks skew. Revisit if CPU cost dominates.
• Precomputed aggregates vs on-request computation. Streaming aggregates in Redis make reads O(1) and fast, at the cost of a streaming job and eventual consistency (seconds). Computing "payments in last 10 minutes" from the database per request is simpler but adds a query to the hot path and load to the payments DB.
• Fail open vs fail closed. Fail closed stops fraud during outages but blocks legitimate payments, which is a visible incident with regulatory implications; fail open with rules and step-up for high amounts is the usual balance. The point is to make it an explicit, product-owned policy.
• Daily retraining vs stability. Fresh models track new fraud patterns, but each release is a risk. Automated gates plus shadow/canary make daily retraining safe; without them, weekly is wiser.
• Batching. Dynamic batching improves throughput for deep models; for a single-row GBM call, batching adds latency for little gain. Don't batch here.

Deep-dive probes:

• "After a deploy, p99 jumps from 35 ms to 120 ms but p50 is unchanged." Look for: per-stage latency histograms, GC pauses (fixed by gc.freeze() after loading, tuning thresholds), a cold model or cache in new workers (warm-up requests before readiness), CPU throttling from container limits, and Redis tail latency; profile with py-spy on a live worker.
• "Each pod's memory is 6 GB with 8 workers and the model is 400 MB. Why, and how do you fix it?" Look for: the model loaded per worker after fork (8 copies), or copy-on-write defeated by reference counting and GC touching object headers; preload before fork plus gc.freeze(), fewer workers per pod with more pods, or a model format that keeps parameters in a few large arrays.
• "Fraud ops says block rate doubled overnight but fraud reports didn't change." Look for: comparing feature distributions and null rates first (an upstream field change), checking which model/threshold version is live, the logged feature vectors for a sample of blocked payments, and a rollback path while investigating.
• "A feature 'payee received from 50+ distinct senders in 1 h' takes 30 ms to compute from raw events." Look for: maintaining it incrementally in the streaming job (HyperLogLog in Redis for distinct counts), accepting approximation, and the same definition in training.

Rubric: Strong: Starts from a stage-by-stage latency budget; processes with preload + gc.freeze() and async I/O; one feature-definition codebase with streaming aggregates in Redis and logged online features; shadow → canary → rollback with automated gates; explicit fail-open/step-up policy and a degraded fallback model; monitoring of latency stages, score/block-rate shifts and delayed-label quality. Acceptable: A sound FastAPI service with Redis features, a model registry and canaries, but vague on the latency budget, training-serving skew, or what happens when a dependency is slow. Red flag: Queries the payments database for aggregates on every request, loads the model per request, ignores the timeout/fallback policy, or measures success by offline AUC alone.

SR-MLE-O01 · Unpickling an old model artifact against new serving code

Medium & Tricky · Output · Senior (5–8 years) · pickle, model artifacts, versioning, __init__ vs __setstate__, model serving, backward compatibility · 7 min

Question: The training pipeline pickles a ChurnModel and uploads it to the model registry. A week later, the serving team adds a decision threshold to the class and deploys, but the registry still serves last week's artifact. In production the two halves run in different processes; here they are squeezed into one file, with the class redefined under the same name (which is what pickle sees: it stores only module.qualname and the instance __dict__). What does this print?

import pickle


# --- training repo, release 1.4 ---
class ChurnModel:
    def __init__(self, weights):
        self.weights = weights

    def predict(self, x):
        return sum(w * v for w, v in zip(self.weights, x))


artifact = pickle.dumps(ChurnModel([0.5, -0.25]))


# --- serving repo, release 1.5: threshold added, artifact NOT retrained ---
class ChurnModel:
    VERSION = "1.5"

    def __init__(self, weights, threshold=0.4):
        self.weights = weights
        self.threshold = threshold
        print("init called")

    def predict(self, x):
        score = sum(w * v for w, v in zip(self.weights, x))
        return score >= self.threshold


model = pickle.loads(artifact)
print(type(model).__name__, model.VERSION, vars(model))
print(hasattr(model, "threshold"))
print(model.predict([1.0, 0.4]))

Expected output:

ChurnModel 1.5 {'weights': [0.5, -0.25]}
False
AttributeError: 'ChurnModel' object has no attribute 'threshold'

The trap: "init called, then ChurnModel 1.5 {'weights': [0.5, -0.25], 'threshold': 0.4}, True, True." The candidate assumes unpickling constructs the object through __init__, so the new default fills the gap. The fact that VERSION shows 1.5 makes the object look fully upgraded.

Explanation:

Pickling a plain instance stores a reference to its class (__main__.ChurnModel) plus the instance __dict__. Unpickling looks the class up by name at load time, gets whatever class currently has that name (the 1.5 one), creates an empty instance with cls.__new__(cls), and sets __dict__ directly. __init__ is never called, so init called never prints and threshold is never set.

So the object is a hybrid: new-class behaviour (VERSION is a class attribute, predict is the new method) on top of old-class state ({'weights': [...]}). The first request that reaches predict crashes with AttributeError. In real systems the failure mode is often worse than a crash: if the 1.5 class had declared threshold = 0.5 as a class attribute "for safety", the old artifact would silently score with a threshold nobody trained or validated.

This is the core reason pickle is a poor model interchange format across releases: - the artifact is coupled to the exact import path and attribute layout of the code that wrote it (moving ChurnModel to another module breaks loading with AttributeError: Can't get attribute); - loading runs arbitrary code from the artifact (__reduce__), so an artifact from an untrusted bucket is a remote-code-execution risk.

What a senior MLE puts in place: store the schema version in the artifact and the registry, check it at load time against what the serving code supports (fail the deploy, not the request), implement __setstate__ for deliberate migrations, run a load-and-predict smoke test on the exact registry artifact in CI before rollout, and prefer data-only formats (weights as arrays/JSON, ONNX, the framework's native format) over pickling whole Python objects.

Takeaway: unpickling restores __dict__ into whatever class has that name today without calling __init__; version your artifacts and test old artifacts against new code.

Follow-ups:

1. "Write the __setstate__ that lets release 1.5 load 1.4 artifacts safely." def __setstate__(self, state): state.setdefault('threshold', 0.4); state.setdefault('schema', 1); self.__dict__.update(state). Better still, store schema explicitly in __getstate__, and in __setstate__ migrate per version and raise on unknown future versions, so a rollback can't load a newer artifact by accident.
2. "How would you catch this before production rather than on the first request?" A deploy gate: load the exact artifact the registry will serve with the new serving image, run predictions on a golden input set and compare with the scores recorded at training time. Also a startup check that loads the model and calls predict once, failing the readiness probe so the rollout halts.
3. "Your security team says no more pickles from the shared bucket. What do you replace it with?" Store parameters as data (NumPy .npz loaded with allow_pickle=False, JSON, safetensors-style formats, ONNX for full graphs) plus a manifest (schema version, feature list, training data hash, metrics), and sign or checksum artifacts in the registry. If pickle stays for some library, restrict loading to signed artifacts from a write-protected registry.

Scoring: Strong: Exact output: no init called, VERSION 1.5 with old __dict__, False, then the AttributeError. Explains that pickle restores state without __init__ and looks classes up by name, and proposes versioned artifacts, __setstate__ migration and a load-and-predict deploy gate; mentions the security risk unprompted. Acceptable: Realises __init__ is skipped after a hint about how objects are rebuilt, then predicts the AttributeError and suggests __setstate__ or retraining. Red flag: Believes pickle stores the class code inside the artifact, or proposes catching AttributeError in predict and returning a default.

SR-BE-C01 · Admission control that sheds load instead of queueing forever

Hard · Coding · Senior (5–8 years) · asyncio, load shedding, backpressure, concurrency limits, cancellation safety, async context managers · 35 min

Problem:

During a cricket-final sale, the order service's downstream (inventory DB) can handle about 200 concurrent queries per pod. When traffic triples, the pods queue requests in memory, latency goes to 30 s, clients time out and retry, and the queue grows further: a retry storm. You decide to shed load early: run at most limit requests at once, let at most max_waiting wait, and reject the rest immediately with 503 so clients back off.

Implement AdmissionLimiter(limit, max_waiting) used as async with limiter.slot(): ...:

• If fewer than limit are running and nobody is waiting, enter immediately.
• Otherwise wait in FIFO order; if max_waiting requests are already waiting, raise Overloaded immediately without waiting.
• Leaving the block (normally or with an exception) frees the slot and hands it to the next waiter; a newcomer must not jump the queue.
• Cancellation safety: a waiter cancelled while queued (client disconnected, request timeout) gives up its place. A waiter that is cancelled after the slot was handed to it but before it resumed must pass the slot on, so slots never leak.
• Expose active and waiting counts for metrics. Invalid arguments raise ValueError.

Signature: class AdmissionLimiter: def __init__(self, limit: int, max_waiting: int) -> None: ... @asynccontextmanager async def slot(self) -> AsyncIterator[None]: ...

Constraints: Python 3.12 standard library only. Single event loop, so no thread locks needed; correctness must not depend on timing (no sleeps). Expected scale: thousands of requests per second per pod.

Examples:

• limit=2, max_waiting=3; six requests arrive together and hold their slot → 2 running, 3 waiting, the 6th raises Overloaded at once; when released, they run in arrival order
• a waiter is cancelled while queued → waiting drops by one; the next release goes to the following waiter
• the slot is handed to waiter W, then W is cancelled before it runs → the slot passes to the next waiter; active returns to 0 when all finish

Hints:

1. Represent each waiter as a future in a deque; releasing a slot means resolving the first future that isn't already done, without decrementing active.
2. When await fut raises CancelledError, look at the future: was it cancelled, or had it already been given a result?
3. Use try/finally to remove the future from the deque on every exit path.

Reference solution:

import asyncio
from collections import deque
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager


class Overloaded(Exception):
    """Raised immediately when the wait queue is full: map it to HTTP 503 + Retry-After."""


class AdmissionLimiter:
    """At most `limit` requests run concurrently; at most `max_waiting` wait (FIFO); the rest are shed."""

    def __init__(self, limit: int, max_waiting: int) -> None:
        if limit < 1 or max_waiting < 0:
            raise ValueError("limit >= 1 and max_waiting >= 0 required")
        self._limit = limit
        self._max_waiting = max_waiting
        self._active = 0
        self._waiters: deque[asyncio.Future[None]] = deque()

    @property
    def active(self) -> int:
        return self._active

    @property
    def waiting(self) -> int:
        return sum(1 for f in self._waiters if not f.done())

    @asynccontextmanager
    async def slot(self) -> AsyncIterator[None]:
        await self._acquire()
        try:
            yield
        finally:
            self._release()

    async def _acquire(self) -> None:
        if self._active < self._limit and not self.waiting:
            self._active += 1
            return
        if self.waiting >= self._max_waiting:
            raise Overloaded(f"{self._active} running, {self.waiting} waiting")
        fut: asyncio.Future[None] = asyncio.get_running_loop().create_future()
        self._waiters.append(fut)
        try:
            await fut
        except asyncio.CancelledError:
            if fut.done() and not fut.cancelled():
                # The slot was handed to us just before we were cancelled: pass it on.
                self._release()
            raise
        finally:
            try:
                self._waiters.remove(fut)
            except ValueError:
                pass

    def _release(self) -> None:
        # Hand the slot directly to the next live waiter, so a newcomer can't barge in.
        while self._waiters:
            fut = self._waiters.popleft()
            if not fut.done():
                fut.set_result(None)
                return
        self._active -= 1

Complexity: Enter and leave are O(1) amortised (deque append/popleft); removing a cancelled waiter is O(w) for w waiters, and waiting is O(w), both bounded by max_waiting, which is small by design. Memory O(max_waiting).

Tests:

import asyncio


async def main() -> None:
    lim = AdmissionLimiter(limit=2, max_waiting=3)
    peak = 0
    order: list[int] = []

    async def handler(i: int, hold: asyncio.Event) -> int:
        nonlocal peak
        async with lim.slot():
            peak = max(peak, lim.active)
            order.append(i)
            await hold.wait()
            return i

    # 2 run, 3 wait, the 6th is shed immediately.
    gate = asyncio.Event()
    tasks = [asyncio.create_task(handler(i, gate)) for i in range(6)]
    await asyncio.sleep(0)
    assert (lim.active, lim.waiting) == (2, 3), (lim.active, lim.waiting)
    async with asyncio.timeout(1):
        try:
            await tasks[5]
        except Overloaded:
            pass
        else:
            raise AssertionError("6th request was not shed")
    gate.set()
    async with asyncio.timeout(1):
        assert await asyncio.gather(*tasks[:5]) == [0, 1, 2, 3, 4]
    assert peak == 2 and order == [0, 1, 2, 3, 4]          # FIFO admission
    assert (lim.active, lim.waiting) == (0, 0)

    # An exception inside the slot releases it.
    try:
        async with lim.slot():
            raise KeyError("boom")
    except KeyError:
        pass
    assert lim.active == 0

    # A waiter cancelled while queued (client went away) gives up its place.
    one = AdmissionLimiter(limit=1, max_waiting=2)
    hold = asyncio.Event()
    holder = asyncio.create_task(handler_on(one, hold))
    quitter = asyncio.create_task(handler_on(one, asyncio.Event()))
    await asyncio.sleep(0)
    quitter.cancel()
    await asyncio.sleep(0)
    assert one.waiting == 0
    hold.set()
    async with asyncio.timeout(1):
        await holder
    assert one.active == 0

    # The nasty case: the slot is handed to a waiter, which is cancelled before it runs.
    # The slot must pass on to the next waiter instead of leaking.
    two = AdmissionLimiter(limit=1, max_waiting=2)
    release = asyncio.Event()
    victim_box: list[asyncio.Task] = []

    async def first() -> None:
        async with two.slot():
            await release.wait()
        victim_box[0].cancel()            # slot already handed to the victim; now cancel it

    t1 = asyncio.create_task(first())
    await asyncio.sleep(0)
    victim = asyncio.create_task(handler_on(two, asyncio.Event()))
    third_done = asyncio.Event()
    third = asyncio.create_task(handler_on(two, third_done, set_first=True))
    victim_box.append(victim)
    await asyncio.sleep(0)
    assert two.waiting == 2
    release.set()
    async with asyncio.timeout(1):
        await t1
        await third
    assert victim.cancelled()
    assert two.active == 0 and two.waiting == 0


async def handler_on(lim: AdmissionLimiter, ev: asyncio.Event, set_first: bool = False) -> None:
    async with lim.slot():
        if set_first:
            ev.set()
        await ev.wait()


asyncio.run(main())

for args in [(0, 1), (1, -1)]:
    try:
        AdmissionLimiter(*args)
    except ValueError:
        pass
    else:
        raise AssertionError(args)

A common wrong solution (the tests catch it):

import asyncio
from collections import deque
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager


class Overloaded(Exception):
    """Raised immediately when the wait queue is full: map it to HTTP 503 + Retry-After."""


class AdmissionLimiter:
    """At most `limit` requests run concurrently; at most `max_waiting` wait (FIFO); the rest are shed."""

    def __init__(self, limit: int, max_waiting: int) -> None:
        if limit < 1 or max_waiting < 0:
            raise ValueError("limit >= 1 and max_waiting >= 0 required")
        self._limit = limit
        self._max_waiting = max_waiting
        self._active = 0
        self._waiters: deque[asyncio.Future[None]] = deque()

    @property
    def active(self) -> int:
        return self._active

    @property
    def waiting(self) -> int:
        return sum(1 for f in self._waiters if not f.done())

    @asynccontextmanager
    async def slot(self) -> AsyncIterator[None]:
        await self._acquire()
        try:
            yield
        finally:
            self._release()

    async def _acquire(self) -> None:
        if self._active < self._limit and not self.waiting:
            self._active += 1
            return
        if self.waiting >= self._max_waiting:
            raise Overloaded(f"{self._active} running, {self.waiting} waiting")
        fut: asyncio.Future[None] = asyncio.get_running_loop().create_future()
        self._waiters.append(fut)
        try:
            await fut
        finally:
            try:
                self._waiters.remove(fut)
            except ValueError:
                pass

    def _release(self) -> None:
        # Hand the slot directly to the next live waiter, so a newcomer can't barge in.
        while self._waiters:
            fut = self._waiters.popleft()
            if not fut.done():
                fut.set_result(None)
                return
        self._active -= 1

Alternatives and tradeoffs:

• asyncio.Semaphore plus a counter: in 3.12 the semaphore already handles the handed-then-cancelled race internally; a waiting counter around acquire() gives the shedding behaviour with less code. Writing it by hand is the interview exercise because it shows you know why that race exists.
• Deadline-aware shedding: carry the client's deadline and drop requests whose remaining budget is smaller than the typical service time, even if a slot is free. Better goodput under overload than a fixed queue length.
• Adaptive limits (AIMD / gradient algorithms like Netflix's concurrency-limits): lower limit when latency rises, raise it when healthy, instead of a hand-tuned constant.
• Shed at the edge: a rate limit at the load balancer or API gateway is cheaper, but it doesn't know a pod's actual concurrency or downstream health; the in-process limiter does. Use both.

Follow-ups:

1. "How do you pick limit and max_waiting for this pod?" limit from the downstream's capacity divided by pod count (here the DB pool size per pod is the natural limit), validated by load tests at the knee of the latency curve. max_waiting from the latency budget: if the p50 service time is 50 ms and the SLO is 500 ms, a waiter behind ~limit × 9 others will miss it anyway, so cap the queue around that and shed the rest.
2. "You return 503 with Retry-After. What must clients do so shedding actually helps?" Retry with exponential backoff and jitter, honour Retry-After, cap total retries, and use a retry budget (e.g. retries ≤ 10% of requests) so a fleet of clients doesn't turn shedding into amplified load. Only retry idempotent requests, or send idempotency keys.
3. "Where would you put this limiter in a FastAPI app, and what would you measure?" As middleware or a dependency around the routes that hit the constrained downstream (not health checks), created in the lifespan so it's bound to the running loop. Export active, waiting, shed count, and wait-time histograms; alert on sustained shedding and use waiting as an autoscaling signal.

Scoring: Strong: FIFO handoff without barging, immediate rejection when the queue is full, release in finally, and the cancelled-after-handoff case handled with a clear explanation of the race. Discusses retry storms, client backoff and how to size the limits. Acceptable: Works for the normal paths and for cancellation while queued; misses the handoff race until the scenario is described, then fixes it. Red flag: Uses an unbounded queue, uses asyncio.sleep polling, or leaks slots on exceptions.

SR-BE-C02 · Hedged reads to cut tail latency, without leaking requests

Medium · Coding · Senior (5–8 years) · tail latency, hedged requests, asyncio.wait, cancellation, ExceptionGroup, idempotency · 25 min

Problem:

The product page of a shopping app calls a pricing service with 3 read replicas. Median latency is 8 ms, but p99 is 400 ms because any replica occasionally stalls (GC pauses, a noisy neighbour on the host). Since the page waits for price, the page's p99 follows the slowest replica. Reads are idempotent, so you'll implement hedged requests: if the first replica hasn't answered within roughly its p95, send the same request to the next replica and take whichever answers first.

Implement async hedged_call(fetch, replicas, hedge_after):

• Start fetch(replicas[0]). Each time hedge_after seconds pass with no result, start the next replica (in order), while keeping the earlier attempts running.
• If an attempt fails, start the next replica immediately rather than waiting for the timer.
• Return the first successful result, and cancel every other in-flight attempt before returning (wait for them to finish cancelling, so nothing outlives the call).
• Never call a replica twice; at most len(replicas) attempts.
• If every attempt fails, raise ExceptionGroup("all replicas failed", [...]) with all the errors.
• If the caller is cancelled (its own timeout), cancel all attempts too.
• Empty replicas or negative hedge_after → ValueError.

Signature: async def hedged_call(fetch: Callable[[str], Awaitable[T]], replicas: Sequence[str], hedge_after: float) -> T:

Constraints: Python 3.12 asyncio only. fetch is an async function (e.g. an httpx.AsyncClient GET). The function is on the hot path of every page view, so no polling loops or sleeps.

Examples:

• replica a stalls for 1 s, b answers in 10 ms; hedge_after=0.05 → 'price from b' after ~60 ms; the attempt on a is cancelled
• replica b answers in 10 ms, listed first → 'price from b'; no second request is sent
• a fails at once, b answers in 10 ms; hedge_after=5 → 'price from b' in ~10 ms, not 5 s

Hints:

1. asyncio.wait(tasks, timeout=..., return_when=FIRST_COMPLETED) tells you both 'something finished' and 'the timer expired' in one call.
2. Keep a set of in-flight tasks and an iterator over the replicas still unused.
3. Where do you put the cancellation so it runs on success, on total failure, and when the caller itself is cancelled?

Reference solution:

import asyncio
from collections.abc import Awaitable, Callable, Sequence
from typing import TypeVar

T = TypeVar("T")


async def hedged_call(fetch: Callable[[str], Awaitable[T]], replicas: Sequence[str],
                      hedge_after: float) -> T:
    """Call `fetch` on replicas[0]; if it hasn't answered within `hedge_after` seconds (or it
    failed), also try the next replica. Return the first success and cancel the rest.

    Only for idempotent reads: several replicas may execute the same request.
    """
    if not replicas:
        raise ValueError("need at least one replica")
    if hedge_after < 0:
        raise ValueError("hedge_after must be >= 0")

    remaining = iter(replicas)
    pending: set[asyncio.Task[T]] = set()
    errors: list[Exception] = []

    def launch_next() -> bool:
        replica = next(remaining, None)
        if replica is None:
            return False
        pending.add(asyncio.create_task(fetch(replica), name=f"hedge:{replica}"))
        return True

    launch_next()
    try:
        while pending:
            done, _ = await asyncio.wait(pending, timeout=hedge_after,
                                         return_when=asyncio.FIRST_COMPLETED)
            if not done:                       # slow: hedge to the next replica
                launch_next()
                continue
            pending -= done
            for task in done:
                if task.exception() is None:
                    return task.result()
            errors.extend(task.exception() for task in done)
            launch_next()                      # failed: don't wait for the timer
        raise ExceptionGroup("all replicas failed", errors)
    finally:
        for task in pending:                   # losers, or everything if we were cancelled
            task.cancel()
        if pending:
            await asyncio.gather(*pending, return_exceptions=True)

Complexity: At most len(replicas) attempts; each loop iteration is O(r) for r in-flight tasks in asyncio.wait. The extra load is the hedging rate: with hedge_after at the p95, about 5% of requests send a second copy, which is the whole point: ~5% more load for a much lower p99.

Tests:

import asyncio
import time

log: list[str] = []


def replica_behaviour(spec: dict[str, tuple[float, str | None]]):
    """spec: replica -> (delay seconds, error message or None)."""
    async def fetch(replica: str) -> str:
        log.append(f"start {replica}")
        delay, error = spec[replica]
        try:
            await asyncio.sleep(delay)
        except asyncio.CancelledError:
            log.append(f"cancelled {replica}")
            raise
        if error:
            raise ConnectionError(f"{replica}: {error}")
        return f"price from {replica}"
    return fetch


async def main() -> None:
    # The primary is stuck in a GC pause; the hedge answers, and the primary is cancelled.
    log.clear()
    fetch = replica_behaviour({"a": (1.0, None), "b": (0.01, None), "c": (0.01, None)})
    t0 = time.perf_counter()
    assert await hedged_call(fetch, ["a", "b", "c"], hedge_after=0.05) == "price from b"
    assert time.perf_counter() - t0 < 0.5
    assert log == ["start a", "start b", "cancelled a"], list(log)

    # A fast primary means no extra load at all.
    log.clear()
    assert await hedged_call(fetch, ["b", "a"], hedge_after=0.05) == "price from b"
    assert log == ["start b"], list(log)

    # A failure moves on immediately instead of waiting out the hedge delay.
    log.clear()
    fetch = replica_behaviour({"a": (0.0, "connection reset"), "b": (0.01, None)})
    t0 = time.perf_counter()
    assert await hedged_call(fetch, ["a", "b"], hedge_after=5.0) == "price from b"
    assert time.perf_counter() - t0 < 1.0

    # Everything fails: every error is reported.
    log.clear()
    fetch = replica_behaviour({"a": (0.0, "down"), "b": (0.01, "down"), "c": (0.0, "down")})
    try:
        await hedged_call(fetch, ["a", "b", "c"], hedge_after=0.05)
    except* ConnectionError as eg:
        assert sorted(str(e) for e in eg.exceptions) == ["a: down", "b: down", "c: down"]
    else:
        raise AssertionError("expected ExceptionGroup")

    # Never more attempts than replicas, even if all are slow.
    log.clear()
    fetch = replica_behaviour({"a": (0.3, None), "b": (0.3, None)})
    assert await hedged_call(fetch, ["a", "b"], hedge_after=0.01) == "price from a"
    assert [m for m in log if m.startswith("start")] == ["start a", "start b"]

    # The caller's own timeout cancels every in-flight attempt; no task outlives the call.
    log.clear()
    fetch = replica_behaviour({"a": (5.0, None), "b": (5.0, None)})
    try:
        async with asyncio.timeout(0.2):
            await hedged_call(fetch, ["a", "b"], hedge_after=0.05)
    except TimeoutError:
        pass
    else:
        raise AssertionError("expected timeout")
    assert sorted(log) == ["cancelled a", "cancelled b", "start a", "start b"], list(log)
    assert asyncio.all_tasks() == {asyncio.current_task()}

    for args in [([], 0.1), (["a"], -1)]:
        try:
            await hedged_call(fetch, *args)
        except ValueError:
            pass
        else:
            raise AssertionError(args)


asyncio.run(main())

A common wrong solution (the tests catch it):

import asyncio
from collections.abc import Awaitable, Callable, Sequence
from typing import TypeVar

T = TypeVar("T")


async def hedged_call(fetch: Callable[[str], Awaitable[T]], replicas: Sequence[str],
                      hedge_after: float) -> T:
    """Call `fetch` on replicas[0]; if it hasn't answered within `hedge_after` seconds (or it
    failed), also try the next replica. Return the first success and cancel the rest.

    Only for idempotent reads: several replicas may execute the same request.
    """
    if not replicas:
        raise ValueError("need at least one replica")
    if hedge_after < 0:
        raise ValueError("hedge_after must be >= 0")

    remaining = iter(replicas)
    pending: set[asyncio.Task[T]] = set()
    errors: list[Exception] = []

    def launch_next() -> bool:
        replica = next(remaining, None)
        if replica is None:
            return False
        pending.add(asyncio.create_task(fetch(replica), name=f"hedge:{replica}"))
        return True

    launch_next()
    while pending:
        done, pending = await asyncio.wait(pending, timeout=hedge_after,
                                           return_when=asyncio.FIRST_COMPLETED)
        if not done:                           # slow: hedge to the next replica
            launch_next()
            continue
        for task in done:
            if task.exception() is None:
                return task.result()           # the slower attempts finish on their own
        errors.extend(task.exception() for task in done)
        launch_next()
    raise ExceptionGroup("all replicas failed", errors)

Alternatives and tradeoffs:

• Plain retry after a timeout (cancel the first, then try the next): lower load, but you pay the full timeout before retrying and lose the chance that the first attempt was about to answer.
• Send to all replicas at once ("tied requests" without delay): lowest latency, but multiplies load by the replica count, so the service saturates under traffic. Hedging after the p95 is the usual compromise; some systems also cancel the duplicate on the server side once one copy starts executing.
• asyncio.TaskGroup: great for "all must succeed" fan-out; for "first success wins" it's awkward because a failing attempt cancels the group. asyncio.wait with explicit cancellation fits this shape better.
• Adaptive hedge_after: track a rolling p95 per endpoint and hedge only when the request is slower than that, with a budget (e.g. at most 5% hedged) so a slow replica set doesn't double traffic during an incident.

Follow-ups:

1. "The pricing team says hedging doubled their traffic during last night's incident. What went wrong, and how do you guard against it?" When every replica slows down, every request crosses hedge_after, so every request is sent twice (or three times), adding load to an already struggling service. Use a hedging budget (token bucket allowing hedges for at most ~5% of requests), an adaptive threshold based on recent latency, and stop hedging when the error rate is high (circuit-breaker style).
2. "Why must this only be used for idempotent requests? Give a concrete failure." Both attempts may execute: a hedged POST /orders could create two orders, or a hedged payment capture could charge twice. Writes need an idempotency key honoured by the server before they can be hedged, and even then server-side dedup must be atomic.
3. "How do you verify in production that hedging actually helps?" Measure end-to-end p99 and p99.9 of the calling endpoint before and after, the hedge rate, how often the hedge wins, and backend QPS. Run it as an experiment on a fraction of traffic. If the hedge rarely wins, the tail is caused by something shared (the network, the DB), and hedging only adds load.

Scoring: Strong: Correct asyncio.wait loop with timer and failure both triggering the next attempt, cancellation of losers in finally with awaiting the cancelled tasks, ExceptionGroup for total failure, and caller cancellation handled. Explains the idempotency requirement and the load-amplification risk with a hedging budget. Acceptable: Returns the first success and hedges on time, but forgets to cancel losers, or waits for the timer after a failure; fixes it once asked what happens to the slow request. Red flag: Sends to all replicas every time without discussing load, uses time.sleep, or hedges non-idempotent writes.

SR-BE-D01 · Outbound webhook delivery for 30,000 merchants

Medium · Design · Senior (5–8 years) · webhooks, at-least-once delivery, retries and backoff, tenant isolation, asyncio, HMAC signing, SSRF, observability · 40 min

Scenario:

A payment gateway in India notifies merchants of events (payment.captured, refund.processed, settlement.completed) by POSTing JSON to a URL each merchant configures. About 30,000 merchants have webhooks; volume is ~6 million events/day, peaking at 2,000 events/s during festive sales.

Today a Celery task per event POSTs synchronously with requests and retries 3 times. Problems: when a few big merchants' servers go slow, all workers are stuck waiting on them and every merchant's webhooks are delayed by hours; merchants complain about duplicates and missing events; one merchant configured http://169.254.169.254/latest/meta-data/ as their URL and the security team is unhappy.

Design the webhook delivery system: how events are queued, delivered, retried, secured and observed, and what the merchant-facing contract is.

Clarifying questions:

• What delivery guarantee do we promise merchants: at-least-once with an event id for dedup? Is per-merchant ordering promised? (At-least-once; no strict ordering, but events carry a sequence/timestamp.)
• How long do we keep retrying a failing endpoint, and what happens after that? (72 hours, then the event goes to a dead-letter view the merchant can replay from the dashboard.)
• What timeout is reasonable for merchant endpoints, and what counts as success? (2xx within 10 s; anything else is a failure.)
• How are merchants told about failures, and can they replay events themselves? (Email after N hours failing; dashboard + API for replay.)
• Is there an existing message broker (Kafka, SQS, RabbitMQ) and a Postgres we can use? (Kafka for events; Postgres for state.)
• Any compliance constraints on payload contents (card data, PII)? (No card data; customer email and phone are present and must be minimised.)

Reference design:

Merchant contract. Each delivery is a POST with JSON body, headers Webhook-Id (the event id, stable across retries), Webhook-Timestamp, and Webhook-Signature: v1=<HMAC-SHA256(secret, id.timestamp.body)>. Merchants must verify the signature, reject timestamps older than 5 minutes, dedup on Webhook-Id, and respond 2xx fast (queue the work on their side). Documented, with SDK helpers for verification.

1. Intake. Payment services publish domain events to Kafka. A fan-out consumer looks up subscriptions (cached) and writes one delivery row per (event, endpoint) to Postgres: delivery_id, event_id, endpoint_id, status, attempt, next_attempt_at, last_error, with a unique key on (event_id, endpoint_id) so consumer replays don't create duplicates. Payloads are stored once per event (compressed), referenced by deliveries.

2. Dispatch with per-merchant isolation. Dispatcher processes claim due deliveries with SELECT ... WHERE next_attempt_at <= now() ... FOR UPDATE SKIP LOCKED LIMIT 500 (or partitioned queues per endpoint hash), and send them using asyncio + httpx.AsyncClient: thousands of in-flight requests per process, since this is pure network wait. Concurrency is limited per endpoint (e.g. 10 in flight) by a semaphore keyed by endpoint, and globally per process. A slow merchant fills only its own 10 slots; everyone else keeps flowing. Endpoints that fail repeatedly are put in a per-endpoint circuit-breaker / suspended state with probe deliveries, so they stop consuming capacity.

3. Retries. On timeout, connection error, 5xx or 429: reschedule with exponential backoff with full jitter (e.g. 30 s, 2 min, 10 min, 1 h … capped at 6 h), honouring Retry-After. On 4xx other than 408/429: retry a few times only (likely merchant bug), then dead-letter. After 72 hours: status failed, visible in the dashboard with the last response, and replayable. Attempts are recorded (status code, latency, truncated response) for support.

4. Security. Validate URLs at configuration and at send time: HTTPS only in production, resolve DNS and reject private, loopback, link-local and metadata ranges (re-check the resolved IP on each connection to defeat DNS rebinding, e.g. a custom transport that pins the validated IP), no redirects followed, response bodies read up to a small cap. Send from a dedicated egress with fixed IPs merchants can allowlist. Secrets are per endpoint, rotatable with two active secrets during rotation (two signatures in the header).

5. Observability. Per-endpoint success rate, latency, queue age (now - created_at of oldest pending delivery) and backlog; global delivery lag SLO (95% of first attempts within 30 s); alerts on dispatcher lag, not on individual merchant failures (those go to the merchant). Structured logs with event_id and delivery_id; a support tool shows every attempt for an event.

6. Capacity. 2,000 events/s × ~1.2 endpoints × average 300 ms response ≈ 720 concurrent requests at steady state, many more when merchants are slow (bounded by per-endpoint limits). A handful of async dispatcher processes handle this; Postgres handles the claim/update rate with partitioning of the deliveries table by day and indexes on (status, next_attempt_at).

Trade-offs:

• Postgres as the delivery queue vs Kafka retry topics vs a managed queue (SQS). Postgres gives per-delivery state, ad-hoc queries for support, and easy scheduling with next_attempt_at; at ~6M deliveries/day plus retries it's well within limits with partitioning. Kafka retry topics scale further but make per-endpoint isolation and replay UI harder. A managed service (or a webhook platform vendor) is worth evaluating if this isn't core.
• asyncio dispatcher vs Celery workers. Celery with sync HTTP needs one worker slot per in-flight request, so slow merchants exhaust the pool; asyncio holds thousands of waiting connections per process. The cost is discipline: no blocking calls in the dispatcher.
• Ordering. Strict per-merchant ordering means one failing event blocks all later ones for that merchant (head-of-line blocking). Most gateways don't promise ordering and give timestamps/sequence numbers plus a "fetch current state" API; that's the recommendation.
• Payload: full object vs thin event. Full payloads are convenient but leak PII into merchant logs and go stale on retries; thin events (id, type, link to fetch) force a call back to the API. A common compromise: full payload with minimal PII, plus the fetch API for the current state.

Deep-dive probes:

• "A top-10 merchant's endpoint starts timing out at 10 s during Diwali." Look for: per-endpoint concurrency limits so it only consumes its own slots, circuit breaker/suspension with probes, backlog growth isolated to that merchant, alerts to their team, and catch-up rate limits when they recover so we don't flood them.
• "A merchant says they received refund.processed twice and refunded the customer twice." Look for: at-least-once is the contract (timeouts after the merchant processed the request cause retries), the stable Webhook-Id for dedup, documentation and SDK support, and checking whether our side created two deliveries (unique key) vs one retried.
• "Security finds a merchant URL that resolves to a public IP at save time and to 10.0.0.5 at send time." Look for: DNS rebinding, validating the resolved IP at connection time, pinning the IP for the request, blocking redirects, and egress firewall rules as defence in depth.
• "Dispatcher lag grows to 20 minutes, but no merchant is slow." Look for: queue-age metric, the claim query's plan (index on status/next_attempt_at, table bloat from updates, autovacuum), dispatcher CPU (JSON serialisation, signing in Python), and scaling dispatchers horizontally since SKIP LOCKED lets them share the table.

Rubric: Strong: Clear merchant contract (event id, timestamps, HMAC signature, dedup expectation); durable delivery state with a unique key; asyncio dispatch with per-endpoint concurrency limits and circuit breaking; backoff with jitter and Retry-After; dead-letter + replay; SSRF defences including DNS rebinding; queue-age based SLOs. Explains why the Celery design failed. Acceptable: Queue + retries with backoff + signing, and some isolation, but misses SSRF details, the duplicate-delivery contract, or how a slow merchant is contained. Red flag: Promises exactly-once delivery, keeps one synchronous worker per in-flight request without isolation, retries immediately in a loop, or fetches any merchant URL without validation.

SR-BE-O01 · Read-modify-write stock decrement oversells under two workers

Medium & Tricky · Output · Senior (5–8 years) · transactions, lost update, race conditions, sqlite3, atomic conditional update, concurrency · 7 min

Question: A flash sale for the last IPL jersey in stock. Two API workers (separate processes in production, two connections here) handle checkouts that land milliseconds apart. The first block is the code that shipped; the second block is the fix from the incident review. What does this print?

import sqlite3
import tempfile
from pathlib import Path


def read_qty(conn):
    return conn.execute("SELECT qty FROM stock WHERE sku = 'IPL-JERSEY'").fetchone()[0]


def main(db):
    setup = sqlite3.connect(db)
    setup.execute("CREATE TABLE stock (sku TEXT PRIMARY KEY, qty INTEGER NOT NULL)")
    setup.execute("INSERT INTO stock VALUES ('IPL-JERSEY', 1)")
    setup.commit()
    worker_a, worker_b = sqlite3.connect(db), sqlite3.connect(db)

    # Shipped code: read, check in Python, write back.
    qty_a = read_qty(worker_a)
    qty_b = read_qty(worker_b)
    for name, conn, qty in [("A", worker_a, qty_a), ("B", worker_b, qty_b)]:
        if qty >= 1:
            conn.execute("UPDATE stock SET qty = ? WHERE sku = 'IPL-JERSEY'", (qty - 1,))
            conn.commit()
            print(f"order {name}: confirmed, qty was {qty}")
    print("qty now:", read_qty(setup))

    # Fix: let the database do the check and the decrement in one statement.
    setup.execute("UPDATE stock SET qty = 1")
    setup.commit()
    for name, conn in [("A", worker_a), ("B", worker_b)]:
        cur = conn.execute(
            "UPDATE stock SET qty = qty - 1 WHERE sku = 'IPL-JERSEY' AND qty >= 1")
        conn.commit()
        print(f"order {name}:", "confirmed" if cur.rowcount == 1 else "sold out")
    print("qty now:", read_qty(setup))
    for conn in (setup, worker_a, worker_b):
        conn.close()


with tempfile.TemporaryDirectory() as tmp:
    main(Path(tmp) / "shop.db")

Expected output:

order A: confirmed, qty was 1
order B: confirmed, qty was 1
qty now: 0
order A: confirmed
order B: sold out
qty now: 0

The trap: "order A confirmed, order B fails (or qty is -1)" for the first block. Candidates expect either the database to stop the second write ("it's in a transaction") or the arithmetic to show the oversell as a negative number. The final qty now: 0 looks perfectly healthy, which is exactly why the bug survives monitoring.

Explanation:

Shipped code: a lost update. Both workers read qty = 1 before either writes. Python's sqlite3 (legacy transaction control, the default) opens a transaction implicitly only before INSERT/UPDATE/DELETE, so each SELECT ran on its own and released its read immediately; there is nothing linking "what I read" to "what I write". Each worker then writes the absolute value it computed, qty = 0. Both commits succeed, both customers are charged, and the stock shows 0: the second write overwrote the first. One jersey, two confirmed orders, and no negative number anywhere to alert on.

Wrapping the read and write in one transaction is not automatically enough either: under READ COMMITTED (PostgreSQL's default) two transactions can still both read 1 and both write 0. You need the database to serialise the decision.

Fix: an atomic conditional update. UPDATE ... SET qty = qty - 1 WHERE ... AND qty >= 1 makes the check and the decrement a single statement that the database executes under a row (here, database) write lock. Worker A's update matches the row; worker B's re-evaluates qty >= 1 against the committed value 0, matches nothing, and rowcount == 0 tells the application to answer "sold out". The invariant lives where the data lives.

Other correct fixes and when to pick them: - SELECT ... FOR UPDATE (PostgreSQL/MySQL) when the decision needs more logic than a WHERE clause can express; it holds a row lock, so keep the transaction short. - Optimistic concurrency with a version column: UPDATE ... SET qty = ?, version = version + 1 WHERE sku = ? AND version = ?, retrying on rowcount == 0. Good for low-contention edits; under a flash sale it causes retry storms. - A CHECK (qty >= 0) constraint as a backstop: it converts a silent oversell into a loud error, but does not fix the absolute-write lost update.

Takeaway: read-check-write in application code is a race across workers; push the condition into a single atomic statement (or lock the row) and use rowcount to learn who won.

Follow-ups:

1. "Would a threading.Lock around the checkout function have prevented this?" Only within one process. Production runs several Gunicorn/Uvicorn workers on several machines; a process-local lock does nothing across them. The serialisation point has to be shared: the database row, or a distributed lock with fencing, and the database is simpler and already authoritative.
2. "The checkout also charges a payment and reserves a courier slot. How do you order those steps with the stock decrement?" Reserve stock first with the atomic decrement (cheap, reversible), then charge with an idempotency key, then confirm; if payment fails, release the reservation in a compensating update. Reservations get an expiry so abandoned checkouts return stock. Never charge first and then discover you're sold out.
3. "How would you write a test that reliably reproduces the original race?" Don't rely on timing. Inject the interleaving: call the read for both workers, then both writes, as this snippet does, or use a hook/barrier between read and write in the code under test. A property test running N concurrent checkouts against a real database asserting confirmed <= initial_stock is a good second layer.

Scoring: Strong: Exact output, names it a lost update, explains why the absolute write hides the oversell, and gives the atomic conditional update with rowcount. Also knows a process lock doesn't help across workers and that READ COMMITTED transactions alone don't fix it. Acceptable: Sees the double confirmation and fixes it with SELECT FOR UPDATE or a version column, but needs a prompt to explain why qty ends at 0 rather than -1. Red flag: Thinks the database or 'the transaction' prevents it by default, or proposes a Python lock or a sleep/retry as the fix.

SR-FS-C01 · Stop two tabs overwriting each other with ETags and If-Match

Medium · Coding · Senior (5–8 years) · optimistic concurrency, ETag, If-Match, HTTP preconditions, FastAPI, lost updates, caching · 25 min

Problem:

Sellers on a marketplace edit product listings from a web dashboard. Support keeps getting tickets like "I changed the price, and an hour later it was back": a colleague had the same listing open in another tab and saved an older copy over it. Add optimistic concurrency to the API using standard HTTP preconditions.

Write create_app(store) returning a FastAPI app over an in-memory dict[int, Listing]:

• GET /listings/{id} returns {"title", "price_paise"} with a strong ETag header derived from the listing's version, e.g. "7-v1" (quoted). If the request's If-None-Match matches (weak comparison: W/ prefixes ignored, * matches), return 304 with no body.
• PATCH /listings/{id} with a partial JSON body (title and/or price_paise, unknown fields → 422):
• no If-Match header → 428 Precondition Required;
• If-Match not matching the current ETag → 412 Precondition Failed, including the current ETag header so the client can refetch and merge;
• If-Match uses strong comparison: a W/ tag never matches; * matches any existing listing; a comma-separated list matches if any entry does;
• on success: apply only the fields sent, bump the version, return 200 with the new ETag.
• Unknown id → 404. A rejected request must not change anything.

Signature: def create_app(store: dict[int, Listing]) -> FastAPI:

Constraints: FastAPI + Pydantic v2. Endpoints may be plain def (run in FastAPI's thread pool), so think about what happens when two PATCHes run at the same moment. Use ListingPatch with extra="forbid" for the body.

Examples:

• GET /listings/7 → 200, ETag: "7-v1"
• PATCH price with If-Match: "7-v1" (Meera), then PATCH title with If-Match: "7-v1" (Arjun) → Meera 200 with ETag "7-v2"; Arjun 412 with ETag "7-v2"; Meera's price survives
• PATCH without If-Match → 428

Hints:

1. What does the client have to send so the server can tell 'I edited the version I saw' from 'I edited an old version'?
2. Treat a missing If-Match as an error, not as 'no precondition': otherwise old clients silently keep the lost-update bug.
3. Is checking the version and then writing atomic when two requests run in different threads? What's the database equivalent?

Reference solution:

import threading
from dataclasses import dataclass

from fastapi import FastAPI, Header, HTTPException, Response
from pydantic import BaseModel, ConfigDict


@dataclass
class Listing:
    title: str
    price_paise: int
    version: int = 1


class ListingPatch(BaseModel):
    model_config = ConfigDict(extra="forbid")
    title: str | None = None
    price_paise: int | None = None


def etag_of(listing_id: int, listing: Listing) -> str:
    return f'"{listing_id}-v{listing.version}"'          # strong validator, quoted per RFC 9110


def _matches(header: str, current: str) -> bool:
    """Strong comparison for If-Match: '*' or any listed strong tag equal to current."""
    if header.strip() == "*":
        return True
    tags = [t.strip() for t in header.split(",")]
    return any(t == current for t in tags if not t.startswith("W/"))


def _weak_matches(header: str, current: str) -> bool:
    """Weak comparison for If-None-Match: ignore any W/ prefix."""
    if header.strip() == "*":
        return True
    return any(t.strip().removeprefix("W/") == current for t in header.split(","))


def create_app(store: dict[int, Listing]) -> FastAPI:
    app = FastAPI()
    write_lock = threading.Lock()      # sync endpoints run in a thread pool: check-and-set must be atomic

    def get_or_404(listing_id: int) -> Listing:
        listing = store.get(listing_id)
        if listing is None:
            raise HTTPException(404, "listing not found")
        return listing

    @app.get("/listings/{listing_id}")
    def read(listing_id: int, response: Response,
             if_none_match: str | None = Header(default=None)):
        listing = get_or_404(listing_id)
        tag = etag_of(listing_id, listing)
        if if_none_match is not None and _weak_matches(if_none_match, tag):
            return Response(status_code=304, headers={"ETag": tag})
        response.headers["ETag"] = tag
        return {"title": listing.title, "price_paise": listing.price_paise}

    @app.patch("/listings/{listing_id}")
    def update(listing_id: int, patch: ListingPatch, response: Response,
               if_match: str | None = Header(default=None)):
        if if_match is None:
            raise HTTPException(428, "If-Match header required")
        with write_lock:
            listing = get_or_404(listing_id)
            current = etag_of(listing_id, listing)
            if not _matches(if_match, current):
                raise HTTPException(412, "listing was changed by someone else",
                                    headers={"ETag": current})
            for field, value in patch.model_dump(exclude_unset=True).items():
                setattr(listing, field, value)
            listing.version += 1
            response.headers["ETag"] = etag_of(listing_id, listing)
            return {"title": listing.title, "price_paise": listing.price_paise}

    return app

Complexity: O(1) per request plus parsing the If-Match list (a handful of tags). The lock serialises writes within one process only, held for microseconds.

Tests:

import warnings

warnings.simplefilter("ignore")  # TestClient deprecation noise
from fastapi.testclient import TestClient

store = {7: Listing("Handloom saree", 249_900)}
client = TestClient(create_app(store))

r = client.get("/listings/7")
assert r.status_code == 200 and r.headers["etag"] == '"7-v1"'
assert r.json() == {"title": "Handloom saree", "price_paise": 249_900}
assert client.get("/listings/7", headers={"If-None-Match": '"7-v1"'}).status_code == 304
assert client.get("/listings/7", headers={"If-None-Match": 'W/"7-v1"'}).status_code == 304
assert client.get("/listings/8").status_code == 404

# Two tabs (Meera on her laptop, her colleague Arjun on his phone) loaded version 1.
meera = client.patch("/listings/7", json={"price_paise": 199_900}, headers={"If-Match": '"7-v1"'})
assert meera.status_code == 200 and meera.headers["etag"] == '"7-v2"'
arjun = client.patch("/listings/7", json={"title": "Silk saree"}, headers={"If-Match": '"7-v1"'})
assert arjun.status_code == 412, arjun.status_code
assert arjun.headers["etag"] == '"7-v2"'                   # lets the client refetch and merge
assert store[7] == Listing("Handloom saree", 199_900, 2)    # Meera's change survived, untouched

# Missing precondition -> 428; weak tags never satisfy If-Match; lists and '*' work.
assert client.patch("/listings/7", json={"title": "x"}).status_code == 428
assert client.patch("/listings/7", json={"title": "x"}, headers={"If-Match": 'W/"7-v2"'}).status_code == 412
ok = client.patch("/listings/7", json={"title": "Silk saree"}, headers={"If-Match": '"7-v1", "7-v2"'})
assert ok.status_code == 200 and ok.json() == {"title": "Silk saree", "price_paise": 199_900}
assert client.patch("/listings/7", json={"price_paise": 1}, headers={"If-Match": "*"}).status_code == 200
assert store[7].version == 4

# Unknown fields are rejected, and a failed precondition changes nothing.
assert client.patch("/listings/7", json={"stock": 3}, headers={"If-Match": '"7-v4"'}).status_code == 422
assert client.patch("/listings/7", json={"price_paise": 5}, headers={"If-Match": '"7-v3"'}).status_code == 412
assert store[7] == Listing("Silk saree", 1, 4)
assert client.patch("/listings/9", json={"title": "x"}, headers={"If-Match": "*"}).status_code == 404

A common wrong solution (the tests catch it):

import threading
from dataclasses import dataclass

from fastapi import FastAPI, Header, HTTPException, Response
from pydantic import BaseModel, ConfigDict


@dataclass
class Listing:
    title: str
    price_paise: int
    version: int = 1


class ListingPatch(BaseModel):
    model_config = ConfigDict(extra="forbid")
    title: str | None = None
    price_paise: int | None = None


def etag_of(listing_id: int, listing: Listing) -> str:
    return f'"{listing_id}-v{listing.version}"'          # strong validator, quoted per RFC 9110


def _matches(header: str, current: str) -> bool:
    """Strong comparison for If-Match: '*' or any listed strong tag equal to current."""
    if header.strip() == "*":
        return True
    tags = [t.strip() for t in header.split(",")]
    return any(t == current for t in tags if not t.startswith("W/"))


def _weak_matches(header: str, current: str) -> bool:
    """Weak comparison for If-None-Match: ignore any W/ prefix."""
    if header.strip() == "*":
        return True
    return any(t.strip().removeprefix("W/") == current for t in header.split(","))


def create_app(store: dict[int, Listing]) -> FastAPI:
    app = FastAPI()
    write_lock = threading.Lock()      # sync endpoints run in a thread pool: check-and-set must be atomic

    def get_or_404(listing_id: int) -> Listing:
        listing = store.get(listing_id)
        if listing is None:
            raise HTTPException(404, "listing not found")
        return listing

    @app.get("/listings/{listing_id}")
    def read(listing_id: int, response: Response,
             if_none_match: str | None = Header(default=None)):
        listing = get_or_404(listing_id)
        tag = etag_of(listing_id, listing)
        if if_none_match is not None and _weak_matches(if_none_match, tag):
            return Response(status_code=304, headers={"ETag": tag})
        response.headers["ETag"] = tag
        return {"title": listing.title, "price_paise": listing.price_paise}

    @app.patch("/listings/{listing_id}")
    def update(listing_id: int, patch: ListingPatch, response: Response,
               if_match: str | None = Header(default=None)):
        with write_lock:
            listing = get_or_404(listing_id)
            current = etag_of(listing_id, listing)
            if if_match is not None and not _matches(if_match, current):
                raise HTTPException(412, "listing was changed by someone else",
                                    headers={"ETag": current})
            for field, value in patch.model_dump(exclude_unset=True).items():
                setattr(listing, field, value)
            listing.version += 1
            response.headers["ETag"] = etag_of(listing_id, listing)
            return {"title": listing.title, "price_paise": listing.price_paise}

    return app

Alternatives and tradeoffs:

• In a real database the lock is replaced by a conditional write: UPDATE listings SET ..., version = version + 1 WHERE id = :id AND version = :v and checking rowcount (0 → 412). This is atomic across processes and pods; the in-process lock is not.
• Content-hash ETags (hash of the serialised representation) need no version column and also work for caching, but are more expensive to compute and change if the serialisation changes.
• Field-level merge / PATCH semantics: instead of rejecting, merge non-conflicting field changes server-side (Arjun changed title, Meera changed price). Friendlier, but you must define conflicts precisely; many teams start with 412 plus a client-side "reload and reapply" UX.
• Pessimistic locking ("Meera is editing this listing"): good for long edits like documents; needs lock expiry and an override; overkill for short forms.

Follow-ups:

1. "What should the frontend do when it gets a 412?" Refetch the listing, show the user what changed (diff of the fields they edited vs the new server values), and let them reapply or discard. Never automatically retry the same PATCH with the new ETag: that recreates the lost update with extra steps.
2. "This API runs on 6 pods behind a load balancer. Is the solution still correct?" No. The threading.Lock and the in-memory dict are per process. The check-and-set must happen in the shared store: a conditional UPDATE ... WHERE version = ? in SQL, or a compare-and-set (WATCH/MULTI or a Lua script) in Redis. The HTTP contract (ETag, If-Match, 412/428) stays the same.
3. "A CDN in front of the API is serving stale listings after updates. How do ETags and caching headers interact here?" ETags let caches revalidate cheaply (If-None-Match → 304), but only if they revalidate: set Cache-Control: no-cache (store but always revalidate) or a short max-age for listing reads, and purge or version URLs on update. The seller dashboard should bypass shared caches entirely (private); public product pages can tolerate brief staleness.

Scoring: Strong: Correct status codes (428/412/304), strong vs weak comparison, partial updates with exclude_unset, 412 returns the current ETag, and recognises that check-then-write needs to be atomic: lock here, conditional UPDATE in a real DB. Covers the 412 UX and multi-pod correctness. Acceptable: Implements If-Match/412 with versions, but treats a missing If-Match as allowed or misses the thread race until asked. Red flag: Compares updated_at timestamps sent by the client, uses last-write-wins and calls it fine, or returns 409/200 with an error flag instead of standard preconditions without a reason.

SR-FS-C02 · Resumable Server-Sent Events with Last-Event-ID and safe framing

Hard · Coding · Senior (5–8 years) · Server-Sent Events, reconnection, Last-Event-ID, protocol framing, injection, bounded buffers, real-time UI · 30 min

Problem:

The order-tracking page of a grocery app shows live updates ("packed", "out for delivery", chat with the rider) over Server-Sent Events. Mobile browsers drop connections constantly; EventSource reconnects automatically and sends a Last-Event-ID header with the last id it received. Users report two bugs: after a reconnect, updates are missing or duplicated; and a rider typed a chat message that made the page show a fake "refund approved" event.

Write the server-side core (the FastAPI endpoint just streams what this returns, then live events):

• sse_frame(event_id, event, data) -> str: one SSE frame: id: …, event: …, then one data: line per line of data (split on \r\n, \r or \n, the three line endings the SSE spec recognises), terminated by a blank line.
• EventLog(retention): keeps the last retention events in memory; append(event, data) -> int assigns ids 1, 2, 3, …; event names must be non-empty and single-line (ValueError otherwise).
• frames_since(last_event_id) returns the frames a reconnecting client must receive:
• None (fresh connection): nothing; the page loads current state via REST first.
• A valid id: every retained event after it, in order (no duplicates, nothing missed).
• If the client has missed events that were already dropped from retention, or the id is in the future (the server restarted and ids began again), negative, or not an integer: a single reset frame (event: reset, data: resync) carrying the latest id, telling the client to refetch state.
• retention < 1 raises ValueError.

Signature: def sse_frame(event_id: int, event: str, data: str) -> str: ... class EventLog: def __init__(self, retention: int) -> None: ... def append(self, event: str, data: str) -> int: ... def frames_since(self, last_event_id: str | None) -> list[str]: ...

Constraints: Standard library only. Memory per stream is bounded by retention. Data is free text from users and staff (chat), so it may contain any characters, including blank lines.

Examples:

• sse_frame(2, 'order', 'a\nb') → 'id: 2\nevent: order\ndata: a\ndata: b\n\n'
• retention=3, ids 1..4 appended; frames_since('1') → frames for 2, 3, 4 (event 2 is still retained, so nothing was missed)
• same log; frames_since('0') → ['id: 4\nevent: reset\ndata: resync\n\n'] (event 1 was dropped)
• a chat message 'hi\n\nid: 999\nevent: payment\ndata: refund approved' → one chat event whose data is that exact text; no 'payment' event is created

Hints:

1. How does an SSE client decide where one event ends? What does a blank line inside data do if you write it out verbatim?
2. The client sends the id of the last event it received. Should the replay include that event?
3. With retention, the oldest retained id is oldest. Which values of last_event_id mean the client is missing something you no longer have?

Reference solution:

import re
from collections import deque
from dataclasses import dataclass

_LINE_BREAK = re.compile(r"\r\n|\r|\n")       # the three line endings the SSE spec recognises


@dataclass(frozen=True)
class StoredEvent:
    id: int
    event: str
    data: str


def sse_frame(event_id: int, event: str, data: str) -> str:
    lines = [f"id: {event_id}", f"event: {event}"]
    lines += [f"data: {part}" for part in _LINE_BREAK.split(data)]   # one data: line per line
    return "\n".join(lines) + "\n\n"


class EventLog:
    """Recent events for one user's notification stream, replayable by Last-Event-ID."""

    def __init__(self, retention: int) -> None:
        if retention < 1:
            raise ValueError("retention must be >= 1")
        self._events: deque[StoredEvent] = deque(maxlen=retention)
        self._last_id = 0

    @property
    def last_id(self) -> int:
        return self._last_id

    def append(self, event: str, data: str) -> int:
        if not event or _LINE_BREAK.search(event):
            raise ValueError("event name must be non-empty and single-line")
        self._last_id += 1
        self._events.append(StoredEvent(self._last_id, event, data))
        return self._last_id

    def frames_since(self, last_event_id: str | None) -> list[str]:
        """Frames a reconnecting client must receive before live events resume."""
        if last_event_id is None:
            return []
        try:
            last = int(last_event_id)
        except ValueError:
            return [self._reset()]
        oldest = self._events[0].id if self._events else self._last_id + 1
        if last > self._last_id or last < oldest - 1 or last < 0:
            return [self._reset()]                 # ids from another epoch, or we dropped events
        return [sse_frame(e.id, e.event, e.data) for e in self._events if e.id > last]

    def _reset(self) -> str:
        # Tell the client to refetch state over REST; moving its id to now avoids a reset loop.
        return sse_frame(self._last_id, "reset", "resync")

Complexity: append is O(1) (a deque with maxlen drops the oldest automatically). frames_since is O(r + total data size) for r retained events; with ids contiguous you could index directly as last - oldest + 1 for O(k) in the number of events returned. Memory is O(retention) events per stream.

Tests:

import re

def parse(stream: str) -> list[tuple[str, str, str]]:
    """Minimal client-side SSE parser (what EventSource does), returning (id, event, data)."""
    out, cur_id, event, data = [], "", "message", []
    for line in re.split(r"\r\n|\r|\n", stream):
        if line == "":
            if data:
                out.append((cur_id, event, "\n".join(data)))
            event, data = "message", []
            continue
        field, _, value = line.partition(":")
        value = value.removeprefix(" ")
        if field == "id":
            cur_id = value
        elif field == "event":
            event = value
        elif field == "data":
            data.append(value)
    return out


log = EventLog(retention=3)
assert log.frames_since(None) == [] and log.frames_since("0") == []
i1 = log.append("order", "Order #A12 packed")
i2 = log.append("order", "Out for delivery\nRider: Imran (+91 98xxxx1234)")
i3 = log.append("chat", "line one\r\nline two\rline three\n")
assert (i1, i2, i3) == (1, 2, 3)

# Multi-line data round-trips through a real parser.
assert sse_frame(2, "order", "a\nb") == "id: 2\nevent: order\ndata: a\ndata: b\n\n"
events = parse("".join(log.frames_since("0")))
assert events == [("1", "order", "Order #A12 packed"),
                  ("2", "order", "Out for delivery\nRider: Imran (+91 98xxxx1234)"),
                  ("3", "chat", "line one\nline two\nline three\n")], events

# Resume after the last id the client saw: no duplicates, nothing missed.
assert parse("".join(log.frames_since("1"))) == events[1:]
assert log.frames_since("3") == []

# User-controlled text cannot forge events.
log.append("chat", "hi\n\nid: 999\nevent: payment\ndata: refund approved")
got = parse("".join(log.frames_since("3")))
assert got == [("4", "chat", "hi\n\nid: 999\nevent: payment\ndata: refund approved")], got

# Retention 3 now holds ids 2..4. A client at 1 has missed nothing (2 is still here)...
assert [e[0] for e in parse("".join(log.frames_since("1")))] == ["2", "3", "4"]
# ...but a client at 0 missed event 1, which was dropped: it must resync.
reset = log.frames_since("0")
assert reset == ["id: 4\nevent: reset\ndata: resync\n\n"], reset
# Ids from the future (server restarted, ids began again) or garbage also resync.
assert parse(log.frames_since("50")[0]) == [("4", "reset", "resync")]
assert parse(log.frames_since("abc")[0]) == [("4", "reset", "resync")]
assert parse(log.frames_since("-1")[0]) == [("4", "reset", "resync")]

for bad in ["", "order\nevent: payment", "x\ry"]:
    try:
        log.append(bad, "data")
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted event name {bad!r}")
try:
    EventLog(0)
except ValueError:
    pass
else:
    raise AssertionError("retention 0 accepted")

# Empty log: a client claiming id 5 resyncs to id 0.
assert EventLog(5).frames_since("5") == ["id: 0\nevent: reset\ndata: resync\n\n"]

A common wrong solution (the tests catch it):

import re
from collections import deque
from dataclasses import dataclass

_LINE_BREAK = re.compile(r"\r\n|\r|\n")       # the three line endings the SSE spec recognises


@dataclass(frozen=True)
class StoredEvent:
    id: int
    event: str
    data: str


def sse_frame(event_id: int, event: str, data: str) -> str:
    lines = [f"id: {event_id}", f"event: {event}"]
    lines.append(f"data: {data}")
    return "\n".join(lines) + "\n\n"


class EventLog:
    """Recent events for one user's notification stream, replayable by Last-Event-ID."""

    def __init__(self, retention: int) -> None:
        if retention < 1:
            raise ValueError("retention must be >= 1")
        self._events: deque[StoredEvent] = deque(maxlen=retention)
        self._last_id = 0

    @property
    def last_id(self) -> int:
        return self._last_id

    def append(self, event: str, data: str) -> int:
        if not event or _LINE_BREAK.search(event):
            raise ValueError("event name must be non-empty and single-line")
        self._last_id += 1
        self._events.append(StoredEvent(self._last_id, event, data))
        return self._last_id

    def frames_since(self, last_event_id: str | None) -> list[str]:
        """Frames a reconnecting client must receive before live events resume."""
        if last_event_id is None:
            return []
        try:
            last = int(last_event_id)
        except ValueError:
            return [self._reset()]
        oldest = self._events[0].id if self._events else self._last_id + 1
        if last > self._last_id or last < oldest - 1 or last < 0:
            return [self._reset()]                 # ids from another epoch, or we dropped events
        return [sse_frame(e.id, e.event, e.data) for e in self._events if e.id > last]

    def _reset(self) -> str:
        # Tell the client to refetch state over REST; moving its id to now avoids a reset loop.
        return sse_frame(self._last_id, "reset", "resync")

Alternatives and tradeoffs:

• JSON-encode data: json.dumps escapes newlines, so one data: line is always enough and the client does JSON.parse(e.data). Most production APIs do this; the line-splitting is still required for any raw text field and is what the spec mandates.
• WebSockets give bidirectional messaging but no built-in resume: you implement ids, replay and heartbeats yourself. SSE gets reconnection and Last-Event-ID from the browser for free, works over plain HTTP/2, and is enough for server-to-client updates.
• Durable log instead of per-process memory: with several API pods, a reconnect may land on a different pod. Keep the per-order event log in Redis Streams (XADD with MAXLEN, XRANGE from the last id) or the database, so any pod can replay.
• Long polling with ?after=<id> is the fallback for proxies that buffer streaming responses.

Follow-ups:

1. "Users behind a corporate proxy see updates arrive in bursts every few minutes. What's going on, and what do you change?" The proxy (or nginx in front of the app) buffers the response. Send X-Accel-Buffering: no for nginx, Cache-Control: no-cache, avoid gzip on the stream, and send a comment heartbeat (: ping\n\n) every 15–30 s, which also keeps idle connections from being killed by load balancer timeouts.
2. "Each pod holds 20,000 open SSE connections. What does that mean for your Python process model?" Use an async server (Uvicorn) so connections are cheap coroutines, not threads; never do blocking work in the stream generator; fan out from Redis pub/sub to local subscribers per pod; raise file-descriptor limits; and plan for reconnect storms after a deploy (jittered retry: field, staggered pod restarts, replay served from Redis, not the primary DB).
3. "The event log is per order. How do you make sure a user can only subscribe to their own order's stream?" Authenticate the SSE request like any other (cookie session; EventSource can't set custom headers, so token-in-header schemes need a cookie or a short-lived signed URL token), check the order belongs to the user before streaming, and re-check on reconnect. Never put a long-lived bearer token in the query string where it ends up in logs.

Scoring: Strong: Splits data on all three line endings, explains the forged-event bug as protocol injection, replays strictly after Last-Event-ID, detects gaps with the off-by-one right, resets on future/garbage ids with the latest id to avoid reset loops. Discusses multi-pod replay and proxy buffering. Acceptable: Correct resume logic and framing for \n, but misses \r or the gap/future-id cases until prompted. Red flag: Writes data verbatim, replays including the last-seen event, or proposes resending the full history on every reconnect.

SR-FS-D01 · Bulk catalogue upload with validation preview, progress and undo

Medium · Design · Senior (5–8 years) · file uploads, background jobs, validation UX, progress reporting, idempotency, CSV injection, security, FastAPI · 40 min

Scenario:

Sellers on a marketplace manage catalogues of up to 50,000 products. They want to download their catalogue as CSV/XLSX, edit prices and stock in Excel, and upload it back. Today the upload is a synchronous FastAPI endpoint that parses the file and updates rows one by one: big files time out at the load balancer's 60 s limit after updating some rows, sellers retry and get a mix of old and new prices, and error messages are a single "Row 18,213 invalid".

On a sale day, ~3,000 sellers upload in the same hour. Design the end-to-end feature: upload, validation, the preview screen the seller sees, applying changes, progress, and undo. The frontend is React; the backend is Python (FastAPI) with Postgres and object storage.

Clarifying questions:

• Which fields can be edited in bulk (price, stock, title, images)? Are some fields restricted, e.g. price changes above 50% need approval? (Price, stock, title, status; price drops above 50% need confirmation.)
• Should a file with some invalid rows be applied partially, or all-or-nothing? (Seller chooses after the preview: apply valid rows only, or fix and re-upload.)
• How fast must changes go live, and are there downstream consumers (search index, pricing cache) that need events? (Within minutes; search and the price cache consume change events.)
• Do sellers edit through the UI and API at the same time as an upload? (Yes; their ops team and integrations both write.)
• Max file size and formats? (CSV and XLSX, up to 20 MB.)
• How long must undo be available? (24 hours, for the whole upload.)

Reference design:

Flow: upload → validate (async) → preview → confirm → apply (async) → done, with undo. Each upload is an import_job row with a state machine: uploaded → validating → ready_for_review → applying → applied | failed | reverted.

1. Upload. The browser gets a pre-signed URL and uploads directly to object storage (no 20 MB body through the API process), then calls POST /imports with the object key and an idempotency key, so a double-clicked or retried confirm creates one job. The API checks size and content type and enqueues validation.

2. Validation in a worker. A background worker (Celery/RQ/Arq, or a simple Postgres-backed job queue) streams the file: csv module with explicit encoding handling (UTF-8 with BOM, Excel's cp1252 fallback), or openpyxl in read-only mode for XLSX, with limits on rows, columns and cell length (an XLSX is a zip: cap uncompressed size to avoid zip bombs). Each row is parsed through a Pydantic model (prices as Decimal → integer paise, stock as non-negative int, SKU must belong to this seller) and compared to the current catalogue to produce a diff: unchanged / changed fields / errors / warnings (e.g. price drop > 50%). Results go to a import_rows table in batches; progress is updated every N rows.

3. Preview UX. The seller sees counts ("48,210 unchanged, 1,702 changes, 88 errors, 12 need confirmation"), a paginated, filterable table of changes with old → new values, and errors per row and column with the reason, plus a downloadable error file (their original rows with an error column) to fix and re-upload. Nothing has changed in the catalogue yet.

4. Apply. On confirm, a worker applies the diff in chunks of ~500 rows per transaction using conditional updates (UPDATE products SET price = :new WHERE id = :id AND version = :seen_version), so a product edited through the UI after validation is reported as a conflict rather than silently overwritten. Each chunk records before-values in an import_changes table (for undo) and emits change events via an outbox table for search and caches. The job is resumable: chunks are idempotent (keyed by job and row), so a worker crash resumes from the last committed chunk.

5. Progress. The UI polls GET /imports/{id} every 2 s (or listens on SSE) for state, processed/total and counts. Polling a small status row is cheap and robust; real-time push is optional here.

6. Undo. Within 24 h, POST /imports/{id}/revert creates a reverse job: for each changed row, restore the before-value only if the product still has the value this import set (version check); otherwise list it as "changed since, not reverted". It's another import job and reuses the same apply machinery.

7. Download/export safety. When generating CSV/XLSX exports, neutralise cells starting with =, +, -, @, tab or CR (prefix with ') to prevent CSV/formula injection, since product titles are user-controlled and the file is opened in Excel. Exports are generated by a worker and delivered by pre-signed URL with a short expiry.

8. Capacity and fairness. 3,000 uploads/hour is ~1 per second; validation of 50k rows is a few seconds of CPU in Python, so a modest worker pool suffices, but use per-seller concurrency limits (one active import per seller) and a fair queue so one seller with 20 uploads doesn't starve others. Track queue wait time as the main SLO.

Trade-offs:

• Validate-then-apply (two steps) vs apply directly. Two steps cost an extra click and storage for the diff, but give the seller a preview, remove partial-failure confusion, and make conflicts visible. For a 3-row edit, inline editing in the UI is better; bulk is where the preview pays off.
• All-or-nothing vs chunked apply. One giant transaction over 50k rows holds locks and blocks UI edits; chunked transactions with per-chunk idempotency and an undo log give practical atomicity ("applied" or "revertible") without long locks.
• Polling vs SSE/WebSocket for progress. Polling a status endpoint every few seconds is simple, cache-friendly, and survives deploys; push is nicer but adds connection management for a feature used briefly. Start with polling.
• Celery vs a lightweight Postgres-backed queue. Celery brings a broker and operational overhead; at ~1 job/s, a Postgres SKIP LOCKED queue (or Arq on existing Redis) is enough and keeps job state transactional with the data.

Deep-dive probes:

• "A seller uploads the same file twice in a row by accident." Look for: idempotency key on job creation, and if two jobs exist, the second's diff is empty after the first applies (since it compares against current values), so applying it is harmless.
• "Excel turned SKU 00123 into 123 and dates into serials." Look for: matching SKUs as strings, exporting with text formatting for ID columns in XLSX, warning when a SKU doesn't exist rather than creating it, and testing with real Excel round-trips.
• "The worker crashes after applying 30,000 of 50,000 rows." Look for: chunk-level commits with an idempotent chunk key, resume from the job's checkpoint, the job state visible to the seller as 'applying (resumed)', and undo still able to revert exactly the applied chunks.
• "Search shows the old price for 20 minutes after the import." Look for: outbox events emitted in the same transaction as each chunk, consumer lag monitoring, and batching of index updates; invalidating the price cache per changed product.

Rubric: Strong: Direct-to-storage upload with idempotent job creation; async streaming validation with Pydantic and a per-row/column diff preview; chunked, idempotent, conflict-aware apply with an undo log and outbox events; polling progress; formula-injection-safe exports and upload limits; per-seller fairness. Explains why the synchronous endpoint produced mixed prices. Acceptable: Moves processing to a background job with progress and error reporting, but applies without preview or conflict checks, or treats undo and export safety as afterthoughts. Red flag: Keeps the synchronous endpoint with a longer timeout, applies in one giant transaction, or reports only the first error.

SR-FS-O01 · CORS wildcard plus credentials reflects any origin

Medium & Tricky · Output · Senior (5–8 years) · CORS, FastAPI, Starlette middleware, cookies, security, same-origin policy · 8 min

Question: The React app moved to app.kiranakart.in and the API stayed on api.kiranakart.in, so a developer "fixed CORS" with a wildcard and credentials. You are reviewing it and compare it with an explicit allowlist. A page on an attacker's domain makes a credentialed fetch to the API. What does this print?

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.testclient import TestClient


def make_app(origins):
    app = FastAPI()
    app.add_middleware(CORSMiddleware, allow_origins=origins, allow_credentials=True,
                       allow_methods=["*"], allow_headers=["*"])

    @app.get("/api/me")
    def me():
        return {"user": "priya", "wallet_paise": 125_000}

    return TestClient(app)


evil = "https://free-ipl-tickets.example"
for label, origins in [("wildcard", ["*"]), ("allowlist", ["https://app.kiranakart.in"])]:
    client = make_app(origins)
    r = client.get("/api/me", headers={"Origin": evil, "Cookie": "session=abc"})
    print(label, r.status_code, r.headers.get("access-control-allow-origin"),
          r.headers.get("access-control-allow-credentials"))
    pre = client.options("/api/me", headers={
        "Origin": evil,
        "Access-Control-Request-Method": "DELETE",
        "Access-Control-Request-Headers": "x-csrf-token",
    })
    print(label, "preflight", pre.status_code, pre.headers.get("access-control-allow-origin"))

Expected output:

wildcard 200 https://free-ipl-tickets.example true
wildcard preflight 200 https://free-ipl-tickets.example
allowlist 200 None true
allowlist preflight 400 None

The trap: "wildcard 200 * true, which browsers reject, so it's safe", and "allowlist 403" for the attacker's request. The first comes from knowing the spec rule that * can't be combined with credentials; the second from thinking CORS is server-side access control.

Explanation:

Wildcard + credentials. Browsers refuse Access-Control-Allow-Origin: * on a credentialed request. To make allow_origins=["*"] "work" with allow_credentials=True, Starlette (which FastAPI's CORSMiddleware is) echoes the request's Origin back and adds Access-Control-Allow-Credentials: true. The same happens on the preflight, which approves DELETE with a custom header. The effect: every website is trusted to make authenticated calls and read the responses. A page on free-ipl-tickets.example can read Priya's wallet balance with her cookie, and call state-changing endpoints too. This configuration is effectively "disable the same-origin policy for logged-in users".

Allowlist. The attacker's origin is not listed, so no Access-Control-Allow-Origin header is sent, and the browser hides the response from the attacker's script. Note what CORS does not do: - The simple GET still returns 200 and the handler ran. CORS is enforced by the browser on reading responses; it is not authorisation. A "simple" cross-site POST (form-encoded, no custom headers) also reaches the handler, which is why state-changing endpoints still need CSRF protection (SameSite cookies, a CSRF token header that forces a preflight). - The preflight for a non-simple request fails with 400, so the browser never sends the DELETE.

Mitigations a senior FS engineer layers: an explicit origin allowlist (from config per environment, no regex like .*kiranakart.* that also matches kiranakart.in.evil.com); SameSite=Lax or Strict session cookies (they stop the browser attaching the cookie to cross-site fetch; but a team that enabled credentialed CORS often also set SameSite=None, re-opening the hole); CSRF tokens on mutations; and a test in CI asserting that an unknown origin gets no Access-Control-Allow-Origin.

Takeaway: allow_origins=["*"] with allow_credentials=True reflects any origin and lets any site read authenticated responses; list origins explicitly, and remember CORS only controls what the browser lets scripts read.

Follow-ups:

1. "Why doesn't the same problem exist when allow_credentials=False?" Then the middleware sends * and the browser will not attach cookies or Authorization from the page's credentials (a fetch with credentials: 'include' fails the CORS check against *). Anonymous public data being readable cross-origin is usually fine; the danger is ambient credentials.
2. "The frontend and API share a registrable domain (kiranakart.in). Does SameSite=Lax fully protect you here?" No: SameSite is about site, not origin. A compromised or user-content subdomain like blog.kiranakart.in is same-site, so cookies are attached. You still need an exact origin allowlist and CSRF tokens; and keep user-generated content on a separate registrable domain.
3. "How would you avoid this class of bug across 30 services?" A shared middleware/config package with a typed allowed_origins setting that rejects * when credentials are on (fail at startup), an integration test template every service runs, and a security scanner or contract test that sends a foreign Origin to each deployed service and alerts on reflection.

Scoring: Strong: All four lines exactly, including the reflected origin and the 200 on the allowlisted app. Explains that reflection makes any site a trusted reader of authenticated responses, that CORS is browser-side read control rather than authorisation, and layers allowlist + SameSite + CSRF. Acceptable: Gets the reflected origin with a nudge about what browsers do with * + credentials; expects 403 for the allowlist case but corrects after discussing where CORS is enforced. Red flag: Treats CORS as server-side protection, or 'fixes' the reflected origin by catching it in the frontend.

SR-CR-01 · Order search for the customer portal, with logging "for debugging"

Easy · Code review · Senior (5–8 years) · security, SQL injection, LIKE escaping, secrets in logs, PII, sqlite3, code review · 15 min

Context:

PR #1184 · portal/search.py · author: Karthik · reviewers: you

Customers of an online grocery store want to search their own past orders by item name ("pickle", "basmati"). The FastAPI route (not shown) takes customer_email from the authenticated session and query from the search box, and passes the raw request headers along so the function can log them.

Karthik's description:

Adds search_orders(). Substring match on the item name, newest-first ordering is a follow-up. I log the full request so on-call can reproduce bad searches. Capped at MAX_RESULTS rows, as agreed with the DBA, because some B2B customers have 40,000+ orders.

Production uses Postgres through a DB-API driver; the tests use sqlite3, which has the same placeholder semantics for this code. Review it as the senior engineer who owns the portal backend.

Code under review:

# --- portal/search.py ---
import logging
import sqlite3

logger = logging.getLogger("portal.search")

MAX_RESULTS = 50


def search_orders(conn: sqlite3.Connection, headers: dict[str, str],
                  customer_email: str, query: str) -> list[tuple[int, str]]:
    """Return (order_id, item) for this customer's orders whose item contains `query`."""
    logger.info("order search headers=%s email=%s query=%r", headers, customer_email, query)
    sql = (
        "SELECT order_id, item FROM orders "
        f"WHERE customer_email = '{customer_email}' AND item LIKE '%{query}%' "
        "ORDER BY order_id"
    )
    return conn.execute(sql).fetchall()

What a strong reviewer raises:

1. High: SQL injection through query (and customer_email). Both values are formatted into the SQL with an f-string. A search for x' OR '1'='1' -- turns the condition into customer_email = '…' AND item LIKE '%x' OR '1'='1' --%'; AND binds tighter than OR, so the OR '1'='1' branch matches every row in the table: one customer reads every other customer's orders. With a richer payload (UNION SELECT) they can read any table the DB user can. "The email comes from the session" doesn't make the email safe either: it is user-controlled at sign-up. The same bug also breaks honest searches: Haldiram's raises a syntax error, which the API turns into a 500. Fix: Use driver placeholders for every value (? for sqlite3, %s for psycopg) and pass a parameter tuple. Never build SQL with f-strings, % or .format() for values; identifiers that must vary (column names for sorting) come from an allow-list. Add a lint rule (Bandit B608, Ruff S608) so the pattern fails CI rather than relying on review.
2. High: Credentials and PII in logs. headers includes Authorization: Bearer … and the session Cookie. Logs are shipped to a log platform that far more people (and vendors) can read than the production database, and are retained for months. Anyone with log access can replay a customer's session. Logging the email and raw query also puts personal data in a store that is outside the data-retention and deletion process (DPDP Act obligations in India). Fix: Redact sensitive headers by name, case-insensitively (an allow-list of safe headers is even safer), and log a request ID plus the shape of the query (query_len), not its content. Best done once, in a logging filter or middleware, so every endpoint is covered; and rotate the tokens that were already logged.
3. Medium: MAX_RESULTS is declared but never applied. The description promises a cap, but the SQL has no LIMIT. A B2B customer with 40,000 orders searching "rice" pulls every row into memory and into the JSON response: slow responses, memory spikes on the API pods, and an easy way to load the database on purpose. Fix: LIMIT ? in SQL (not slicing in Python after fetching everything), and clamp any caller-supplied limit to the maximum. For real pagination, use keyset pagination on order_id.
4. Low: % and _ from the search box are treated as wildcards. Even after parameterising, a search for 50% matches "Mango pickle 500g", and a search of just % or _ matches everything. It's a correctness bug and a cheap way to force full scans. Fix: Escape \, %, _ in the user's text and add ESCAPE '\' to the LIKE. Mention that a leading % can't use a B-tree index, so at scale this wants a trigram index (pg_trgm) or a search engine.

Tests that expose the bug:

# --- tests/test_search.py ---
captured = []


class ListHandler(logging.Handler):
    def emit(self, record):
        captured.append(record.getMessage())


logger.addHandler(ListHandler())
logger.setLevel(logging.INFO)
logger.propagate = False

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (order_id INTEGER PRIMARY KEY, customer_email TEXT, item TEXT)")
conn.executemany("INSERT INTO orders VALUES (?, ?, ?)", [
    (1, "priya@example.in", "Mango pickle 500g"),
    (2, "priya@example.in", "Mango pickle 50% off pack"),
    (3, "priya@example.in", "Haldiram's bhujia 1kg"),
    (4, "rahul@example.in", "Lime pickle 1kg"),
    (5, "rahul@example.in", "Silk saree"),
])
conn.executemany("INSERT INTO orders VALUES (?, ?, ?)",
                 [(100 + i, "b2b@kirana.example.in", f"Basmati rice 25kg lot {i}") for i in range(80)])
headers = {"Authorization": "Bearer sk_live_9f8e7d6c", "Cookie": "session=abc123xyz",
           "User-Agent": "Mozilla/5.0"}

# Happy path.
assert search_orders(conn, headers, "priya@example.in", "pickle") == [
    (1, "Mango pickle 500g"), (2, "Mango pickle 50% off pack")]

# A legitimate apostrophe must not break the query.
assert search_orders(conn, headers, "priya@example.in", "Haldiram's") == [(3, "Haldiram's bhujia 1kg")]

# Search-box input must never widen the result to other customers' orders.
leaked = search_orders(conn, headers, "priya@example.in", "x' OR '1'='1' --")
assert leaked == [], f"returned {len(leaked)} rows, including other customers' orders"

# % and _ typed by the user are literal characters, not wildcards.
assert search_orders(conn, headers, "priya@example.in", "50%") == [(2, "Mango pickle 50% off pack")]

# The agreed cap holds even for customers with thousands of orders.
assert len(search_orders(conn, headers, "b2b@kirana.example.in", "Basmati")) == MAX_RESULTS

# Credentials never reach the log pipeline.
assert captured, "search should still log something useful for on-call"
for line in captured:
    assert "sk_live_9f8e7d6c" not in line and "abc123xyz" not in line, line

Fixed version:

# --- portal/search.py ---
import logging
import sqlite3

logger = logging.getLogger("portal.search")

MAX_RESULTS = 50
_SENSITIVE_HEADERS = {"authorization", "cookie", "x-api-key", "proxy-authorization"}


def _redact(headers: dict[str, str]) -> dict[str, str]:
    # Header names are case-insensitive, so compare lower-cased.
    return {k: ("[REDACTED]" if k.lower() in _SENSITIVE_HEADERS else v) for k, v in headers.items()}


def _escape_like(text: str) -> str:
    # Escape the escape character first, then the two LIKE wildcards.
    return text.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")


def search_orders(conn: sqlite3.Connection, headers: dict[str, str],
                  customer_email: str, query: str,
                  limit: int = MAX_RESULTS) -> list[tuple[int, str]]:
    """Return (order_id, item) for this customer's orders whose item contains `query`."""
    limit = max(1, min(limit, MAX_RESULTS))
    # Log what on-call needs (shape of the request), not credentials or the customer's identity.
    logger.info("order search headers=%s query_len=%d", _redact(headers), len(query))
    sql = (
        "SELECT order_id, item FROM orders "
        "WHERE customer_email = ? AND item LIKE ? ESCAPE '\\' "
        "ORDER BY order_id LIMIT ?"
    )
    return conn.execute(sql, (customer_email, f"%{_escape_like(query)}%", limit)).fetchall()

Follow-ups:

1. "The author replies: "Parameterising is fine, but the email comes from our own session, so it's trusted." How do you respond in the review?" Trust is about where the bytes originally came from, not which variable holds them. The email was typed by a user at sign-up; if any path (OAuth provider, admin import, a future feature) stores a'--@x.in, this query breaks or leaks. Placeholders cost nothing, so the team standard should be "no string-built SQL with values, ever", enforced by a linter so it doesn't depend on judging trust case by case.
2. "How would you make sure secrets never reach logs across 30 services, not just in this function?" Centralise it: a shared logging config with a logging.Filter (or structlog processor) that redacts known keys (authorization, cookie, password, token, PAN/Aadhaar patterns) before any handler sees the record; middleware that logs requests through an allow-list of headers; secret-scanning on the log pipeline with alerts; and a test that logs a fake token through the real config and asserts it is redacted. Include rotation in the runbook for when a leak is found anyway.
3. "Is a LIKE '%term%' search acceptable at 50 million order rows? What would you propose?" Not on a hot path: a leading wildcard prevents B-tree index use, so each search scans the customer's rows (fine if customer_email is indexed and per-customer counts are small) or the whole table (not fine). Options: a composite index on (customer_email, order_id) so the scan is per customer; a pg_trgm GIN index for substring search; or a search service (OpenSearch) fed by change data capture if ranking and typo tolerance matter. Choose based on per-customer cardinality and the latency target.
4. "Which tests would you ask the author to add so this class of bug can't come back?" Tests with hostile inputs (quotes, --, %, _, backslash, very long strings) asserting results stay within the caller's own rows; a test that a B2B account's search is capped at the limit; a test that captures log output (pytest's caplog) and asserts no token or cookie appears. Plus a static-analysis rule in CI, because a test covers one function while a lint rule covers all of them.

Scoring: Strong: Spots the injection immediately, demonstrates the exploit (why OR defeats the email filter), and rejects "the email is trusted". Raises the token-in-logs issue as equally serious and talks about rotation and a central redaction filter. Notices the unused MAX_RESULTS, fixes with LIMIT ?, and mentions LIKE wildcard escaping. Proposes linters and tests that enforce the standard team-wide. Acceptable: Finds the injection and fixes it with placeholders, and finds the logged Authorization header once asked to look at the logging line. Misses the wildcard escaping or the missing LIMIT without a prompt. Red flag: Fixes injection by escaping quotes by hand (query.replace("'", "''")) or by validating with a regex instead of placeholders. Sees nothing wrong with logging full headers "because logs are internal". Comments mainly on naming and type hints. At five-plus years, missing SQL injection in a string-built query is disqualifying for a reviewer role.

SR-CR-02 · Upcasting v1 order events to a v2 data contract

Medium · Code review · Senior (5–8 years) · data contracts, schema evolution, pydantic v2, money as integers, time zones, tolerant reader · 20 min

Context:

PR #412 · contracts/order_events.py · author: Sneha · reviewers: you (owner of the revenue pipeline)

The order-placed Kafka topic carries two versions of the same event while the checkout team migrates:

• v1 (old monolith, still ~30% of traffic): {"order_id": "OD-1001", "amount": 19.99, "placed_at": "2026-03-31 23:45:00"}. amount is in rupees as a JSON float; placed_at is the IST wall-clock time with no offset (the monolith always ran on IST servers).
• v2 (new checkout service): {"schema_version": 2, "order_id": "OD-1002", "amount_paise": 1999, "placed_at": "2026-03-31T18:15:00+00:00"}.

The consumer validates every message into OrderPlaced (always v2 shape) and writes it to the warehouse, where finance builds daily revenue by IST business date. The consumer keeps the raw dict so that a message that fails later in the pipeline can be sent to the dead-letter topic unchanged.

Sneha's description: "Single model for both versions; a before validator upcasts v1. extra="forbid" so we catch contract drift early. Frozen so nobody mutates events downstream."

Code under review:

# --- contracts/order_events.py ---
from datetime import date, datetime, timedelta, timezone

from pydantic import BaseModel, ConfigDict, ValidationError, model_validator

IST = timezone(timedelta(hours=5, minutes=30))


class OrderPlaced(BaseModel):
    model_config = ConfigDict(extra="forbid", frozen=True)

    schema_version: int = 2
    order_id: str
    amount_paise: int
    placed_at: datetime

    @model_validator(mode="before")
    @classmethod
    def upcast_v1(cls, data):
        if isinstance(data, dict) and "amount" in data:
            data["amount_paise"] = int(data.pop("amount") * 100)
            data["schema_version"] = 2
            data["placed_at"] = datetime.fromisoformat(data["placed_at"]).replace(tzinfo=timezone.utc)
        return data


def ist_business_date(event: OrderPlaced) -> date:
    return event.placed_at.astimezone(IST).date()

What a strong reviewer raises:

1. High: Money through a float, then truncated. 19.99 * 100 is 1998.9999999999998 in binary floating point, and int() truncates toward zero, so ₹19.99 becomes 1998 paise; ₹0.29 becomes 28. Every affected order is under-reported by a paisa. Small per row, but across lakhs of orders the revenue table no longer reconciles with the payment gateway's settlement file, and finance opens an incident. Fix: Convert through Decimal(str(value)) (which recovers the shortest decimal text, '19.99'), multiply by 100, and reject anything that isn't a whole number of paise instead of rounding silently. Once the event is v2, keep money as integer paise end to end.
2. High: Wrong time zone for v1. The v1 contract says placed_at is IST wall-clock time, but the upcaster stamps it as UTC. An order at 23:45 IST on 31 March becomes 23:45 UTC, which is 05:15 IST on 1 April, so it moves to the next business day and here into the next financial year. Every v1 order is shifted 5h30m; anything after 18:30 IST lands on the wrong date in the daily revenue report. Fix: replace(tzinfo=IST) for naive v1 values (that is the documented meaning), and convert to UTC only for storage if wanted. Write the assumption into the contract docs and add a boundary test at 23:45 IST.
3. Medium: extra="forbid" on the consumer turns every additive producer change into an outage. When the checkout team adds an optional coupon_code, every new message fails validation, the consumer stops or dead-letters 100% of traffic, and revenue goes missing until someone redeploys. Adding an optional field should be a backward-compatible change. Fix: Consumers use extra="ignore" (tolerant reader); strictness belongs at the producer boundary and in a schema registry/compatibility check in CI (for example, a test that validates sample messages from every producer version). Raise "catch drift early" as a CI concern, not a runtime one.
4. Medium: The validator mutates the caller's dict. data.pop("amount") and the assignments rewrite the consumer's raw message, so the dead-letter topic receives a half-upcast payload (amount_paise and a datetime object that isn't even JSON-serialisable) instead of what was received, which makes replay and debugging unreliable. It also makes validation non-idempotent: validating the same dict twice takes a different branch the second time. Fix: Copy first (data = dict(data)). A before validator should treat its input as read-only.
5. Medium: v2 accepts naive datetimes. placed_at: datetime lets a v2 producer send "2026-04-02T10:00:00"; pydantic produces a naive datetime, and astimezone() then interprets it in the consumer host's local time zone, so the result depends on where the consumer runs. Fix: AwareDatetime for the field so a missing offset is a validation error at the contract boundary.
6. Low: Version detection by "amount is present" is fragile: a v2 producer that also sends a legacy amount (common during a migration) would be upcast again and overwrite amount_paise. Fix: Branch on schema_version (absent means v1), and log/metric the count of v1 messages so the team knows when v1 can be removed.

Tests that expose the bug:

# --- tests/test_order_events.py ---
# v1: ₹19.99 must be exactly 1999 paise, not "about" 1999.
raw_v1 = {"order_id": "OD-1001", "amount": 19.99, "placed_at": "2026-03-31 23:45:00"}
ev = OrderPlaced.model_validate(raw_v1)
assert ev.amount_paise == 1999, ev.amount_paise
assert OrderPlaced.model_validate({"order_id": "OD-1003", "amount": 0.29,
                                   "placed_at": "2026-04-01 09:00:00"}).amount_paise == 29

# v1 timestamps are IST wall-clock: 23:45 on 31 March is still FY 2025-26.
assert ist_business_date(ev) == date(2026, 3, 31), ist_business_date(ev)
assert ev.placed_at == datetime(2026, 3, 31, 18, 15, tzinfo=timezone.utc)

# The consumer's raw message must be left as it was received (for the dead-letter topic).
assert raw_v1 == {"order_id": "OD-1001", "amount": 19.99, "placed_at": "2026-03-31 23:45:00"}, raw_v1

# v2 passes through unchanged.
ev2 = OrderPlaced.model_validate({"schema_version": 2, "order_id": "OD-1002", "amount_paise": 1999,
                                  "placed_at": "2026-03-31T18:15:00+00:00"})
assert ev2.amount_paise == 1999 and ist_business_date(ev2) == date(2026, 3, 31)

# Producers add optional fields over time; consumers must keep working (tolerant reader).
ev3 = OrderPlaced.model_validate({"schema_version": 2, "order_id": "OD-1004", "amount_paise": 50000,
                                  "placed_at": "2026-04-02T10:00:00+05:30", "coupon_code": "DIWALI10"})
assert ev3.amount_paise == 50000

# A v2 producer that forgets the offset is a contract violation, not "assume UTC".
try:
    OrderPlaced.model_validate({"schema_version": 2, "order_id": "OD-1005", "amount_paise": 100,
                                "placed_at": "2026-04-02T10:00:00"})
except ValidationError:
    pass
else:
    raise AssertionError("naive v2 timestamp was accepted")

# Sub-paisa v1 amounts are rejected instead of silently truncated.
try:
    OrderPlaced.model_validate({"order_id": "OD-1006", "amount": 10.005, "placed_at": "2026-04-02 10:00:00"})
except ValidationError:
    pass
else:
    raise AssertionError("₹10.005 was accepted")

Fixed version:

# --- contracts/order_events.py ---
from datetime import date, datetime, timedelta, timezone
from decimal import Decimal

from pydantic import AwareDatetime, BaseModel, ConfigDict, ValidationError, model_validator

IST = timezone(timedelta(hours=5, minutes=30))


class OrderPlaced(BaseModel):
    # Consumers ignore unknown fields (tolerant reader); producers are validated strictly on their side.
    model_config = ConfigDict(extra="ignore", frozen=True)

    schema_version: int = 2
    order_id: str
    amount_paise: int
    placed_at: AwareDatetime          # naive datetimes are rejected

    @model_validator(mode="before")
    @classmethod
    def upcast_v1(cls, data):
        # Decide on the version field, not on which keys happen to be present.
        if not isinstance(data, dict) or "schema_version" in data:
            return data
        data = dict(data)                                   # never mutate the caller's message
        rupees = Decimal(str(data.pop("amount")))           # str() keeps the decimal text: 19.99 -> '19.99'
        paise = rupees * 100
        if paise != paise.to_integral_value():
            raise ValueError(f"amount {rupees} has more than 2 decimal places")
        data["amount_paise"] = int(paise)
        naive = datetime.fromisoformat(data["placed_at"])
        if naive.tzinfo is None:
            naive = naive.replace(tzinfo=IST)               # v1 contract: IST wall-clock time
        data["placed_at"] = naive
        data["schema_version"] = 2
        return data


def ist_business_date(event: OrderPlaced) -> date:
    return event.placed_at.astimezone(IST).date()

Follow-ups:

1. "Finance says the last three weeks of revenue are wrong. What's your plan beyond merging the fix?" Quantify the damage (count of v1 orders, and how many fall after 18:30 IST or have truncated paise) by re-validating the raw messages, which is possible only if raw events are retained (topic retention or a raw landing zone). Backfill the affected warehouse partitions with the fixed code idempotently (overwrite partitions by date, don't append), reconcile against the gateway settlement totals, and tell finance which dates changed, including the FY boundary. Add a daily reconciliation check so the next drift is caught in a day, not three weeks.
2. "Where should the contract live so three producer teams and two consumer teams stay compatible?" In one versioned place: a schema registry (Avro/Protobuf/JSON Schema with compatibility rules), or a small shared package of pydantic models with semantic versioning. Producers run a CI check that new schemas are backward compatible with the last released one; consumers run contract tests against sample messages from each live producer version. The rules: add optional fields freely, never change the meaning or unit of an existing field (that's why it's amount_paise, a new name), and remove fields only after consumers stop reading them.
3. "Why does Decimal(str(19.99)) work but Decimal(19.99) doesn't?" Decimal(19.99) converts the exact binary value of the float: 19.989999999999998436805981327779591083526611328125. str(19.99) gives the shortest decimal string that round-trips to the same float, '19.99', which is what the producer meant. Even better is receiving the amount as a string or integer paise in the first place, so no float ever touches money.
4. "Should the consumer crash, skip, or dead-letter a message that fails validation?" Dead-letter it with the original bytes, the error and the consumer version, increment a metric, and alert on the rate, so one bad producer doesn't stop the partition (crashing blocks every message behind it) and the data isn't silently lost (skipping). If the failure rate jumps, for example because a new field broke validation, page someone: that's a contract break, not bad data.

Scoring: Strong: Finds the float truncation and explains it with the actual value (1998.999…), finds the UTC/IST mix-up and its date and financial-year impact, and pushes back on extra="forbid" for consumers with the tolerant-reader argument. Notices the mutation of the raw dict and the dead-letter consequence. Talks about backfilling and reconciliation, not just the code fix. Acceptable: Finds the money bug (suggests round() or Decimal) and the time zone bug, and accepts the extra="forbid" problem once the coupon-code scenario is described. Misses the dict mutation or the naive-v2 hole. Red flag: Suggests round(amount * 100) and keeping floats for money "because it's close enough". Thinks a naive timestamp "is UTC by default". Approves extra="forbid" on the consumer as "safer". Reviews only the shape of the pydantic code without asking how finance uses the data.

SR-CR-03 · Request coalescing in an async price cache

Medium & Tricky · Code review · Senior (5–8 years) · asyncio, caching, request coalescing, cancellation, asyncio.shield, TTL, failure handling · 20 min

Context:

PR #2290 · checkout/price_cache.py · author: Aditya · reviewers: you (owner of the checkout service)

Checkout is an async FastAPI service. Every cart render calls the internal pricing service (p50 ~300 ms) once per SKU. During flash sales the same few SKUs are requested thousands of times a second, and the pricing team has asked us to stop hammering them.

Aditya's description:

In-process async cache with a TTL (30 s, agreed with pricing). Concurrent requests for the same SKU are coalesced: the first caller starts the load as a task and everyone else awaits the same task, so a burst makes one upstream call. No lock needed because asyncio is single-threaded. Clock is injectable for tests.

Two facts about the environment: the API gateway cancels a request after 2 s (the handler sees CancelledError), and the pricing service returns occasional 503s during deploys.

Code under review:

# --- checkout/price_cache.py ---
import asyncio
import time


class AsyncTTLCache:
    def __init__(self, loader, ttl: float = 30.0, clock=time.monotonic):
        self._loader = loader              # async def loader(key) -> value
        self._ttl = ttl
        self._clock = clock
        self._values = {}                  # key -> (expires_at, value)
        self._inflight = {}                # key -> asyncio.Task

    async def get(self, key):
        hit = self._values.get(key)
        if hit and hit[0] > self._clock():
            return hit[1]
        task = self._inflight.get(key)
        if task is None:
            task = asyncio.create_task(self._loader(key))
            self._inflight[key] = task
        value = await task
        self._values[key] = (self._clock() + self._ttl, value)
        return value

What a strong reviewer raises:

1. High: Finished tasks are never removed from _inflight. After the first load, the task stays in the dict forever. When the TTL expires, get finds that old, completed task and await task returns its old result immediately, so the cache never refreshes: the TTL check runs, but its only effect is to rewrite the expiry. A price change (or a flash-sale price ending) is never seen until the pod restarts. Worse, a failed task stays too: one 503 during a pricing deploy means every later request for that SKU re-raises the same ConnectionError until restart. Fix: Remove the entry when the load finishes, in a finally inside the task itself (so it happens even if every waiter has gone). Store the value inside the task too, so the cache is filled even when no caller is left to do it.
2. High: One cancelled caller cancels the shared load for everyone. When a coroutine awaiting a task is cancelled, asyncio cancels the awaited task as well. The gateway's 2 s timeout cancels request A, which cancels the coalesced pricing call, and requests B..Z, which were sharing it, all get CancelledError, so one slow client turns into a burst of failures. The busier the SKU, the bigger the blast radius, which is the opposite of what coalescing is for. Fix: await asyncio.shield(task): the cancelled caller stops waiting, but the inner task keeps running for the others. This is why the task must own its own cleanup and cache write (previous issue).
3. Medium: Unbounded memory and no negative caching or jitter. _values grows with every SKU ever seen (a crawler walking the catalogue fills it), and all entries loaded at the start of a sale expire in the same second, producing a synchronised miss storm every 30 s. After the fix, a pricing outage means every request retries upstream immediately. Fix: Bound the cache (an LRU of N entries, or evict expired entries on write), add a small random jitter to the TTL, and consider caching failures briefly (1–2 s) or serving the stale value on upstream error ("stale-if-error") with a metric.
4. Low: if hit and … relies on the tuple being truthy; fine here, but the same habit breaks when a cached value is falsy (0, "") in a cache that stores bare values. Also no metrics: hit/miss/coalesced counts are what tell you whether the cache is doing its job. Fix: if hit is not None, and add counters for hits, misses, coalesced waits and load failures.

Tests that expose the bug:

# --- tests/test_price_cache.py ---
class FakeClock:
    def __init__(self):
        self.now = 1_000.0

    def __call__(self):
        return self.now


async def coalesces_a_burst():
    calls, gate = [], asyncio.Event()

    async def loader(sku):
        calls.append(sku)
        await gate.wait()
        return 49_900                      # paise

    cache = AsyncTTLCache(loader, ttl=30, clock=FakeClock())
    waiters = [asyncio.create_task(cache.get("SKU-1")) for _ in range(5)]
    await asyncio.sleep(0)
    gate.set()
    assert await asyncio.gather(*waiters) == [49_900] * 5
    assert calls == ["SKU-1"]


async def refreshes_after_ttl():
    clock, prices, calls = FakeClock(), iter([49_900, 52_900]), []

    async def loader(sku):
        calls.append(sku)
        return next(prices)

    cache = AsyncTTLCache(loader, ttl=30, clock=clock)
    assert await cache.get("SKU-2") == 49_900
    clock.now += 10
    assert await cache.get("SKU-2") == 49_900 and len(calls) == 1
    clock.now += 31                        # past the TTL: the price change must show up
    assert await cache.get("SKU-2") == 52_900, "stale price served after TTL"


async def retries_after_a_failure():
    attempts = []

    async def flaky(sku):
        attempts.append(sku)
        if len(attempts) == 1:
            raise ConnectionError("pricing-svc 503")
        return 19_900

    cache = AsyncTTLCache(flaky, ttl=30, clock=FakeClock())
    try:
        await cache.get("SKU-3")
    except ConnectionError:
        pass
    else:
        raise AssertionError("first call should fail")
    assert await cache.get("SKU-3") == 19_900, "one 503 must not poison the key"


async def one_cancelled_caller_does_not_fail_the_others():
    gate = asyncio.Event()

    async def slow(sku):
        await gate.wait()
        return 74_900

    cache = AsyncTTLCache(slow, ttl=30, clock=FakeClock())
    first = asyncio.create_task(cache.get("SKU-4"))
    others = [asyncio.create_task(cache.get("SKU-4")) for _ in range(2)]
    await asyncio.sleep(0)
    first.cancel()                         # gateway timeout on one request
    await asyncio.sleep(0)
    gate.set()
    results = await asyncio.gather(*others, return_exceptions=True)
    assert results == [74_900, 74_900], results
    assert first.cancelled()


async def main():
    await coalesces_a_burst()
    await refreshes_after_ttl()
    await retries_after_a_failure()
    await one_cancelled_caller_does_not_fail_the_others()


asyncio.run(asyncio.wait_for(main(), timeout=5))

Fixed version:

# --- checkout/price_cache.py ---
import asyncio
import time


class AsyncTTLCache:
    def __init__(self, loader, ttl: float = 30.0, clock=time.monotonic):
        self._loader = loader              # async def loader(key) -> value
        self._ttl = ttl
        self._clock = clock
        self._values = {}                  # key -> (expires_at, value)
        self._inflight = {}                # key -> asyncio.Task

    async def get(self, key):
        hit = self._values.get(key)
        if hit is not None and hit[0] > self._clock():
            return hit[1]
        task = self._inflight.get(key)
        if task is None:
            task = asyncio.create_task(self._load(key))
            self._inflight[key] = task
        # shield: a cancelled caller stops waiting, but the shared load keeps going for the others.
        return await asyncio.shield(task)

    async def _load(self, key):
        try:
            value = await self._loader(key)
            self._values[key] = (self._clock() + self._ttl, value)
            return value
        finally:
            # Success or failure, the load is over: the next miss starts a fresh one.
            self._inflight.pop(key, None)

Follow-ups:

1. "Aditya says "no lock needed because asyncio is single-threaded". When is that true, and when would it stop being true?" It's true here because there is no await between checking _inflight and inserting the new task, so no other coroutine can run in between: the check-and-set is atomic with respect to the event loop. It stops being true if someone adds an await in that window (for example an async metrics call or an async Redis lookup before creating the task), or if the cache is shared with code running in threads (to_thread, a sync FastAPI route running in the thread pool). Then you need an asyncio.Lock per key, or, across threads, a threading.Lock.
2. "With shield, if every caller is cancelled and the load then fails, what happens?" The task finishes with an exception that nobody retrieves, so asyncio logs "Task exception was never retrieved" when the task is garbage-collected. The cache itself is fine (the finally clears the entry), but the log is noisy. Add a done-callback that retrieves the exception (t.exception()) and records a metric, or keep a strong reference set of background tasks, which the docs also recommend so tasks aren't garbage-collected mid-flight.
3. "We run 40 pods. Is an in-process cache still the right design, or should this be Redis?" In-process coalescing cuts upstream calls to at most one per SKU per pod per TTL, i.e. at most 40 per 30 s for a hot SKU, which is often plenty and costs no network hop. Redis gives one shared copy (consistent prices across pods, fewer upstream calls at large pod counts) but adds a dependency, a network round-trip and its own stampede problem (needs a lock or "early refresh"). A common answer is both: a small in-process L1 with a short TTL in front of a shared L2. The choice depends on how much price inconsistency between pods the business tolerates for 30 s.
4. "How would you have caught the never-refreshes bug before production?" With exactly the test that was missing: inject the clock, advance it past the TTL, and assert a second upstream call and the new value. Tests that only cover "first call misses, second call hits" pass with this bug. More generally, every cache PR should have tests for expiry, failure, and concurrent misses, because those are the paths that break.

Scoring: Strong: Finds both high-severity bugs by reasoning about the task lifecycle: completed tasks stay in _inflight (stale forever, failures cached forever), and cancellation propagates into the awaited task. Fixes with shield plus cleanup inside the task, and explains why the cleanup has to move. Explains when the no-lock claim holds, and raises bounding, jitter and metrics. Acceptable: Finds the missing cleanup (at least the failure-poisoning half) and fixes it with try/finally in get. Gets to the cancellation problem once the gateway timeout is pointed out, even if unsure of the shield API. Red flag: Approves because "coalescing works" (the burst test passes). Proposes wrapping everything in one global asyncio.Lock, which serialises all SKUs behind the slowest load. Doesn't know that cancelling a waiter cancels the awaited task, or says CancelledError "can't happen in FastAPI".

SR-CR-04 · Batched backfill of a new paise column on a live payments table

Hard · Code review · Senior (5–8 years) · migrations, backfills, pagination, transactions, resumability, sqlite3, data quality · 25 min

Context:

PR #877 · migrations/0042_backfill_amount_paise.py · author: Meera · reviewers: you (owner of the payments DB)

Migration 0041 (already deployed) added a nullable amount_paise INTEGER column to payments, which has about 9 million rows. The old column amount_rupees is TEXT and was filled by several generations of code, so it contains values like "1299.50", "1,29,999.00" (Indian digit grouping from a CSV import in 2019) and a few junk values such as "N/A". This PR fills the new column so that 0043 can switch reads.

Meera's description:

Batches of 1,000 so we never lock the table for long. Ordered by id so paging is stable. Safe to rerun: it only touches rows where amount_paise IS NULL. on_batch reports progress to our metrics.

The deploy pipeline runs the script against production (Postgres) while the app is live; the pod running it can be killed by a node rotation at any point, and the pipeline then simply reruns it. The tests use sqlite3, whose connect() opens a transaction implicitly before the first UPDATE, just as a DB-API Postgres driver does.

Code under review:

# --- migrations/0042_backfill_amount_paise.py ---
import sqlite3
from decimal import Decimal

BATCH_SIZE = 1000


def to_paise(text: str) -> int:
    return int(Decimal(text) * 100)


def backfill_amount_paise(conn: sqlite3.Connection, batch_size: int = BATCH_SIZE, on_batch=None) -> int:
    offset = 0
    total = 0
    while True:
        rows = conn.execute(
            "SELECT id, amount_rupees FROM payments WHERE amount_paise IS NULL "
            "ORDER BY id LIMIT ? OFFSET ?",
            (batch_size, offset),
        ).fetchall()
        if not rows:
            break
        for payment_id, rupees in rows:
            conn.execute("UPDATE payments SET amount_paise = ? WHERE id = ?", (to_paise(rupees), payment_id))
        total += len(rows)
        offset += batch_size
        if on_batch:
            on_batch(total)
    conn.commit()
    return total

What a strong reviewer raises:

1. High: OFFSET over a shrinking result set skips rows. The WHERE amount_paise IS NULL filter and the updates interact: after batch 1 fills ids 1–3, those rows leave the filtered set, so OFFSET 3 now skips ids 4–6 and the second batch processes 7–9. Roughly half of each pass is skipped; with 10 rows and batches of 3 only 6 rows are filled, and the loop ends because the offset passes the end. On 9M rows the job reports success with millions of NULLs left, and 0043 then switches reads to a column that is half empty. Fix: Keyset pagination: remember the last id and query WHERE id > :last_id … ORDER BY id LIMIT :n. It's correct under concurrent updates, and it's faster too: OFFSET n makes the database walk and discard n rows on every page, so a 9M-row backfill becomes quadratic.
2. High: One transaction for the whole run. commit() is called once at the end, so all 9M updates sit in one transaction: row locks are held for the entire run (app writes to those rows block, and can time out), on Postgres the table bloats and vacuum can't clean up, replication lag spikes on commit, and when the pod is killed at row 8.9M, everything rolls back and the rerun starts from zero. The "batches so we never lock the table for long" claim is false: batching the reads doesn't batch the transactions. Fix: Commit after each batch, so each batch is a short transaction and finished work is durable. Because the selection is amount_paise IS NULL, a rerun naturally resumes where the last committed batch ended. Consider a short sleep between batches and a lock_timeout / statement_timeout so the backfill backs off instead of blocking the live app.
3. High: One bad row aborts the whole migration. Decimal("1,29,999.00") and Decimal("N/A") raise InvalidOperation, so the job dies at the first legacy value, and because of the single transaction, nothing is saved. Every rerun dies at the same row: the deploy is blocked until someone hand-edits production data under pressure. Fix: Normalise the known legacy format (strip the grouping commas), and for anything still unparseable, skip and report (log the id, count it, leave it NULL for a follow-up data fix) rather than fail the run. 0043 must then gate on "zero NULLs remain" so the skipped rows are dealt with before reads switch.
4. Medium: Silent truncation of money. int(Decimal("45.678") * 100) is 4567, quietly dropping 0.8 paise with no record. A value with more than two decimals is a data-quality signal, not something to round away in a migration. Fix: Reject values that aren't a whole number of paise (paise != paise.to_integral_value()) and report them with the other skipped rows.
5. Medium: Race with the live app. Between the SELECT and the UPDATE, the app (already deploying the dual-write code) may have written a correct amount_paise for that row; the backfill then overwrites it with a value derived from the old column. Fix: Make the write conditional: UPDATE … WHERE id = ? AND amount_paise IS NULL, so the backfill never overwrites a value written by newer code.
6. Low: One execute per row is 9M round-trips to Postgres; on_batch reports rows seen, which hides skipped rows; and the function returns a count that mixes success with nothing else, so the caller can't tell a clean run from a run that skipped 40,000 rows. Fix: executemany per batch (or one UPDATE … FROM (VALUES …)), and report updated and skipped separately in logs and metrics.

Tests that expose the bug:

# --- tests/test_0042.py ---
import logging

logging.disable(logging.CRITICAL)


def make_db(values):
    conn = sqlite3.connect(":memory:")
    conn.execute("CREATE TABLE payments (id INTEGER PRIMARY KEY, amount_rupees TEXT, amount_paise INTEGER)")
    conn.executemany("INSERT INTO payments (id, amount_rupees) VALUES (?, ?)", enumerate(values, start=1))
    conn.commit()
    return conn


def nulls(conn):
    return [r[0] for r in conn.execute("SELECT id FROM payments WHERE amount_paise IS NULL ORDER BY id")]


# 1. Every row is filled, with small batches so paging is exercised.
conn = make_db([f"{i}99.50" for i in range(1, 11)])
backfill_amount_paise(conn, batch_size=3)
assert nulls(conn) == [], f"rows left unfilled: {nulls(conn)}"
assert conn.execute("SELECT amount_paise FROM payments WHERE id = 10").fetchone() == (109950,)


# 2. A pod killed mid-run keeps the batches it finished; the rerun completes the job.
class PodKilled(Exception):
    pass


def kill_after_two_batches(done):
    if done >= 6:
        raise PodKilled


conn = make_db([f"{i}00.00" for i in range(1, 11)])
try:
    backfill_amount_paise(conn, batch_size=3, on_batch=kill_after_two_batches)
except PodKilled:
    conn.rollback()                      # what the server does when the connection drops
assert len(nulls(conn)) == 4, f"work lost on crash: {len(nulls(conn))} rows still NULL"
assert backfill_amount_paise(conn, batch_size=3) == 4
assert nulls(conn) == []

# 3. Legacy formats are converted; junk rows are skipped (left NULL) without aborting the run.
conn = make_db(["1299.50", "1,29,999.00", "N/A", "0.10", "", "45.678"])
backfill_amount_paise(conn, batch_size=2)
got = dict(conn.execute("SELECT id, amount_paise FROM payments"))
assert got == {1: 129950, 2: 12999900, 3: None, 4: 10, 5: None, 6: None}, got

Fixed version:

# --- migrations/0042_backfill_amount_paise.py ---
import logging
import sqlite3
from decimal import Decimal, InvalidOperation

BATCH_SIZE = 1000
logger = logging.getLogger("migrations.0042")


def to_paise(text: str) -> int | None:
    """Parse legacy rupee text; return None for values a human must look at."""
    try:
        rupees = Decimal(text.replace(",", "").strip())
    except (InvalidOperation, AttributeError):
        return None
    paise = rupees * 100
    if not paise.is_finite() or paise != paise.to_integral_value():
        return None                      # e.g. "45.678": don't guess which way to round money
    return int(paise)


def backfill_amount_paise(conn: sqlite3.Connection, batch_size: int = BATCH_SIZE, on_batch=None) -> int:
    last_id = 0
    updated = 0
    skipped = 0
    while True:
        # Keyset pagination: resume after the last id we saw. OFFSET is wrong here because
        # the rows we update drop out of the WHERE clause.
        rows = conn.execute(
            "SELECT id, amount_rupees FROM payments WHERE id > ? AND amount_paise IS NULL "
            "ORDER BY id LIMIT ?",
            (last_id, batch_size),
        ).fetchall()
        if not rows:
            break
        last_id = rows[-1][0]
        params = []
        for payment_id, rupees in rows:
            paise = to_paise(rupees)
            if paise is None:
                skipped += 1
                logger.warning("payment %s: unparseable amount_rupees %r, left NULL", payment_id, rupees)
            else:
                params.append((paise, payment_id))
        conn.executemany(
            "UPDATE payments SET amount_paise = ? WHERE id = ? AND amount_paise IS NULL", params)
        conn.commit()                    # one short transaction per batch: finished work survives a crash
        updated += len(params)
        if on_batch:
            on_batch(updated + skipped)
    logger.info("0042 done: updated=%d skipped=%d", updated, skipped)
    return updated

Follow-ups:

1. "Why doesn't the existing test suite catch the OFFSET bug? What test would you require?" With the default batch_size=1000 and a test table of a few dozen rows, everything fits in one batch, so paging never happens. The required test uses a batch size smaller than the table (and not a divisor of it) and asserts zero NULLs remain, plus a crash-and-rerun test that asserts finished batches survive. Tests of batched code must exercise more than one batch.
2. "Meera asks why not just UPDATE payments SET amount_paise = CAST(…) WHERE amount_paise IS NULL in one statement. When is that fine and when isn't it?" For a small table it's simplest and fine. For 9M rows on a live table it's one huge transaction (the same locking, bloat, replication-lag and all-or-nothing problems as above), and the parsing rules (Indian digit grouping, rejecting sub-paisa amounts) are awkward and error-prone in SQL. Batched, committed, resumable chunks, throttled to watch replication lag, are the standard for live backfills; a single statement is acceptable in a maintenance window or for tables that fit in seconds.
3. "How does this backfill fit into a zero-downtime rollout of the new column?" Expand/contract: (1) add the nullable column (0041); (2) deploy code that dual-writes both columns for new rows; (3) run this backfill for old rows (conditional writes so it never overwrites dual-written values); (4) verify zero NULLs and that the columns agree on a sample; (5) switch reads; (6) enforce NOT NULL without a long lock (on Postgres: add CHECK (amount_paise IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, then SET NOT NULL, which Postgres 12+ can do without rescanning the table) and later drop the old column. Each step is separately deployable and reversible.
4. "How would you watch this job while it runs in production?" Progress metrics (rows updated and skipped, rows per second, current last_id against max(id) for an ETA), DB health (replication lag, lock waits, p99 latency of the app's queries on payments), and a kill switch or throttle setting the job reads between batches so on-call can slow or pause it without killing it mid-batch.

Scoring: Strong: Traces the OFFSET loop with concrete ids and shows rows are skipped, and proposes keyset pagination with the performance argument as well. Spots the single transaction and explains its effect on locks, crash recovery and the "safe to rerun" claim. Treats bad legacy data as an expected case (skip-and-report, gate the next migration) and adds the conditional UPDATE for the dual-write race. Asks for a test with more than one batch. Acceptable: Finds either the OFFSET bug or the single commit without hints, and the other with a nudge ("what happens after batch one?"). Handles the comma format but would let N/A fail the run, or needs prompting on the dual-write race. Red flag: Accepts "ordered by id so paging is stable" at face value. Suggests wrapping the whole job in one transaction "for safety". Fixes bad data by except Exception: continue around the whole batch (silently losing 1,000 rows). Has no plan for how the column becomes the source of truth.

SR-CR-05 · Dynamic micro-batching in front of a ranking model

Hard & Tricky · Code review · Senior (5–8 years) · model serving, asyncio, micro-batching, futures, failure isolation, cancellation, backpressure, FastAPI · 30 min

Context:

PR #3051 · serving/batcher.py, serving/app.py · author: Rohan · reviewers: you (owner of the search-ranking service)

The search-ranking service scores product titles with a model that is ~20x cheaper per item when called on a batch (vectorised numpy; it releases the GIL). Today every HTTP request calls the model with a batch of one. This PR adds a micro-batcher: each request enqueues its title and awaits a future; a single background worker collects up to max_batch items or waits at most max_wait seconds, runs the model in a thread, and resolves the futures.

Rohan's description:

Load test: p99 down from 180 ms to 45 ms at 2,000 RPS. I also dedupe identical titles inside a batch, because the same popular titles ("Redmi Note 13 5G") show up many times per batch. The model is a pure function, so this is safe.

Facts you know as the owner: the model raises ValueError on some malformed inputs (a few per hour); the API gateway cancels requests after 800 ms (the handler then sees CancelledError); traffic doubles during sale events.

Code under review:

# --- serving/batcher.py ---
import asyncio


class MicroBatcher:
    """Turns many concurrent predict() calls into a few batched model calls."""

    def __init__(self, model_fn, max_batch: int = 32, max_wait: float = 0.005):
        self._model_fn = model_fn          # sync: list[str] -> list[float], same order as input
        self._max_batch = max_batch
        self._max_wait = max_wait
        self._queue: asyncio.Queue = asyncio.Queue()
        self._worker: asyncio.Task | None = None

    def start(self) -> None:
        self._worker = asyncio.create_task(self._run())

    async def predict(self, text: str) -> float:
        fut = asyncio.get_running_loop().create_future()
        await self._queue.put((text, fut))
        return await fut

    async def _run(self) -> None:
        loop = asyncio.get_running_loop()
        while True:
            batch = [await self._queue.get()]
            deadline = loop.time() + self._max_wait
            while len(batch) < self._max_batch:
                remaining = deadline - loop.time()
                if remaining <= 0:
                    break
                try:
                    batch.append(await asyncio.wait_for(self._queue.get(), remaining))
                except TimeoutError:
                    break
            unique = list(dict.fromkeys(text for text, _ in batch))     # dedupe, keep order
            scores = await asyncio.to_thread(self._model_fn, unique)
            for i, (_, fut) in enumerate(batch):
                fut.set_result(scores[i])


# --- serving/app.py ---
from contextlib import asynccontextmanager

from fastapi import FastAPI
from pydantic import BaseModel


def score_titles(texts: list[str]) -> list[float]:
    return [min(len(t) / 100, 1.0) for t in texts]      # stand-in for the real model


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.batcher = MicroBatcher(score_titles)
    app.state.batcher.start()
    yield


app = FastAPI(lifespan=lifespan)


class ScoreRequest(BaseModel):
    title: str


@app.post("/score")
async def score(req: ScoreRequest) -> dict:
    return {"score": await app.state.batcher.predict(req.title)}

What a strong reviewer raises:

1. High: Dedup misaligns scores with callers. scores lines up with unique, but the loop indexes it by position in batch. For the batch ["saree", "kurta", "saree", "pickle"] the model sees ["saree", "kurta", "pickle"]; the second "saree" caller (i=2) receives pickle's score, and the "pickle" caller (i=3) hits IndexError. So the PR silently returns the wrong ranking score to some users (no error, no log), and in the same batch crashes the worker (next issue). The load test didn't catch it because it compared latency, not answers. Fix: Map by key, not position: by_text = dict(zip(unique, scores)) and fut.set_result(by_text[text]). Validate len(scores) == len(unique) so a model that drops items fails loudly. Require a test with duplicates whose expected values differ by position.
2. High: Any exception kills the worker, and then every request hangs forever. _run has no error handling. A ValueError from the model (a few per hour), the IndexError above, or an InvalidStateError (next issue) ends the task. Its exception is stored on a task nobody awaits, so nothing is logged until garbage collection; the futures in that batch never resolve, and every later request waits until the gateway's 800 ms timeout. The process stays up and /health still returns 200, so the orchestrator never restarts it: a full outage that looks like latency. Fix: Catch exceptions around the model call and set_exception on that batch's futures (the callers get a 500 for their own bad input), then continue the loop. Add a done-callback on the worker task that logs and fails health checks if it ever exits, and a liveness probe that checks the worker is alive and the queue is moving.
3. High: A cancelled caller crashes the worker. When the gateway cancels a request, await fut in predict is cancelled, which cancels the future. When the batch finishes, fut.set_result(...) on a cancelled future raises asyncio.InvalidStateError, which kills the worker (previous issue). So under load, when timeouts become common, the first timeout in any batch takes the whole service down. Fix: Only resolve futures that aren't done (if not fut.done()), and drop already-cancelled requests before calling the model so the batch doesn't spend GPU/CPU on answers nobody will read.
4. Medium: Unbounded queue, no backpressure. During a sale at 2x traffic, if the model can't keep up, the queue grows without limit; every request waits longer than the 800 ms gateway timeout, so the service does full work for requests that are already cancelled and its effective throughput drops to zero (with the cancellation bug fixed, these are at least skipped). Fix: Bounded queue with put_nowait and an explicit Overloaded → HTTP 503 so clients retry or degrade (e.g. fall back to popularity ranking). Size it from the latency budget: queue capacity ≈ throughput × acceptable wait. Export queue depth and batch size as metrics.
5. Low: No shutdown path: on deploy, lifespan exits without stopping the worker or failing queued futures, so in-flight requests hang until the process is killed instead of failing fast. Fix: A stop() that cancels the worker and fails pending futures, called after yield in lifespan.

Tests that expose the bug:

# --- tests/test_batcher.py ---
import threading

SCORES = {"saree": 0.91, "kurta": 0.42, "pickle": 0.13, "slow": 0.5}


async def main():
    batches = []
    model_started, release = threading.Event(), threading.Event()

    def model(texts):
        batches.append(list(texts))
        if "corrupt" in texts:
            raise ValueError("tokenizer failed on input")
        if "slow" in texts:
            model_started.set()
            release.wait(2)
        return [SCORES[t] for t in texts]

    b = MicroBatcher(model, max_batch=8, max_wait=0.01)
    b.start()

    async def ask(text):
        return await asyncio.wait_for(b.predict(text), 1)

    # 1. Duplicates inside a batch: every caller gets the score for ITS title; the model sees each once.
    got = await asyncio.gather(*(ask(t) for t in ["saree", "kurta", "saree", "pickle"]))
    assert got == [0.91, 0.42, 0.91, 0.13], got
    assert batches[-1] == ["saree", "kurta", "pickle"], batches[-1]

    # 2. A model error fails that batch's callers only; the service keeps serving.
    try:
        await ask("corrupt")
    except ValueError:
        pass
    else:
        raise AssertionError("model error was not propagated to the caller")
    assert await ask("kurta") == 0.42

    # 3. A caller that goes away mid-batch must not take the worker down.
    gone = asyncio.create_task(b.predict("slow"))
    stays = asyncio.create_task(ask("kurta"))
    while not model_started.is_set():
        await asyncio.sleep(0.005)
    gone.cancel()                          # gateway timeout while the model is running
    await asyncio.sleep(0)
    release.set()
    assert await stays == 0.42
    assert await ask("pickle") == 0.13


asyncio.run(main())

Fixed version:

# --- serving/batcher.py ---
import asyncio


class Overloaded(Exception):
    """Queue is full: shed load (HTTP 503) instead of queueing past the latency budget."""


class MicroBatcher:
    """Turns many concurrent predict() calls into a few batched model calls."""

    def __init__(self, model_fn, max_batch: int = 32, max_wait: float = 0.005, max_queue: int = 1024):
        self._model_fn = model_fn          # sync: list[str] -> list[float], same order as input
        self._max_batch = max_batch
        self._max_wait = max_wait
        self._queue: asyncio.Queue = asyncio.Queue(maxsize=max_queue)
        self._worker: asyncio.Task | None = None

    def start(self) -> None:
        self._worker = asyncio.create_task(self._run())

    async def stop(self) -> None:
        if self._worker:
            self._worker.cancel()
        while not self._queue.empty():     # don't leave callers hanging at shutdown
            _, fut = self._queue.get_nowait()
            if not fut.done():
                fut.set_exception(Overloaded("shutting down"))

    async def predict(self, text: str) -> float:
        fut = asyncio.get_running_loop().create_future()
        try:
            self._queue.put_nowait((text, fut))
        except asyncio.QueueFull:
            raise Overloaded("ranking queue full") from None
        return await fut

    async def _collect(self) -> list:
        loop = asyncio.get_running_loop()
        batch = [await self._queue.get()]
        deadline = loop.time() + self._max_wait
        while len(batch) < self._max_batch:
            remaining = deadline - loop.time()
            if remaining <= 0:
                break
            try:
                batch.append(await asyncio.wait_for(self._queue.get(), remaining))
            except TimeoutError:
                break
        return batch

    async def _run(self) -> None:
        while True:
            batch = await self._collect()
            batch = [(text, fut) for text, fut in batch if not fut.done()]   # drop callers already gone
            if not batch:
                continue
            unique = list(dict.fromkeys(text for text, _ in batch))
            try:
                scores = await asyncio.to_thread(self._model_fn, unique)
                if len(scores) != len(unique):
                    raise RuntimeError(f"model returned {len(scores)} scores for {len(unique)} inputs")
            except Exception as exc:
                for _, fut in batch:
                    if not fut.done():
                        fut.set_exception(exc)
                continue
            by_text = dict(zip(unique, scores))
            for text, fut in batch:
                if not fut.done():         # the caller may have been cancelled while the model ran
                    fut.set_result(by_text[text])


# --- serving/app.py ---
from contextlib import asynccontextmanager

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel


def score_titles(texts: list[str]) -> list[float]:
    return [min(len(t) / 100, 1.0) for t in texts]      # stand-in for the real model


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.batcher = MicroBatcher(score_titles)
    app.state.batcher.start()
    yield
    await app.state.batcher.stop()


app = FastAPI(lifespan=lifespan)


class ScoreRequest(BaseModel):
    title: str


@app.post("/score")
async def score(req: ScoreRequest) -> dict:
    try:
        return {"score": await app.state.batcher.predict(req.title)}
    except Overloaded:
        raise HTTPException(status_code=503, detail="overloaded, retry later") from None

Follow-ups:

1. "How do you choose max_batch and max_wait? What's the trade-off?" max_wait is latency you add to the first request in every batch, so it comes out of the p99 budget; max_batch is bounded by memory and by the point where per-batch model latency stops scaling sublinearly. Measure model latency as a function of batch size, then pick the largest batch whose latency plus max_wait fits the budget at peak RPS. At low traffic a batch rarely fills, so max_wait dominates; some teams make it adaptive (don't wait when the queue is empty and the model is idle).
2. "Why asyncio.to_thread for the model? When would you use a process pool or a separate model server instead?" to_thread keeps the event loop free to accept and enqueue requests while the model runs, and it works because numpy/BLAS release the GIL during the heavy math. If the model is pure Python (holds the GIL), threads add no parallelism and the loop stalls, so a process pool or a dedicated model server (Triton, TorchServe, a separate gRPC service) is needed. A separate server also lets you scale model replicas and the API independently and use GPUs efficiently, at the cost of a network hop and another deployable.
3. "The model call is CPU-heavy. Why is there exactly one worker, and what would you change on a 16-core box?" One worker serialises model calls, which is right for a single GPU or a model that already uses all cores through BLAS threads. On a 16-core box with a single-threaded model, run N workers consuming the same queue (or N uvicorn worker processes, each with its own batcher), and cap BLAS threads per worker to avoid oversubscription. Measure throughput as N increases; it usually peaks below the core count.
4. "How would you roll this out safely, given the misalignment bug got past a load test?" Shadow mode first: run the batched path alongside the existing path for a slice of traffic and compare scores per request (they must be identical for a pure model), alerting on any mismatch. Then a canary with latency, error-rate and score-distribution metrics, and a feature flag to fall back to unbatched calls. Correctness checks belong in the rollout, not just latency numbers.

Scoring: Strong: Finds the dedup misalignment by tracing a concrete batch, and recognises that wrong answers with no error are worse than a crash. Finds that an uncaught exception kills the single worker and that the service then hangs while looking healthy, and connects the gateway timeout to InvalidStateError on a cancelled future. Proposes per-batch error isolation, fut.done() checks, a bounded queue with 503s, worker liveness monitoring, and a shadow-mode rollout. Acceptable: Finds the misalignment and the missing error handling around the model call. Gets to the cancelled future issue with a hint ("what happens to fut when the client times out?"). Mentions backpressure only when asked about sale traffic. Red flag: Approves on the strength of the p99 numbers. Thinks dedup is "obviously safe because the model is pure" without checking the indexing. Proposes except Exception: pass inside the loop (callers hang instead of the worker dying). Can't explain what happens to a background task's exception in asyncio.

SR-TH-ASY-01 · When not to make a service async

Easy · Theory · Senior (5–8 years) · asyncio, architecture, threads vs asyncio, function colouring, trade-offs · 5 min

Question: A team proposes rewriting a Flask service (sync, gunicorn with threads) in async FastAPI "for performance". It does CRUD on PostgreSQL via SQLAlchemy and calls two internal APIs, at about 80 requests per second. How would you evaluate the proposal?

Answer:

Async buys one thing: cheap concurrency for waiting. Thousands of in-flight requests that mostly wait on sockets cost a coroutine each instead of a thread each. It doesn't make any single request faster and doesn't help CPU-bound work.

So ask what limits the service today:

• Needed concurrency. 80 req/s × ~100 ms ≈ 8 requests in flight. A sync server with a few workers × threads handles that easily. Async pays off at hundreds or thousands of concurrent waits: websockets, long-polling, streaming, fan-out to many upstreams per request.
• Where time goes. If p99 is dominated by slow queries or CPU (serialisation, validation), async changes nothing; fix the queries or scale processes.

Costs to weigh:

• It's all-or-nothing down the stack ("function colouring"). Every I/O call on the request path must be async: the DB driver (asyncpg / SQLAlchemy's async engine), HTTP clients, caches, SDKs. One forgotten sync call blocks the event loop and stalls every request in that process, which is a worse failure mode than a blocked thread.
• Migration and review cost: two styles in one codebase, async-aware testing, harder debugging (cancellation, tracebacks across tasks), and people who need to learn new failure modes.
• Library gaps: some internal SDKs and drivers are sync-only, so you'd end up with to_thread wrappers.

A reasonable conclusion: keep the sync service and measure; adopt async where the workload is genuinely high-concurrency I/O (a notification gateway, a websocket hub, an API aggregator), possibly as a separate service. "Async for performance" without a concurrency problem is a cost without a benefit.

Follow-ups:

1. "FastAPI supports plain def endpoints. Doesn't that make migration incremental?" Yes. def endpoints run in a threadpool, so you can move framework first and convert hot paths later. But it's only safe if nobody writes async def endpoints that call sync libraries; that mix is where event-loop blocking bugs come from. Set a lint/review rule for it.
2. "What metric would convince you async is needed?" Thread or worker saturation while CPU is low: requests queueing for a free worker, with most of each request's time spent waiting on I/O, and memory limiting how many threads/workers you can add. Then async (or more processes) addresses the actual constraint.

Scoring: Strong: Frames async as cheap concurrency for I/O waits, does a concurrency estimate, checks where time actually goes, and weighs the whole-stack conversion, blocking-call risk and team cost. Gives a reasoned recommendation with a metric that would change it. Acceptable: Knows async helps I/O-bound high concurrency and not CPU work, but doesn't quantify or discuss migration cost. Red flag: Assumes async is faster per request, or plans to keep sync DB calls inside async def handlers.

SR-TH-ASY-02 · An OTP rate limiter that races in single-threaded asyncio

Easy & Tricky · Theory · Senior (5–8 years) · asyncio, race conditions, check-then-act, asyncio.Lock, rate limiting, Redis · 5 min

Question: An async auth service allows at most 3 OTPs per phone number per 10 minutes, tracked in an in-memory dict: check the count, await the SMS gateway, then record the send. A reviewer asks for a lock; the author says "asyncio is single-threaded, so there are no races". Who's right?

Answer:

Trap: "Single-threaded means no races." In asyncio, a coroutine runs uninterrupted only between awaits. Any await can hand control to another task. Here the check (count < 3) and the act (record the send) are separated by await send_sms(...), so five concurrent requests all pass the check before any of them records a send: five OTPs go out, as below. It's the classic check-then-act race; asyncio only makes the switch points explicit.

Fixes, in order of preference:

• No await between check and act: reserve first (append a pending entry, then await the send, and remove the entry if the send fails). The decision and the state update happen atomically from the loop's point of view.
• asyncio.Lock per key around the whole sequence, as below. Bound the lock dict (evict idle keys) or it grows with every phone number, and hold locks only as long as needed.
• But in production this state can't be in-process at all. With 4 workers × 6 pods, each process enforces its own "3", i.e. up to 72 OTPs. Use Redis: INCR otp:{phone} with EXPIRE, or a small Lua script for a sliding window, so the check-and-increment is atomic across the fleet.

Reviewer guidance: in async code, read every await as "anything can change here", and look for decisions made before an await and acted on after it.

import asyncio

sent: dict[str, list[int]] = {}
LIMIT = 3


async def send_sms(phone):
    await asyncio.sleep(0)                     # the SMS gateway call


async def request_otp(phone, n):
    if len(sent.get(phone, [])) < LIMIT:       # check
        await send_sms(phone)                  # other requests run here
        sent.setdefault(phone, []).append(n)   # act on a stale check
        return "sent"
    return "rate-limited"


async def main():
    results = await asyncio.gather(*(request_otp("98450xxxxx", n) for n in range(5)))
    print(results, len(sent["98450xxxxx"]))

    sent.clear()
    locks: dict[str, asyncio.Lock] = {}

    async def request_otp_locked(phone, n):
        async with locks.setdefault(phone, asyncio.Lock()):   # serialise per phone
            return await request_otp(phone, n)

    results = await asyncio.gather(*(request_otp_locked("98450xxxxx", n) for n in range(5)))
    print(results, len(sent["98450xxxxx"]))


asyncio.run(main())

Output:

['sent', 'sent', 'sent', 'sent', 'sent'] 5
['sent', 'sent', 'sent', 'rate-limited', 'rate-limited'] 3

Follow-ups:

1. "With "reserve first", what happens if the SMS gateway call is cancelled by a request timeout?" CancelledError is raised at the await; the rollback must run in except BaseException: or finally (and re-raise) so a cancelled send releases its reservation. Whether a cancelled send counts is a product decision: the SMS may already have been delivered.
2. "Is locks.setdefault(phone, asyncio.Lock()) itself race-free?" Yes in asyncio: it has no await, so it runs atomically with respect to other tasks. The same line in threaded code would also be fine under the GIL for a dict, but the general check-then-act rule applies to anything spanning an await or a thread switch.

Scoring: Strong: Explains that tasks interleave at await points, predicts 5 sends, fixes it with reserve-before-await or a per-key lock, and points out that per-process state is wrong across workers, proposing an atomic Redis counter. Acceptable: Agrees with the reviewer and adds an asyncio.Lock, but doesn't address multiple workers. Red flag: Agrees there's no race because asyncio is single-threaded, or adds a threading.Lock inside the coroutine.

SR-TH-ASY-03 · def vs async def endpoints in FastAPI, and the 40-thread limit

Medium · Theory · Senior (5–8 years) · FastAPI, asyncio, threadpool, AnyIO, blocking I/O, capacity · 6 min

Question: In a FastAPI service, some endpoints are async def and some are plain def. Explain how each kind runs, how you decide which to use, and what happens when 200 concurrent requests hit a def endpoint whose DB call takes 2 seconds.

Answer:

How they run:

• async def endpoints run directly on the event loop. They must only do non-blocking work and await async libraries. Any blocking call (sync DB driver, requests, time.sleep, heavy CPU) freezes the loop and every other request in that worker.
• def endpoints (and sync dependencies) run in a threadpool via AnyIO's to_thread, so blocking is fine. That pool has a default limit of 40 threads per process, shared by all sync endpoints, sync dependencies and other run_in_threadpool users.

Rule of thumb: use async def only when everything inside is async (asyncpg, httpx.AsyncClient, async Redis); use def when the code uses sync libraries (sync SQLAlchemy sessions, boto3, most SDKs). The dangerous combination is async def calling a sync library.

The 200-request scenario: 40 requests occupy the 40 threads for 2 s each; the other 160 wait for a thread token, so latencies stack up in waves (2 s, 4 s, ...), up to 10 s. Health checks that are also sync def endpoints queue behind them and time out, so the orchestrator may restart a pod that is just slow. CPU stays low, which is a useful diagnostic signal.

What to do about it:

• Fix the 2-second query first; no concurrency model hides it.
• Size deliberately: raise the limiter (anyio.to_thread.current_default_thread_limiter().total_tokens = N) only alongside a matching DB pool size, and remember total DB connections = workers × pods × pool size.
• Keep health/readiness endpoints async def and dependency-free so they don't queue behind work.
• Or move that path to async drivers so waiting doesn't need threads.

Follow-ups:

1. "A def endpoint returns a StreamingResponse over a sync generator that reads a big file. Any concern?" Starlette iterates a sync generator in the threadpool chunk by chunk, so a slow client keeps taking thread tokens throughout the download. Many concurrent downloads can starve the pool. Use an async generator with async file I/O, or offload large downloads to object storage with pre-signed URLs.
2. "How would you detect in production that an async def endpoint is blocking the loop?" Event-loop lag metrics (a task measuring how late sleep wakes up), asyncio debug mode's slow-callback warnings in staging, and correlated latency spikes across unrelated endpoints in the same worker. py-spy dump during a spike shows the blocking call on the main thread.

Scoring: Strong: Explains loop vs threadpool execution, the shared 40-token limit, the queueing waves in the scenario and the health-check side effect, and ties pool sizing to DB pool limits. Gives a clear rule for choosing def vs async def. Acceptable: Knows sync endpoints run in a threadpool and async ones on the loop, but not the limit or its consequences. Red flag: Thinks async def makes a sync DB call non-blocking, or makes every endpoint async def by default.

SR-TH-ASY-04 · Feeding an asyncio.Queue from a Kafka consumer thread

Medium & Tricky · Theory · Senior (5–8 years) · asyncio, thread safety, call_soon_threadsafe, run_coroutine_threadsafe, threads and event loops · 6 min

Question: An async service uses a sync Kafka client running in a background thread. The thread calls inbox.put_nowait(msg) on an asyncio.Queue that a coroutine is awaiting. Messages sometimes sit unprocessed for seconds, and occasionally the consumer seems stuck until another request arrives. What's wrong, and how should the thread hand messages to the loop?

Answer:

Trap: "asyncio.Queue is a queue, so it's thread-safe like queue.Queue." Almost nothing in asyncio is thread-safe. asyncio.Queue, Lock, Event, futures and the loop's scheduling methods assume they're used from the loop's own thread. When the Kafka thread calls put_nowait, it appends to the deque and resolves the waiting getter's future from the wrong thread. That schedules the getter's wake-up on the loop's ready queue, but nothing wakes the loop from its select()/epoll wait, so the consumer resumes only when some other I/O event happens to wake the loop. Hence the multi-second delays and "stuck until another request arrives". Unsynchronised access from two threads can also corrupt the internal state.

Correct hand-offs from a foreign thread:

• loop.call_soon_threadsafe(inbox.put_nowait, msg): schedules the call on the loop's thread and writes to the loop's self-pipe to wake it. Capture loop = asyncio.get_running_loop() in the loop thread and pass it to the worker thread.
• asyncio.run_coroutine_threadsafe(coro, loop) when the thread needs a result back: it returns a concurrent.futures.Future the thread can block on with a timeout.
• The other direction (loop → blocking code) is await asyncio.to_thread(fn) or loop.run_in_executor.

Also add backpressure: call_soon_threadsafe on an unbounded queue lets a fast producer outrun the consumer. Use a bounded asyncio.Queue and have the thread wait on run_coroutine_threadsafe(inbox.put(msg), loop).result() so it blocks when the loop falls behind. Running the loop with debug=True in tests makes many wrong-thread calls raise RuntimeError instead of misbehaving silently. Better still, use an async Kafka client.

import asyncio
import threading


async def main():
    loop = asyncio.get_running_loop()
    inbox: asyncio.Queue[str] = asyncio.Queue()

    def kafka_consumer_thread():
        # Wrong: inbox.put_nowait(msg) from this thread is not thread-safe and
        # may not wake the waiting consumer. Hand the call to the loop instead:
        for msg in ["order-1", "order-2"]:
            loop.call_soon_threadsafe(inbox.put_nowait, msg)

        async def lookup(sku):
            return f"price({sku})=499"
        fut = asyncio.run_coroutine_threadsafe(lookup("SKU-7"), loop)
        print("thread got", fut.result(timeout=5))

    t = threading.Thread(target=kafka_consumer_thread)
    t.start()
    print("loop got", await inbox.get())
    print("loop got", await inbox.get())
    await asyncio.to_thread(t.join)


asyncio.run(main())

Output:

loop got order-1
loop got order-2
thread got price(SKU-7)=499

Follow-ups:

1. "Why not just use a queue.Queue and have the coroutine call q.get()?" queue.Queue.get() blocks the thread, which is the event loop's thread, so the whole service stops while waiting. You could await asyncio.to_thread(q.get), but that ties up an executor thread per waiting consumer. The call_soon_threadsafe bridge (or a library like janus with sync and async faces) keeps the loop free.
2. "The thread calls fut.result() with no timeout and the loop is shutting down. What happens?" If the loop stops before running the coroutine, the future never completes and the thread blocks forever, preventing a clean exit. Always pass a timeout, and on shutdown signal the thread to stop before closing the loop.

Scoring: Strong: States asyncio primitives aren't thread-safe, explains the missing loop wake-up and data race, uses call_soon_threadsafe/run_coroutine_threadsafe correctly with a captured loop, and adds backpressure plus debug-mode detection. Acceptable: Knows to use call_soon_threadsafe but can't explain the symptom of delayed processing. Red flag: Wraps put_nowait in a threading.Lock, or calls asyncio.run() inside the thread to "use the queue".

SR-TH-ASY-05 · Finding what blocks the event loop in production

Hard · Theory · Senior (5–8 years) · asyncio, event loop lag, observability, debug mode, py-spy, performance · 8 min

Question: An async API gateway shows periodic p99 spikes to 2-3 s across all endpoints at once, while CPU averages 30%. You suspect something blocks the event loop. How do you prove it and find the culprit in production, and what are the usual suspects?

Answer:

Simultaneous latency spikes across unrelated endpoints in the same worker are the signature of a blocked loop: while one callback runs without yielding, no other task makes progress, timers fire late and even timeouts are delayed.

Prove it with a lag metric. Run a background task that sleeps for a fixed interval (say 250 ms) and records how late it woke up (loop.time() after minus expected). Export it as a histogram per worker. A healthy loop shows lag in single-digit milliseconds; spikes to seconds that line up with the latency spikes are your proof. This is cheap enough to run always.

Find the culprit:

• asyncio debug mode (PYTHONASYNCIODEBUG=1 or asyncio.run(main(), debug=True)) logs every callback or task step slower than loop.slow_callback_duration (100 ms by default) with the task's coroutine. It has real overhead, so use it in staging or on one canary instance with production-like traffic.
• py-spy dump during a spike shows the main thread's stack at the moment it's blocked; py-spy record over a few minutes gives a flame graph where blocking calls show up on the loop thread.
• Tracing spans that include loop lag at span start help find which request kind precedes the stalls.

Usual suspects: a sync client hidden in a helper or SDK (requests, boto3, a sync Redis call, a DB driver); CPU work on the loop (serialising a large JSON response, pydantic validation of big payloads, regexes on large strings, password hashing with bcrypt, gzip); synchronous logging handlers writing to a network or a slow disk; file I/O; big sorted() or pandas operations in a handler.

Fixes: async libraries for I/O; asyncio.to_thread for blocking calls that release the GIL; a process pool or a separate service for CPU-heavy work; chunk large CPU work with await asyncio.sleep(0) between pieces if it must stay inline; QueueHandler/QueueListener for logging. Then keep the lag metric and an alert on it so regressions are caught by monitoring, not by users.

Follow-ups:

1. "Why might moving bcrypt hashing to asyncio.to_thread not fully fix the latency?" It frees the loop only if the hashing releases the GIL; many bcrypt implementations do, but pure-Python CPU work in a thread still competes for the GIL with the loop thread, so the loop gets slower in proportion. CPU-heavy work belongs in a process pool or a separate service, and the default thread pool's size limits concurrency too.
2. "Debug mode reports "Executing took 0.9 seconds". The handler is all awaits. How is that possible?" The 0.9 s is time between two await points inside it, i.e. synchronous code that runs in one step: a loop over results, a big json.dumps, or a call into a library function that turns out to be sync. Every await that returns without suspending also counts as part of the same step.

Scoring: Strong: Explains the blocked-loop signature, proposes an always-on loop-lag metric, uses debug mode safely and py-spy for attribution, lists realistic culprits including CPU and logging, and matches fixes to cause (threads vs processes vs async libraries). Acceptable: Knows a sync call can block the loop and suggests debug mode or profiling, without a production-safe metric. Red flag: Adds more workers or bigger instances without finding the blocking call, or blames the network.

SR-TH-ASY-06 · An await that never yields starves the heartbeat task

Hard & Tricky · Theory · Senior (5–8 years) · asyncio, scheduling, starvation, asyncio.Queue, heartbeats, fairness · 7 min

Question: After a slow deploy, an async consumer has 10,000 messages backed up in an asyncio.Queue. While it drains them, its heartbeat task stops running, the broker declares the consumer dead and triggers a rebalance. The drain loop does await queue.get() on every iteration. How can a loop full of awaits starve other tasks?

Answer:

Trap: "Every await gives control back to the event loop." An await only suspends the task if the thing being awaited isn't ready. await some_coroutine() simply runs the coroutine inline; the task yields to the loop only when, somewhere down the chain, it awaits a future that isn't done yet. Queue.get() on a non-empty queue returns immediately without suspending. So while the backlog lasts, the drain loop runs 10,000 iterations in one task step: no other task, timer, or I/O callback runs until it's finished. The heartbeat (and health checks, timeouts, other consumers) is starved, exactly as in the first output line below.

The same happens with an uncontended asyncio.Lock, a StreamReader that already has buffered data, async for over an async generator that never truly waits, or a cache hit wrapped in a coroutine.

Fixes:

• Yield explicitly in long loops: await asyncio.sleep(0) every N items (second output below). It's the idiomatic "let others run" in asyncio.
• Bound the work per step: process in batches and yield between batches; keep CPU work per item small, or move it to a process pool.
• Don't put liveness on the same loop as heavy work when it matters: some clients run heartbeats in a separate thread for exactly this reason. Configure session/poll timeouts with backlog processing time in mind.
• Measure event-loop lag; this kind of starvation shows up as lag even though no single call is "blocking".

import asyncio

log = []


async def heartbeat():
    for _ in range(3):
        log.append("heartbeat")
        await asyncio.sleep(0)


async def drain(queue, yield_every=None):
    n = 0
    while not queue.empty():
        await queue.get()                  # an item is ready: returns WITHOUT suspending
        n += 1
        if yield_every and n % yield_every == 0:
            await asyncio.sleep(0)         # explicitly give other tasks a turn
    log.append(f"drained {n}")


async def run(yield_every):
    log.clear()
    q = asyncio.Queue()
    for i in range(10_000):
        q.put_nowait(i)                    # a backlog built up during a slow deploy
    hb = asyncio.create_task(heartbeat())
    await drain(q, yield_every)
    await hb
    print(log)


asyncio.run(run(yield_every=None))
asyncio.run(run(yield_every=2_500))

Output:

['drained 10000', 'heartbeat', 'heartbeat', 'heartbeat']
['heartbeat', 'heartbeat', 'heartbeat', 'drained 10000']

Follow-ups:

1. "Is await asyncio.sleep(0) after every single item a good idea?" It's correct but costs a trip through the event loop per item, which can be a significant overhead for tiny items. Yield every N items or every few milliseconds of work (check loop.time()), trading a little latency for others against throughput.
2. "Why didn't the heartbeat at least run once before the drain started?" create_task only schedules the task; it starts running at the next loop iteration. The current task went straight into drain and never suspended, so the heartbeat's first step waited until run reached await hb.

Scoring: Strong: Explains that await suspends only on a not-yet-done future, predicts the first log order, lists other non-yielding awaits, and fixes it with periodic sleep(0), batching and separating liveness, plus loop-lag monitoring. Acceptable: Fixes it with await asyncio.sleep(0) but believes the queue is somehow "blocking", without the suspension model. Red flag: Insists every await yields and blames the broker, or moves the drain into to_thread with the same asyncio.Queue.

SR-TH-BE-01 · Rate limiting a public API across many instances

Easy · Theory · Senior (5–8 years) · rate limiting, token bucket, Redis, API design, 429, abuse protection · 6 min

Question: Your public order-tracking API runs on 12 instances behind a load balancer, and one partner's buggy integration is hammering it. You're asked to add rate limiting. Which algorithm, where does the state live, and what does a client see?

Answer:

Decide what you're protecting and per what key first: per API key or tenant (fairness between partners), per user or IP for unauthenticated endpoints (abuse), and sometimes per expensive endpoint (a search costs more than a status lookup). Limits live in configuration per plan, not in code.

Algorithms:

Where state lives: with 12 instances, per-process counters give each client 12× the limit and reset on deploys, so use a shared store, usually Redis. The check-and-decrement must be atomic: a Lua script (or a single INCR + EXPIRE for fixed windows) so two instances can't both take the last token. Keys carry a TTL so idle clients cost nothing. Decide the failure mode: if Redis is down, fail open (allow, with an alert) for a business API, fail closed for something like an OTP endpoint, where abuse is costly.

Often the right layering is: coarse per-IP limits at the edge (API gateway, CDN, ingress) to absorb floods cheaply, and precise per-tenant limits in the application where the identity is known.

What the client sees: 429 Too Many Requests with Retry-After, and ideally rate-limit headers (limit, remaining, reset) on every response, so well-behaved clients can pace themselves. Document the limits, and log and meter rejections per tenant so you can tell a buggy partner from an attack, and talk to them.

Follow-ups:

1. "Why is rate limiting not enough to protect the service from overload?" Limits are per client; many legitimate clients together can still exceed capacity, and a few expensive requests can saturate a dependency. You also need load shedding based on the server's own state (queue depth, concurrency limits, latency), timeouts, and bulkheads, so the service degrades instead of collapsing.
2. "A Redis round trip on every request adds latency. Can you avoid it?" Keep a small local token allowance per instance that's refilled from Redis in batches (accepting slight over-admission), or do coarse limiting at the gateway and exact limiting only on expensive endpoints. Measure first: a pipelined Redis call in the same zone is usually well under a millisecond.

Scoring: Strong: Chooses keys by purpose; compares algorithms and picks token bucket with reasons; puts state in Redis with atomic operations and TTLs; chooses fail-open vs fail-closed deliberately; layers edge and app limits; returns 429 with Retry-After and headers. Acceptable: Describes a fixed or sliding window in Redis returning 429, without atomicity, failure mode or layering. Red flag: Keeps counters in a per-process dict on 12 instances, or blocks the client's IP permanently as the fix.

SR-TH-BE-02 · "We set a 30-second timeout" and the request still hangs for minutes

Easy & Tricky · Theory · Senior (5–8 years) · timeouts, httpx, retries, deadlines, retry storms, resilience · 6 min

Question: A checkout service calls a KYC provider with httpx.Client(timeout=30). During a provider incident, checkout requests hung for several minutes, every worker was busy, and the whole site went down. The author says "the timeout is 30 seconds, so a call can't take longer than that". What's wrong with that belief, and how should the service be configured?

Answer:

Trap: "timeout=30 caps the call at 30 seconds."

In httpx, timeout=30 sets four separate limits to 30 s each: connect, read, write and pool (waiting for a free connection from the pool). None of them is a total deadline. The read timeout is the maximum gap between received chunks, so a provider that trickles a byte every 20 seconds keeps the call alive indefinitely. A call can spend up to 30 s waiting for a pooled connection, 30 s connecting, then a long time reading.

Then retries multiply it. If the browser SDK, the API gateway and the service's HTTP client each make 3 attempts, a single user action can become 27 attempts against a provider that's already failing, each allowed 30 s per phase. That's both the multi-minute hang and a retry storm that keeps the provider down.

What I'd change:

• Per-phase timeouts from real latency data: connect around 1 s (a TCP connect within a region takes milliseconds), read just above the provider's p99.9, and pool timeout short, so you fail fast when saturated.
• A total deadline around the whole operation, including retries: asyncio.timeout() (or anyio.fail_after) in async code, or a deadline that's checked before each attempt and passed down as the remaining budget (e.g. in a header), so downstream services don't keep working on a request the user already abandoned.
• Retry at one layer only, only for idempotent operations or with an idempotency key, with exponential backoff and jitter, and a retry budget (e.g. retries at most 10% of traffic).
• Circuit breaker around the provider so that after repeated failures calls fail immediately, plus a fallback (queue the KYC check, let the user continue to a "pending verification" state).
• Bulkhead: a separate, bounded connection pool and concurrency limit for this dependency, so one slow provider can't consume every worker.

import httpx

t = httpx.Timeout(30.0)
print(t.connect, t.read, t.write, t.pool)

# What a "30 s timeout" can mean end to end with retries at three layers:
browser_sdk, api_gateway, service_client = 3, 3, 3   # attempts per layer
attempts = browser_sdk * api_gateway * service_client
print(f"worst case: {attempts} attempts, each allowed 30 s per read")

# A deadline-style budget instead: one connect limit, a shorter read limit.
budget = httpx.Timeout(5.0, connect=1.0)
print(budget.connect, budget.read, budget.write, budget.pool)

Output:

30.0 30.0 30.0 30.0
worst case: 27 attempts, each allowed 30 s per read
1.0 5.0 5.0 5.0

Follow-ups:

1. "What's the default timeout in requests and in httpx if you don't pass one?" requests has no default timeout at all: a call can wait forever, which is why linters flag missing timeout= arguments. httpx defaults to 5 seconds for each phase. Either way, set timeouts explicitly per dependency, based on its measured latency.
2. "Which layer should own retries?" Usually the one closest to the failing dependency that knows whether the operation is safe to repeat, typically the service's client for that dependency, with outer layers not retrying (or retrying only on explicit signals like 503 with Retry-After). The important rule is that layers don't each retry independently.

Scoring: Strong: Knows the timeout is per phase and read is per chunk, with no total deadline; quantifies retry multiplication; prescribes measured per-phase timeouts, an overall deadline propagated downstream, single-layer retries with backoff and jitter, circuit breaker and bulkhead. Acceptable: Suspects retries made it worse and suggests smaller timeouts and a circuit breaker, without understanding httpx's timeout semantics. Red flag: Raises the timeout or the number of workers, or adds another retry layer.

SR-TH-BE-03 · Sizing database connection pools across workers and pods

Medium · Theory · Senior (5–8 years) · connection pooling, SQLAlchemy, PostgreSQL, PgBouncer, autoscaling, capacity planning · 7 min

Question: After autoscaling kicked in during a sale, the API started failing with FATAL: sorry, too many clients already from Postgres, even though database CPU was at 40%. Each pod runs 4 Gunicorn workers with SQLAlchemy's default pool settings. Explain what happened and how you'd size pools.

Answer:

Connection pools are per process. SQLAlchemy's default QueuePool allows pool_size=5 plus max_overflow=10: up to 15 connections per worker process. So the ceiling is

pods × workers per pod × 15, e.g. 20 pods × 4 × 15 = 1,200 connections,

against a Postgres max_connections that is often 100–500. Autoscaling multiplies the pods, and so the connections, exactly when load is highest. CPU being at 40% is irrelevant; the connection limit is a separate hard limit, and each Postgres connection is a backend process with its own memory.

How I'd size it:

1. Start from the database's budget: max_connections minus headroom for admin, migrations, replication, and other services. Say 300 usable.
2. Divide by the maximum number of processes at the autoscaler's upper bound: 300 / (30 pods × 4 workers) ≈ 2–3 connections per process. That forces a design question rather than a config tweak.
3. Small pools are usually enough, and faster: Postgres throughput peaks at a modest number of active connections (on the order of a few per CPU core); beyond that, connections queue on locks and CPU. A request should hold a connection for milliseconds, so a pool of 2–5 per process serves a lot of traffic, as long as code doesn't hold a connection during external calls (an HTTP call to a payments provider inside an open transaction is a classic pool-exhauster).
4. Put a pooler in front when the process count is large: PgBouncer in transaction pooling mode lets thousands of client connections share a few dozen server connections. It breaks session state: session-level SET, advisory locks, LISTEN/NOTIFY, temporary tables, and server-side prepared statements (asyncpg's statement cache must be disabled or PgBouncer configured for prepared statements).
5. Fail fast and observe: a short pool_timeout so requests fail quickly instead of piling up, pool_pre_ping (or pool_recycle) for connections killed by idle timeouts, and metrics on checked-out connections and wait time per pool. Cap autoscaling where the database, not the app, is the bottleneck.

Follow-ups:

1. "With an async SQLAlchemy engine in each Uvicorn worker, does the per-process math change?" No: there's still one pool per process (per event loop). Async lets one process run many concurrent requests, which makes it easier to exhaust a small pool, so the pool timeout and per-request connection hold time matter even more. Never share an engine across event loops or forked processes; create it after fork, in the lifespan handler.
2. "Why is pool_size larger than needed a problem if connections sit idle?" Idle connections still use memory on the database server and count against max_connections; under a traffic spike every process opens its maximum at once, causing a connection storm exactly when the database is busiest. Large pools also hide slow queries and connection leaks until they cause an outage.

Scoring: Strong: Computes the per-process multiplication with autoscaling; sizes from the database budget downward; knows small pools are often faster and that holding connections during external I/O exhausts them; uses PgBouncer and knows what transaction pooling breaks; fails fast with metrics. Acceptable: Recognises too many connections from many processes and suggests lowering pool size or adding PgBouncer, without the budget calculation or its caveats. Red flag: Raises max_connections to several thousand, or increases pool sizes because requests are waiting.

SR-TH-BE-04 · Deploys drop requests even though Uvicorn shuts down gracefully

Medium & Tricky · Theory · Senior (5–8 years) · graceful shutdown, Kubernetes, SIGTERM, Uvicorn, readiness, background tasks · 7 min

Question: Every deploy of a FastAPI service on Kubernetes causes a burst of 502s and a few lost "send receipt email" background tasks. The team says "Uvicorn handles SIGTERM gracefully, so it can't be the shutdown". Where do the errors come from, and what do you change?

Answer:

Trap: "The app server drains in-flight requests on SIGTERM, so shutdown is safe."

Uvicorn does stop accepting new connections and waits for in-flight requests when it gets SIGTERM. The failures come from what happens around that:

1. Endpoint removal races the signal. When a pod is terminated, Kubernetes sends SIGTERM (after any preStop hook) and, in parallel, removes the pod from the Service endpoints. It takes seconds for kube-proxy, the ingress and any cloud load balancer to stop sending traffic. Meanwhile the app has already closed its listening socket, so new requests get connection refused, which the proxy reports as 502.
2. Keep-alive connections: clients and proxies reuse idle connections that the server is now closing, and a request sent on one just as it closes fails.
3. Grace period: after terminationGracePeriodSeconds (30 s by default) the container gets SIGKILL. Long requests, streaming responses and WebSockets are cut.
4. BackgroundTasks are in-process: they run after the response is sent, in the same process. Anything still running or not yet started at SIGKILL is simply lost; nothing records that the receipt was never sent.
5. Signal delivery: if the container starts with a shell script (sh -c "uvicorn ...") and the shell is PID 1, SIGTERM may never reach Uvicorn, so it never drains and is killed after 30 s.

Changes:

• A preStop hook that sleeps (e.g. 10–15 s) so the pod keeps serving while it's removed from load balancers, then receives SIGTERM. Optionally fail the readiness probe as soon as shutdown starts.
• Budget the grace period: preStop sleep + longest expected request + lifespan shutdown work (closing pools, flushing telemetry) < terminationGracePeriodSeconds; set Uvicorn's --timeout-graceful-shutdown within it.
• Move work that must happen to a durable queue (Celery, RQ, a jobs table, or an outbox) instead of BackgroundTasks; keep BackgroundTasks for best-effort work.
• Exec form in the Dockerfile (CMD ["uvicorn", ...]) or a small init like tini, so signals reach the server.
• Rolling update settings (maxUnavailable: 0, a sensible maxSurge) and a PodDisruptionBudget, so capacity doesn't dip during the deploy.

Follow-ups:

1. "How do you verify the fix rather than waiting for the next deploy?" Run a steady load test (e.g. a few hundred requests per second with keep-alive) against a staging deployment and trigger a rolling restart; count non-2xx responses and dropped jobs. Make it a release check, because regressions here are invisible until a busy-hour deploy.
2. "What should a queue consumer do on SIGTERM?" Stop fetching new messages, finish (or abandon cleanly) the current one, acknowledge only after the work's side effects are committed, and exit within the grace period. Unacknowledged messages are redelivered to another worker, which is why handlers must be idempotent.

Scoring: Strong: Explains the endpoint-propagation race, keep-alive, the grace-period SIGKILL, in-process background tasks and PID 1 signal handling; fixes with preStop delay, a grace budget, a durable queue and exec-form startup. Acceptable: Identifies the load balancer still sending traffic after SIGTERM and adds a preStop sleep, but misses lost background work or signal delivery. Red flag: Blames Uvicorn and proposes retrying 502s at the client, or increases the grace period to several minutes without understanding the race.

SR-TH-BE-05 · Saving an order and publishing its event without losing either

Hard · Theory · Senior (5–8 years) · transactional outbox, dual writes, Kafka, event-driven architecture, idempotent consumers, CDC · 8 min

Question: The order service does session.commit() and then producer.send("order-placed", event). Occasionally the warehouse never hears about an order; once, it shipped an order that had been rolled back. Explain the failure and design a fix.

Answer:

This is the dual-write problem: two systems (Postgres and Kafka) with no shared transaction. Whatever order you write them in, a crash or error between the two leaves them inconsistent:

• Commit, then publish: the process dies, the broker is unreachable, or the send fails after commit, so the order exists but no event is ever sent. Fire-and-forget producers (an un-awaited future) hide this completely.
• Publish, then commit: the commit fails (constraint violation, deadlock, timeout), so the event describes an order that doesn't exist. That's how a rolled-back order got shipped.

Retrying doesn't solve it, and Kafka transactions don't help because they don't include the database.

Transactional outbox:

1. In the same database transaction as the order, insert the event into an outbox table (event ID, aggregate ID, type, payload, created time). Either both commit or neither does.
2. A relay publishes outbox rows to Kafka and marks them sent. Either a poller (SELECT ... ORDER BY id LIMIT 500 FOR UPDATE SKIP LOCKED, so several relays can run safely), or log-based CDC (Debezium reading the WAL), which avoids polling and preserves commit order.
3. The relay can publish and crash before marking the row sent, so delivery is at least once: every event carries a unique ID, and consumers are idempotent (an inbox/processed-events table keyed by event ID, or naturally idempotent upserts).
4. Ordering: use the order ID as the Kafka key so events for one order land in one partition in order, and have the relay publish per aggregate in outbox order.
5. Operate it: alert on outbox lag (oldest unsent row age), purge or archive sent rows, and version the event schema as a data contract.

Alternatives: CDC directly on the business tables (no outbox, but consumers become coupled to your table schema), or "listen to yourself" (publish only, and have the service consume its own event to write the DB), which makes the write path asynchronous for the user. Distributed two-phase commit is rarely available or worth it.

Follow-ups:

1. "Why FOR UPDATE SKIP LOCKED in the poller?" It lets several relay instances poll concurrently without blocking on, or double-processing, each other's rows: each picks up rows that aren't currently locked. Without it you either run a single relay (a single point of failure) or get duplicate publishes from concurrent pollers.
2. "The consumer is idempotent by event ID. Is that enough if two events for the same order arrive out of order?" No. Idempotency handles duplicates, not ordering. Keep per-key ordering (same partition key and a relay that preserves order), and make consumers order-aware where it matters: store a per-order version or sequence number and ignore events older than what's already applied.

Scoring: Strong: Explains both dual-write orders and why retries or Kafka transactions don't fix it; designs an outbox in the same transaction with a relay (SKIP LOCKED poller or CDC), at-least-once delivery with idempotent consumers, per-key ordering and lag monitoring; knows the alternatives. Acceptable: Proposes an outbox table and a background publisher, but misses duplicates, ordering or operations. Red flag: Wraps both calls in a try/except with a retry, or publishes before committing "so the event is never lost".

SR-TH-BE-06 · An "additive" API change that breaks clients

Hard & Tricky · Theory · Senior (5–8 years) · API design, backward compatibility, versioning, pydantic, enums, tolerant reader · 7 min

Question: Your payments API adds a new status value, partially_refunded, and a new optional response field, upi_ref. The PR description says "additive change, non-breaking, no version bump needed". Within an hour two partner integrations are failing. How, and what policy would you set?

Answer:

Trap: "Adding fields and enum values is always backward compatible."

Compatibility is defined by what clients do with the response, not by the schema diff. Two very common client behaviours break here:

• Closed enums: a client SDK generated from last quarter's spec models status as a fixed enum, so the new value fails validation (or falls through an exhaustive match/switch into an error path, or is silently mapped to the wrong bucket).
• Strict objects: clients that validate responses with "no unknown fields" (Pydantic extra="forbid", JSON Schema additionalProperties: false, some strict deserialisers in typed languages) reject the extra key.

The snippet reproduces both with a Pydantic client model: the old payload validates, the new enum value and the new field each fail. (Pydantic's default is extra="ignore"; the client opted into strictness.)

More generally, by Hyrum's law every observable behaviour gets depended on: default sort order, page size, a field that was never null, error message text, timing.

Policy I'd set for the API:

1. Publish a compatibility contract for clients: ignore unknown fields; treat enums as open and handle unknown values with a fallback. Make our own SDKs do this (enum fields typed as str with known constants, or an UNKNOWN member via a validator), so the official clients can't be the ones that break.
2. Classify changes: new optional fields are compatible under that contract; new enum values are announced ahead with a date (or gated behind an opt-in/version) because they change client logic; removing/renaming fields, changing types or meaning, and making fields required are breaking.
3. Breaking changes get a new version (path /v2 or a dated version header, Stripe-style), with old versions supported for a published period, Deprecation/Sunset headers, and per-client usage telemetry to know who still calls what.
4. Automated checks in CI: diff the OpenAPI schema against the released one and flag breaking changes; contract tests with the largest partners' expectations.

from enum import Enum
from pydantic import BaseModel, ConfigDict, ValidationError

class PaymentStatus(str, Enum):        # the client SDK's copy, generated last quarter
    PENDING = "pending"
    SUCCEEDED = "succeeded"
    FAILED = "failed"

class Payment(BaseModel):
    model_config = ConfigDict(extra="forbid")
    id: str
    status: PaymentStatus

old = {"id": "pay_1", "status": "succeeded"}
new = {"id": "pay_2", "status": "partially_refunded"}                  # new enum value
added = {"id": "pay_3", "status": "pending", "upi_ref": "4321"}        # new optional field

for payload in (old, new, added):
    try:
        print("ok  ", Payment.model_validate(payload).status.value)
    except ValidationError as e:
        err = e.errors()[0]
        print("FAIL", err["type"], err["loc"])

Output:

ok   succeeded
FAIL enum ('status',)
FAIL extra_forbidden ('upi_ref',)

Follow-ups:

1. "A partner refuses to change their strict client. What can you do without blocking your roadmap?" Pin them to a version that never shows the new value and field (per-client API version, as Stripe does with dated versions), and translate at the edge: map partially_refunded to the nearest old value for that version, and strip new fields. That's a response-transformation layer per version, which is why versioning needs infrastructure, not just a URL prefix.
2. "Should your own server models use extra="forbid" for request bodies?" For requests, often yes: rejecting unknown fields catches client typos (ammount) that would otherwise be silently ignored. The asymmetry is deliberate: be strict about what you accept from clients you can reject clearly, and tolerant in what clients should accept from you.

Scoring: Strong: Explains closed enums and strict deserialisers as the break; cites Hyrum's law; sets a client contract (tolerant reader), makes own SDKs tolerant, classifies enum additions as needing notice, versions breaking changes with deprecation and telemetry, and adds schema-diff checks in CI. Acceptable: Explains why the strict client broke and suggests telling partners to ignore unknown fields, but has no versioning or change-classification policy. Red flag: Insists additive changes can't break anyone and blames the partners, or bumps the major version for every change.

SR-TH-CLO-01 · What a closure holds, and how to inspect one during an incident

Easy · Theory · Senior (5–8 years) · closures, cells, __closure__, inspect, debugging · 4 min

Question: A callback built by a factory function is producing wrong invoices in production. You have a debugger or a REPL attached to the process. What exactly does a closure hold, and how do you inspect what this callback captured?

Answer:

A closure is a function object plus references to cells for the enclosing-scope variables it uses (its free variables). The compiler decides which names are free. At run time:

• fn.__code__.co_freevars lists the free variable names.
• fn.__closure__ is a tuple of cell objects in the same order, and each cell.cell_contents is the current value of that variable.
• inspect.getclosurevars(fn) gives a readable summary: nonlocals (closure values), globals and builtins the code refers to, and unbound names.

Globals such as GST_RATE are not captured. They're looked up in the module at call time, so a module-level change affects every closure immediately. Only enclosing-function variables live in cells, and a cell is shared by every closure created in the same call of the outer function.

In an incident, this lets you confirm which configuration a long-lived callback actually holds (currency, decimals, tenant, a stale client) without guessing from the code. A common finding: the factory ran once at startup with the wrong config, and the callback has carried it ever since.

import inspect

GST_RATE = 18


def make_invoice_formatter(currency, decimals):
    template = "{cur} {amt:,.{d}f} (GST {gst}%)"

    def fmt(amount):
        return template.format(cur=currency, amt=amount, d=decimals, gst=GST_RATE)

    return fmt


fmt = make_invoice_formatter("₹", 2)
print(fmt(4500))
print(fmt.__code__.co_freevars)
print([c.cell_contents for c in fmt.__closure__])
cv = inspect.getclosurevars(fmt)
print(cv.nonlocals["currency"], cv.globals)

Output:

₹ 4,500.00 (GST 18%)
('currency', 'decimals', 'template')
['₹', 2, '{cur} {amt:,.{d}f} (GST {gst}%)']
₹ {'GST_RATE': 18}

Follow-ups:

1. "When would you replace this closure with a small class or a frozen dataclass with __call__?" When the captured state needs to be visible: a useful repr in logs and error reports, equality in tests, pickling for multiprocessing, or more than one operation on the same state. A closure hides its state behind __closure__, which is fine for small private helpers but makes incidents harder to debug.
2. "If the code changes GST_RATE at run time, does an existing fmt see the new rate?" Yes. GST_RATE is a global, resolved at each call, not captured. That's the opposite of what some people expect, and it's another reason to pass such values in explicitly.

Scoring: Strong: Explains cells and free variables, uses __closure__, co_freevars and inspect.getclosurevars, distinguishes captured enclosing variables from globals read at call time, and connects it to incident debugging. Acceptable: Defines a closure correctly and knows __closure__ exists. Red flag: Thinks closures copy globals too, or has no way to inspect captured state other than adding print statements and redeploying.

SR-TH-CLO-02 · The default-argument fix binds the object, not a snapshot

Easy & Tricky · Theory · Senior (5–8 years) · closures, late binding, default arguments, lambdas, aliasing · 4 min

Question: To avoid late binding, a teammate registers handlers in a loop with lambda region=region, cfg=cfg: ... and says "default arguments freeze the values at definition time". Later the shared cfg dict is updated. What do the handlers see, and is there any other side effect of this fix?

Answer:

Trap: "Default arguments freeze the values, so each handler keeps limit = 100."

A default argument is evaluated once, at definition time, and stores a reference to the resulting object. For region (a string, immutable) that's effectively a snapshot. For cfg it's a reference to the same dict, and the later cfg["limit"] = 5 mutates that dict, so every handler reports 5. The trick protects against the name being rebound, not against the object being mutated.

The second side effect: defaults become part of the public signature. Anyone can call handler(cfg={...}) and override the "captured" value by accident, for example a dispatcher that passes keyword arguments it thinks the handler wants. Signature-introspecting frameworks (FastAPI, pytest, click) will also see region and cfg as parameters.

Better options:

• functools.partial(handle, region, limit=cfg["limit"]): binds actual values, keeps them visible in .args/.keywords, and doesn't add fake parameters.
• A factory function make_handler(region, limit) that captures only the immutable values it needs.
• Make the config immutable (a frozen dataclass), so sharing it is harmless. Note that MappingProxyType(cfg) doesn't help: it's a read-only view that still reflects changes to cfg.

handlers = []
cfg = {"limit": 100}
for region in ["north", "south"]:
    handlers.append(lambda region=region, cfg=cfg: f"{region}:{cfg['limit']}")

cfg["limit"] = 5
print([h() for h in handlers])
print(handlers[0](cfg={"limit": 1}))

Output:

['north:5', 'south:5']
north:1

Follow-ups:

1. "Which approach would you prefer if cfg is reloaded by rebinding (cfg = load()) rather than mutated?" Then the default-argument version keeps the old dict, which may be what you want (a stable snapshot) or a staleness bug. Decide explicitly: capture values (snapshot) or read through an accessor at call time (live config). Don't let the mechanism decide by accident.
2. "Is late binding ever what you want?" Yes. A callback that should always use the current value of a module-level setting or a mutable registry benefits from reading at call time. The bug is relying on it unknowingly, not the mechanism itself.

Scoring: Strong: Predicts 5 for both, explains reference vs snapshot, raises the signature leak, and proposes partial or a factory capturing immutable values; knows MappingProxyType is a live view. Acceptable: Predicts the output once prompted to think about mutation, and suggests copying cfg per handler. Red flag: Insists default arguments copy values, or proposes copy.deepcopy(cfg) in every lambda default without considering what behaviour is wanted.

SR-TH-CLO-03 · Per-decoration vs per-call state in a parametrised decorator

Medium · Theory · Senior (5–8 years) · closures, decorators, nonlocal, retries, shared state, thread safety · 6 min

Question: A @retry(times=3) decorator keeps attempts_left in the factory's scope and decrements it with nonlocal in the wrapper. The first payment push retries correctly; later pushes give up after one failure. Explain the bug in terms of closure lifetimes and state where each kind of state belongs.

Answer:

A parametrised decorator has three nested scopes with three different lifetimes:

attempts_left lives in retry's scope, so one cell is shared by every call of the decorated function. T1's two failures use up the budget, and T2 starts with 1 attempt left and gives up on its first failure. Because the cell is shared, concurrent calls from several threads also race on attempts_left -= 1. The bug depends on traffic history, so it's invisible in a unit test that calls the function once.

Rule: configuration (times, backoff policy) belongs in the outer scopes. Per-call state (attempt counters, start time, the last exception) is local to wrapper, for example for attempt in range(1, times + 1):. State deliberately shared across calls (a circuit breaker's failure count, metrics) can live in an outer scope, but then it's shared mutable state and needs a lock, and it's often clearer as an explicit object, such as a CircuitBreaker instance passed to the decorator.

For the real retry: retry only on errors that are safe to retry, on idempotent operations or with an idempotency key; add backoff with jitter; and preserve the original exception.

import functools


def retry(times):
    attempts_left = times                     # per decoration, not per call!

    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            nonlocal attempts_left
            while True:
                try:
                    return fn(*args, **kwargs)
                except ConnectionError:
                    attempts_left -= 1
                    if attempts_left <= 0:
                        raise
        return wrapper
    return decorator


failures = {}


@retry(times=3)
def push(txn):
    failures[txn] = failures.get(txn, 0) + 1
    if failures[txn] <= 2:                    # every txn flakes twice
        raise ConnectionError(f"{txn}: gateway reset")
    return f"{txn} ok"


for txn in ["T1", "T2"]:
    try:
        print(push(txn))
    except ConnectionError as e:
        print("gave up:", e)

Output:

T1 ok
gave up: T2: gateway reset

Follow-ups:

1. "Write the corrected wrapper loop." for attempt in range(1, times + 1): try: return fn(*args, **kwargs) except ConnectionError: if attempt == times: raise; time.sleep(backoff(attempt)). Here times is read-only configuration from the closure, and attempt is local to each call.
2. "How would you test a decorator so this bug is caught?" Call the decorated function several times in one test, each call with its own failure pattern, and assert each call gets the full retry budget. Also test concurrent calls with a fake that records attempts per call.

Scoring: Strong: Maps the three scopes to their lifetimes, explains the shared cell (plus the thread race), states where config, per-call and shared state belong, and mentions idempotency for retries. Acceptable: Finds that the counter must be reset per call and moves it into the wrapper. Red flag: Fixes it by resetting attempts_left at the end of a successful call (still shared and racy), or can't say why only later calls fail.

SR-TH-CLO-04 · A generator expression binds its first iterable early and everything else late

Medium & Tricky · Theory · Senior (5–8 years) · closures, late binding, generator expressions, laziness · 5 min

Question: A report builder creates big = (r for r in rows if r > threshold), then, before consuming it, the same function reassigns both threshold and rows for the next section. Which rows and which threshold does list(big) use?

Answer:

Trap: either "both values at creation time" or "both values at consumption time".

It's split. A generator expression is compiled as a nested function, a closure. Only the outermost iterable (rows) is evaluated immediately, in the enclosing scope, and passed into the generator as an argument. Everything else (the if condition, the element expression, any inner for iterables) is evaluated lazily when items are pulled, and reads enclosing variables through closure cells with their current values. So list(big) iterates the original list [500, 1500, 2500] with the new threshold = 2000, giving [2500].

The design is deliberate: evaluating the source early means errors in obtaining it (a bad file path, a failed query) surface where the expression is written, while the per-item work stays lazy.

Where it bites: building several lazy filters or pipelines in a loop (each closes over the loop's variables and sees only their final values), or creating a generator and handing it to a consumer that runs later, after the parameters changed.

Fixes: consume immediately when the parameters are about to change, use a list comprehension when the data is small, or bind the parameters explicitly in a generator function: def above(rows, threshold): for r in rows: if r > threshold: yield r. Here the arguments are evaluated at call time and can't change underneath it.

threshold = 1000
rows = [500, 1500, 2500]
big = (r for r in rows if r > threshold)

threshold = 2000
rows = [5000]
print(list(big))

Output:

[2500]

Follow-ups:

1. "Does the same split apply to a list comprehension in Python 3.12, given PEP 709 inlines comprehensions?" For a list comprehension it doesn't matter, because it runs to completion immediately and nothing changes in between. PEP 709 inlines list, dict and set comprehensions but not generator expressions, which still have their own frame because they must suspend.
2. "What happens if the outermost iterable expression itself raises?" The exception is raised at the line that creates the generator expression, not at consumption, because that part is evaluated eagerly. An error in the condition or an inner iterable is raised later, during iteration, often far from where the expression was written.

Scoring: Strong: Correct output, explains the eager outermost iterable vs late-bound closure reads, why it's designed that way, and offers a generator-function fix. Acceptable: Gets the output after reasoning, knows generators are lazy, but can't explain why rows didn't change. Red flag: Answers [5000] or [1500, 2500] and can't revise with the closure model.

SR-TH-CLO-05 · A cached closure that leaks one user's data into another's request

Hard · Theory · Senior (5–8 years) · closures, caching, request-scoped state, contextvars, security, data leaks · 7 min

Question: To avoid rebuilding renderers, a statements service caches a closure per locale. The factory reads the current user from request context while building the closure. After deploy, a support ticket says customer Ravi saw Anita's user ID in his statement header. Explain the bug, why it's a security incident and not just a correctness bug, and the design rules that prevent it.

Answer:

The renderer closure captured request-scoped data (user_id) when it was built, and was then stored in a cache that outlives the request. The cache key is locale only, so every later request with the same locale gets Anita's closure, with her user ID baked into its cell. That's cross-user data exposure: a reportable privacy incident, not just a wrong value. It's also hard to reproduce, because it depends on which user warmed the cache in each worker process.

The same pattern appears with:

• a DB session or transaction captured in a callback that runs after the request closed (use-after-close, or writes in the wrong transaction);
• an auth token captured in a cached HTTP client;
• a tenant ID captured in a background task or an event-bus subscription.

Rules to set as a reviewer:

1. A cached value may depend only on its cache key. If the closure reads anything else (user, tenant, locale-plus-plan), either add it to the key or stop capturing it.
2. Long-lived closures capture only request-independent, immutable configuration. Per-request data is passed as call arguments, e.g. render(user_id, amount).
3. If code deep in a call stack needs request context, read it at call time from a contextvars.ContextVar (asyncio-safe) instead of capturing it when an object is built.
4. Background tasks receive IDs, not sessions or user objects, and open their own resources.

To detect it: inspect.getclosurevars on cached callables in a test, and integration tests that serve two different users in sequence from one process and assert no data crosses over.

current_user = {"id": None}        # stands in for request-scoped context
_renderers = {}


def statement_renderer(locale):
    if locale not in _renderers:
        user_id = current_user["id"]               # captured at build time
        def render(amount):
            return f"[{locale}] statement for {user_id}: ₹{amount}"
        _renderers[locale] = render
    return _renderers[locale]


def handle_request(user_id, locale, amount):
    current_user["id"] = user_id
    return statement_renderer(locale)(amount)


print(handle_request("U-ANITA", "en-IN", 1200))
print(handle_request("U-RAVI", "en-IN", 560))

Output:

[en-IN] statement for U-ANITA: ₹1200
[en-IN] statement for U-ANITA: ₹560

Follow-ups:

1. "Why does contextvars fix the asyncio case where a plain global or thread-local would not?" Each asyncio task runs in a copy of the context taken when it was created, so a ContextVar.set in one request's task isn't visible in another's, even on the same thread. A thread-local is shared by every coroutine on that thread, and a global by everything in the process.
2. "How would you respond operationally once this is found?" Roll back or hotfix (drop the cache), estimate the exposure window and affected users from logs, follow the incident and privacy notification process (in India, DPDP Act obligations), and add the regression test. Then review other caches for the same pattern.

Scoring: Strong: Identifies request data captured in a closure that outlives the request, the cache key not covering dependencies, the security impact, and gives clear rules (key completeness, pass per-request data as arguments, contextvars, IDs to background tasks). Acceptable: Finds the captured user_id and moves it to a call argument, without generalising the rule. Red flag: Fixes it by clearing the cache after each request, or treats it as a minor display bug.

SR-TH-CLO-06 · Closures are not private, and their cells can be rewritten

Hard & Tricky · Theory · Senior (5–8 years) · closures, cells, security, secrets, error reporting, encapsulation · 5 min

Question: A payments SDK hides its signing secret inside a closure (sign = make_signer(secret)) and the author says the key is "private: nothing outside the factory can read or change it". Is that true? What are the practical consequences for handling secrets?

Answer:

Trap: "Captured variables are like private fields: only the inner function can reach them."

Neither half is true. Anyone with a reference to sign can read the secret via sign.__closure__[0].cell_contents or inspect.getclosurevars(sign). Since Python 3.7, cell_contents is also writable, so code can swap the key for every holder of that closure. Python has no in-process privacy boundary: closures, _private names and name-mangled __attrs are conventions against accidents, not protection against code running in the same interpreter.

The practical risk is less about an attacker poking at cells and more about accidental disclosure:

• Error trackers and rich tracebacks (Sentry, rich, better_exceptions) capture frame locals. A frame's locals include its free variables, so an exception inside sign ships the secret to a third-party service.
• Debug logging of objects with default reprs, pickled callbacks in task queues, and heap dumps all expose captured values.

Practices to standardise:

• Keep secrets in a dedicated type that redacts its repr and str (pydantic's SecretStr, or a small class), and unwrap at the last moment.
• Configure error reporting to scrub locals or variables matching secret patterns, or disable local-variable capture in payment paths.
• Load secrets from a secret manager with short lifetimes, and rotate keys. Assume anything in process memory can leak.

import hashlib
import hmac


def make_signer(secret: bytes):
    def sign(payload: bytes) -> str:
        return hmac.new(secret, payload, hashlib.sha256).hexdigest()[:12]
    return sign


sign = make_signer(b"live-key-123")
print(sign.__closure__[0].cell_contents)
before = sign(b"order=A-101")
sign.__closure__[0].cell_contents = b"attacker-key"
print(before == sign(b"order=A-101"))

Output:

b'live-key-123'
False

Follow-ups:

1. "Is there any legitimate use for writing cell_contents?" Rarely: debugging tools, hot-patching in a REPL during an incident, or some testing utilities. In application code it's a red flag. If state needs to change, make it an explicit attribute of an object.
2. "Would a class with __slots__ and a name-mangled __secret be better protected?" Not against in-process code: obj._Signer__secret still reads it, and the slot descriptor is on the class. The real gains come from a redacting repr, not storing the raw secret longer than needed, and scrubbing error reports.

Scoring: Strong: Shows closures are readable and (since 3.7) writable, explains that there's no in-process security boundary, and focuses on realistic leaks (frame locals in error reporters, logs) with concrete mitigations. Acceptable: Knows __closure__ exposes captured values and suggests a secret type or vault. Red flag: Defends closures or name mangling as a security mechanism.

SR-TH-CMP-01 · When you reject a comprehension in code review

Easy · Theory · Senior (5–8 years) · comprehensions, readability, code review, side effects, team standards · 4 min

Question: You're writing the team's Python style guide. When is a comprehension the right tool, and which comprehensions would you ask authors to rewrite in review?

Answer:

A comprehension is right when it builds a collection from one transform and at most one filter, and reads as a single sentence: [o.id for o in orders if o.is_paid], {u.id: u for u in users}.

Ask for a rewrite (an explicit loop or a named helper) when:

• It's used for side effects: [db.save(r) for r in rows] builds a throwaway list of Nones and hides that this is a write loop. Use a for loop.
• It has more than two for/if clauses, or nested comprehensions inside comprehensions. The left-to-right nesting order stops being obvious.
• The condition or expression needs a comment, or repeats an expensive call ([f(x) for x in xs if f(x)]). Extract a named function, or use := sparingly when it's still readable.
• It needs error handling per item. try/except can't go inside a comprehension, and one bad row aborts the whole thing. A loop or generator function can log or dead-letter the bad item and continue.
• The result is huge and consumed once. Use a generator expression, or stream it, instead of materialising.

The reasoning to share with the team: comprehensions are a readability feature first. Their speed advantage over an equivalent append loop is real but modest, and it rarely matters next to I/O. When the comprehension is harder to read than the loop, the loop wins.

Follow-ups:

1. "Would you enforce any of this with tooling?" Partly. Ruff and flake8-comprehensions flag unnecessary comprehensions and side-effect patterns, and complexity limits catch some nesting. Readability judgements stay with review, so put examples in the style guide.
2. "[x for x in rows if x.amount] vs filter(None, ...) vs a loop: which do you prefer for a filter-only case?" The comprehension, because the condition is explicit. filter with a lambda is less readable, and filter(None, ...) relies on truthiness, which may silently drop legitimate zero amounts.

Scoring: Strong: Gives a clear positive rule plus concrete rejection criteria (side effects, nesting, repeated calls, per-item error handling, size), and keeps performance in perspective. Acceptable: Has sensible readability instincts but few concrete criteria. Red flag: Prefers comprehensions everywhere "because they're faster", including for side effects.

SR-TH-CMP-02 · `any([...])` calls the expensive check on every item

Easy & Tricky · Theory · Senior (5–8 years) · comprehensions, generator expressions, short-circuit, any, performance · 4 min

Question: An onboarding flow checks PANs against a blocklist service with if any([is_blocked(p) for p in pans]):. Each check is a ~200 ms network call. The first PAN in a typical batch of four is the blocked one. How many calls are made, and how does the generator-expression version differ?

Answer:

Trap: "any stops at the first True, so only one call is made either way."

any does short-circuit, but only over values it is given lazily. With square brackets, the list comprehension runs completely before any starts, so all four network calls happen (about 800 ms), and any then stops at the first element of a list that already exists. With a generator expression, any(is_blocked(p) for p in pans), each call happens only when any asks for the next value, so it stops after the first call (about 200 ms). all, next, min/max with early exits in your own loops, and itertools.takewhile behave the same way.

The opposite case matters too: sometimes you want every check to run, for example validating a form and reporting all errors, or checks with side effects such as audit logging. Then build the full list explicitly and name it, so nobody "optimises" it into a generator later.

At senior level, also question the design: four sequential 200 ms calls is the bigger problem. A batch endpoint, concurrent calls, or a cached local copy of the blocklist would change the latency far more than short-circuiting.

calls = []


def is_blocked(pan):
    calls.append(pan)            # imagine a 200 ms call to a KYC service
    return pan.startswith("ZZ")


pans = ["ZZ1234", "AB5678", "CD9012", "EF3456"]

print(any([is_blocked(p) for p in pans]), len(calls))
calls.clear()
print(any(is_blocked(p) for p in pans), len(calls))

Output:

True 4
True 1

Follow-ups:

1. "Linters flag any([...]). Is the rewrite always safe?" Safe for correctness when the per-item calls have no side effects you rely on. If they do (logging, counting, mutating), switching to a generator changes behaviour by skipping calls. Check before applying the auto-fix.
2. "How would you make this check fast if the blocklist has 2 lakh entries?" Load it into a set (or a Bloom filter if memory matters) and refresh it periodically, making each check an O(1) local lookup. Then a short-circuit vs full scan barely matters.

Scoring: Strong: Says 4 vs 1 calls with the correct reason (eager list built before any), names the cases where full evaluation is wanted, and raises the bigger latency design issue. Acceptable: Gets the counts and the reason. Red flag: Thinks both versions make one call, or that list comprehensions are lazy.

SR-TH-CMP-03 · A dict comprehension silently keeps the last duplicate key

Medium · Theory · Senior (5–8 years) · dict comprehensions, duplicates, data contracts, data quality, pandas · 5 min

Question: A credit-limit loader builds by_pan = {r["pan"]: r for r in rows} from a partner's daily file. Downstream, one customer's limit dropped from ₹50,000 to ₹5,000 overnight. The file had a duplicate PAN row. What did the comprehension do, and how would you design this step so duplicates can't go unnoticed?

Answer:

A dict comprehension assigns by_pan[key] = value once per row. A duplicate key overwrites silently: the last row wins. The key keeps the position of its first insertion, though, so the dict's order doesn't reveal that anything was replaced. Set comprehensions collapse duplicates in the same silent way. Nothing raises and nothing is logged, and the record count just shrinks (3 rows, 2 keys).

This is a data-contract problem more than a Python one. Uniqueness of the business key is an assumption about the partner's data, and it must be checked, not implied by a dict:

• Count before building: Counter(r["pan"] for r in rows), and act on any key with a count above 1.
• Decide the policy explicitly: reject the file, keep the latest row by a timestamp or sequence column, keep the first, or merge fields. "Last row in file order wins" is rarely the right business rule.
• Emit metrics (rows in, unique keys, duplicates) and alert on unexpected changes, and quarantine duplicates for review.
• In pandas, the equivalents are df.duplicated("pan", keep=False), set_index("pan", verify_integrity=True), and merge(..., validate="one_to_one"), all of which raise instead of silently choosing.

from collections import Counter

rows = [
    {"pan": "ABCDE1234F", "name": "Priya", "limit": 50_000},
    {"pan": "PQRSX5678K", "name": "Arjun", "limit": 20_000},
    {"pan": "ABCDE1234F", "name": "Priya S", "limit": 5_000},
]

by_pan = {r["pan"]: r for r in rows}
print(len(rows), len(by_pan), by_pan["ABCDE1234F"]["limit"])
print(list(by_pan))

dupes = [k for k, n in Counter(r["pan"] for r in rows).items() if n > 1]
print(dupes)

Output:

3 2 5000
['ABCDE1234F', 'PQRSX5678K']
['ABCDE1234F']

Follow-ups:

1. "Write a helper that builds the index but raises on duplicates." def index_unique(rows, key): out = {}; for r in rows: k = r[key]; if k in out: raise ValueError(f"duplicate {key}: {k}"); out[k] = r; return out. A loop is clearer than a comprehension here because it needs a check per item.
2. "The partner says duplicates are corrections and the later row is authoritative. Is last-wins fine then?" Only if file order is guaranteed to be chronological. Ask for a sequence number or timestamp and pick the maximum explicitly. Record the agreement in the data contract and test it, so a reordered file doesn't flip limits.

Scoring: Strong: Explains last-wins with first-insertion order, frames it as an unchecked uniqueness assumption, proposes an explicit duplicate policy with metrics and alerting, and knows the pandas validation options. Acceptable: Knows the last row wins and adds a duplicate check. Red flag: Expects an error on duplicate keys, or thinks the first row wins.

SR-TH-CMP-04 · The `next()`-in-a-generator-expression chunking idiom and PEP 479

Medium & Tricky · Theory · Senior (5–8 years) · generator expressions, StopIteration, PEP 479, chunking, itertools.batched, islice · 5 min

Question: An old batching helper does chunk = list(next(it) for _ in range(size)) in a loop and stops when the chunk is empty. It was copied from a Python 3.6 codebase into a new 3.12 upload job. What happens on the last, partial batch?

Answer:

Trap: "It yields the short final chunk [6], then stops." That was the behaviour before Python 3.7.

When it runs out, next(it) raises StopIteration inside the generator expression's frame. Before PEP 479, a StopIteration escaping a generator body looked like a normal end, so the generator expression silently ended early and the idiom "worked", while hiding real bugs elsewhere. Since 3.7, a StopIteration that escapes a generator's body (including a generator expression) is converted to RuntimeError: generator raised StopIteration. The upload job processes the full batches, then crashes on the partial one.

A related, nastier variant: in ordinary (non-generator) code, a stray StopIteration from a bare next() propagates normally. If that code is called from inside a caller's for loop body or __next__, it can end the caller's iteration early, without an error, losing data.

Fixes:

• Use the standard tools: itertools.batched(it, n) (3.12+, yields tuples) or while chunk := list(islice(it, size)):. islice stops cleanly at exhaustion.
• Never call bare next(it) where exhaustion is possible. Use next(it, sentinel) or handle StopIteration locally.

from itertools import batched, islice


def chunks(iterable, size):
    it = iter(iterable)
    while chunk := list(islice(it, size)):
        yield chunk


def chunks_legacy(iterable, size):
    it = iter(iterable)
    while True:
        chunk = list(next(it) for _ in range(size))
        if not chunk:
            return
        yield chunk


print(list(chunks(range(7), 3)))
print(list(batched(range(7), 3)))
print(list(chunks_legacy(range(7), 3)))

Output:

[[0, 1, 2], [3, 4, 5], [6]]
[(0, 1, 2), (3, 4, 5), (6,)]
RuntimeError: generator raised StopIteration

Follow-ups:

1. "Why did the Python core developers make this a hard error rather than keep the old behaviour?" Because silently ending iteration on an accidental StopIteration (from a helper calling next() on the wrong iterator) caused data loss that was very hard to debug. Converting it to RuntimeError makes the bug visible where it happens. The deliberate way to end a generator is return.
2. "Would [next(it) for _ in range(size)], a list comprehension, behave the same?" Not exactly. A list comprehension isn't a generator, so StopIteration propagates as itself. Inside chunks_legacy (a generator) it then escapes the generator body and becomes the same RuntimeError. Called from ordinary code it stays a StopIteration and could silently stop an enclosing for loop.

Scoring: Strong: Predicts the RuntimeError on the last batch, explains PEP 479 and why the pre-3.7 idiom "worked", knows the ordinary-code variant, and fixes with batched/islice. Acceptable: Knows StopIteration inside generators becomes RuntimeError and uses islice. Red flag: Expects [6] as the last chunk, or catches RuntimeError to restore the old behaviour.

SR-TH-CMP-05 · Where comprehension speed comes from, and when to stop optimising Python loops

Hard · Theory · Senior (5–8 years) · performance, comprehensions, PEP 709, profiling, vectorisation, memory · 7 min

Question: A teammate's PR rewrites a 2-million-row transform from a for loop with append into a list comprehension "for speed", reporting a 20% gain. Explain where that gain comes from, what PEP 709 changed in 3.12, and how you'd decide what to do next if the job still misses its SLA.