Staff / Lead 8–12 years

ST-E-O01 · Retries at every layer multiply load on the database that is already down

Easy · Output · Staff / Lead (8–12 years) · reliability, retries, retry amplification, decorators, incident leadership, load shedding · 5 min

Question: Three services in a payments call chain each adopted the platform team's @retry(attempts=3) decorator independently: checkout calls wallet_debit, which calls ledger_write. During an incident the ledger database rejects every write. What does this print?

import functools

calls = {"ledger": 0}


def retry(attempts):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args):
            for i in range(attempts):
                try:
                    return fn(*args)
                except ConnectionError:
                    if i == attempts - 1:
                        raise
        return wrapper
    return deco


@retry(attempts=3)
def ledger_write(txn):
    calls["ledger"] += 1
    raise ConnectionError("ledger DB overloaded")


@retry(attempts=3)
def wallet_debit(txn):
    return ledger_write(txn)


@retry(attempts=3)
def checkout(txn):
    return wallet_debit(txn)


try:
    checkout("TXN-1")
except ConnectionError as e:
    print("checkout failed:", e)
print("ledger calls for one checkout:", calls["ledger"])
print("at 2,000 checkouts/s the ledger sees:", 2_000 * calls["ledger"], "writes/s")

Expected output:

checkout failed: ledger DB overloaded
ledger calls for one checkout: 27
at 2,000 checkouts/s the ledger sees: 54000 writes/s

Explanation:

Each layer's wrapper calls the layer below up to 3 times, and the layer below is itself a wrapper that makes up to 3 calls. Attempts multiply down the chain: 3 × 3 × 3 = 27 calls to ledger_write for one user action. The final ConnectionError is re-raised by the innermost wrapper, caught and retried by each outer wrapper, and finally escapes from checkout's third attempt, so the message is the original one.

This is retry amplification. With k attempts per layer and d layers, a fully failing dependency receives k^d times the normal traffic, exactly when it is least able to absorb it. A ledger that normally sees 2,000 writes/s is hit with 54,000/s while it is trying to recover, which is how a brief overload becomes a long outage (and why "the DB came back and fell over again" is a common incident timeline).

The platform-level fixes, which a Staff engineer should own rather than leave to each team:

• Retry at one layer only, usually the one closest to the user or the one that owns the idempotency key; inner layers fail fast.
• Retry budgets (e.g. retries may add at most 10% to a client's traffic) and circuit breakers, so a dead dependency is not hammered.
• Exponential backoff with jitter, and only for errors that are actually retryable.
• A deadline propagated down the chain, so inner layers stop when the caller has already given up.

Takeaway: retries compose multiplicatively; a standard decorator applied independently by every team is a load amplifier unless the platform sets a policy for where retries live.

Follow-ups:

1. "You are writing the org-wide guideline. Where should retries live in a chain like this?" At one layer, typically the outermost service that can make the operation idempotent (it holds the idempotency key) or the client SDK, with a budget. Inner calls use short timeouts and fail fast, returning a clear retryable/non-retryable error so the one retrying layer can decide. The guideline should be enforced in the shared HTTP/DB client, not left to per-function decorators.
2. "What is a retry budget, and why is it better than a fixed attempt count?" A token-bucket-style limit on retries as a fraction of successful or first-attempt requests (for example retries ≤ 10% of requests over the last minute). When the dependency is healthy, failed requests still get retried; when it is failing broadly, the budget runs out and extra load is capped at +10% instead of ×3 per layer. gRPC and Envoy implement this; a Python SDK can do it with a shared counter per client.
3. "In the incident review, how would you have detected this amplification from dashboards?" The ledger's request rate would jump to a multiple of upstream traffic (27× here) while user-facing checkout rate stays flat. Tagging requests with an attempt number or retry header and graphing first attempts vs retries per caller makes it visible immediately and identifies which layers are retrying.

Scoring: Strong: Gets 27 and 54,000 quickly, names retry amplification, and moves straight to policy: one retrying layer, budgets or circuit breakers, backoff with jitter, deadline propagation, enforced in a shared client. Acceptable: Computes 27 correctly and explains why, and suggests removing inner retries when asked about the incident impact. Red flag: Answers 9 or 3, or treats the fix as 'increase the DB capacity' or 'add more retries with longer sleeps' without seeing the multiplication.

ST-E-O02 · Fan-out turns a backend's p99 into the page's median

Easy · Output · Staff / Lead (8–12 years) · tail latency, fan-out, SLOs, architecture, capacity · 5 min

Question: A product page in a shopping app calls many backend services in parallel (pricing, stock, offers, reviews, delivery estimate per seller...) and renders only when all have answered. Each backend call is slower than its own p99 exactly 1% of the time, independently. What does this print?

p_slow = 0.01            # each backend call is slower than its own p99 1% of the time

for fan_out in (1, 10, 100, 400):
    p_request_slow = 1 - (1 - p_slow) ** fan_out
    print(f"fan-out {fan_out:>3}: {p_request_slow:5.1%} of page loads wait on a slow call")

# To keep fan-out-100 page loads slow only 1% of the time, how rare must a slow backend call be?
per_call = 1 - (1 - 0.01) ** (1 / 100)
print(f"needed per-call slow rate: {per_call:.4%}")

Expected output:

fan-out   1:  1.0% of page loads wait on a slow call
fan-out  10:  9.6% of page loads wait on a slow call
fan-out 100: 63.4% of page loads wait on a slow call
fan-out 400: 98.2% of page loads wait on a slow call
needed per-call slow rate: 0.0100%

Explanation:

A page load is fast only if every backend call is fast. With independent calls that is (1 - p)^n, so the chance at least one is slow is 1 - 0.99^n: about 9.6% at 10 calls, 63.4% at 100 and 98.2% at 400. At a fan-out of 100, the backend's p99 is what the majority of users experience; a "p99 problem" on one service becomes a median problem for the product.

The last line inverts the formula: for only 1% of fan-out-100 page loads to wait on a slow call, each backend must be slow only 1 - 0.99^(1/100) ≈ 0.01% of the time. Each backend now needs a p99.99 target, which is a very different engineering problem from p99. (The exact value is about 0.01005%, printed to four decimal places as 0.0100%.)

Format details: {:5.1%} multiplies by 100, adds %, and pads to width 5, which is why 1.0% gets a leading space; {:>3} right-aligns the fan-out.

Why it matters at Staff level: SLOs for a page cannot be set by giving each team the same p99 target. Architecture choices change the maths: reduce fan-out (aggregate on the backend, batch per-seller calls into one), render progressively with per-widget timeouts and fallbacks so optional calls don't block, or hedge requests (send a second copy after the p95) for idempotent reads.

Takeaway: in a fan-out, tail latency compounds; per-service p99s must be set from the composite page SLO, not copied from a template.

Follow-ups:

1. "Product wants p99 page latency under 800 ms with this 100-way fan-out. What do you change in the architecture rather than in each service?" Make most calls non-blocking: a deadline per widget with a degraded fallback (cached price, 'reviews unavailable'), so only the 3–5 critical calls gate the render. Collapse per-seller calls into a batch endpoint to cut fan-out. Hedge or retry-after-p95 idempotent reads. Then set per-service SLOs only for the critical path.
2. "What does hedging cost, and when is it unsafe?" It adds load: hedging after the p95 sends about 5% extra requests, which is acceptable for reads with spare capacity. It is unsafe for non-idempotent writes and can make overload worse; cap it with a budget and disable it when the backend is saturated.
3. "The formula assumes independence. Is that realistic?" No. Slowness is correlated: a GC pause, a noisy neighbour, a shared cache miss storm or one slow database hits many calls at once. Correlation lowers the probability that some call is slow in normal times but makes the bad cases much worse (everything slow together). Load tests and tracing on real traffic, not the formula, set the final targets.

Scoring: Strong: Gets the numbers or the formula immediately, explains the inversion to p99.99, and proposes architectural levers (fewer blocking calls, fallbacks with deadlines, batching, hedging) and SLO decomposition from the page SLO. Acceptable: Derives 1 - 0.99^n and the approximate values, and understands why per-service p99 targets are insufficient once prompted. Red flag: Says the page is slow 1% of the time because each service is, or sums the probabilities (100% at fan-out 100, 400% at 400).

ST-E-C01 · Which repos block the fleet's move to Python 3.12?

Easy · Coding · Staff / Lead (8–12 years) · migration strategy, Python upgrades, version specifiers, parsing, tuples · 20 min

Problem:

You lead the Python 3.12 upgrade across about 300 repositories. Each repo declares requires-python in its pyproject.toml, for example ">=3.8,<3.12". Before planning waves, you want a report of the repos whose declared range excludes the target version.

Implement:

• allows(spec: str, version: str) -> bool: does the specifier allow this version?
• blocking_repos(repos: dict[str, str], target: str) -> list[str]: names of repos whose requires-python does not allow target, sorted by name.

The specifier subset you must support (it covers every repo in the fleet):

• Comma-separated clauses, all of which must hold. Whitespace around clauses and operators is allowed. An empty or whitespace-only spec allows everything.
• Operators: >=, >, <=, <, ==, !=.
• Versions are dot-separated non-negative integers (3, 3.9, 3.12.4). Missing trailing components count as zero, so 3.12 equals 3.12.0.
• == and != may end in .* as a prefix match: ==3.11.* allows 3.11 and 3.11.9 but not 3.12.
• Anything else (unknown operator, non-numeric version) raises ValueError.

Signature: def allows(spec: str, version: str) -> bool: ... def blocking_repos(repos: dict[str, str], target: str) -> list[str]: ...

Constraints: Standard library only (the real tool, packaging.specifiers.SpecifierSet, is not available in this exercise). Up to a few thousand repos; performance is not a concern, correctness is.

Examples:

• allows('>=3.8,<3.12', '3.12') → False
• allows('>=3.9', '3.12') → True
• allows('==3.11.*', '3.11.7') → True
• blocking_repos({'billing': '>=3.8,<3.12', 'search': '>=3.10', 'kyc': '~=3.9'}, '3.12') → ValueError (the '~=' operator is outside the supported subset)

Hints:

1. Version strings do not compare correctly as strings: '3.12' < '3.9' is True. What type compares component by component?
2. Normalise both sides to tuples of ints of the same length before comparing, so 3.12 and 3.12.0 are equal.
3. For ==X.Y.*, compare only the first len(prefix) components of the version.
4. Check two-character operators (>=, <=, ==, !=) before one-character ones.

Reference solution:

import operator

_OPS = {
    ">=": operator.ge, "<=": operator.le, "==": operator.eq,
    "!=": operator.ne, ">": operator.gt, "<": operator.lt,
}


def _parse_version(text: str) -> tuple[int, ...]:
    parts = text.strip().split(".")
    if not parts or not all(p.isdigit() for p in parts):
        raise ValueError(f"invalid version: {text!r}")
    return tuple(int(p) for p in parts)


def _pad(a: tuple[int, ...], n: int) -> tuple[int, ...]:
    return a + (0,) * (n - len(a))


def _clause_allows(clause: str, version: tuple[int, ...]) -> bool:
    clause = clause.strip()
    for symbol in (">=", "<=", "==", "!=", ">", "<"):      # two-character operators first
        if clause.startswith(symbol):
            rest = clause[len(symbol):].strip()
            break
    else:
        raise ValueError(f"unsupported clause: {clause!r}")

    if rest.endswith(".*"):
        if symbol not in ("==", "!="):
            raise ValueError(f"wildcard only allowed with == or !=: {clause!r}")
        prefix = _parse_version(rest[:-2])
        matches = _pad(version, len(prefix))[: len(prefix)] == prefix
        return matches if symbol == "==" else not matches

    bound = _parse_version(rest)
    n = max(len(bound), len(version))
    return _OPS[symbol](_pad(version, n), _pad(bound, n))


def allows(spec: str, version: str) -> bool:
    v = _parse_version(version)
    clauses = [c for c in spec.split(",") if c.strip()]
    return all(_clause_allows(c, v) for c in clauses)


def blocking_repos(repos: dict[str, str], target: str) -> list[str]:
    return sorted(name for name, spec in repos.items() if not allows(spec, target))

Complexity: O(c × k) per call, for c clauses and k version components; O(r) specifier evaluations for r repos plus O(r log r) to sort the names. Memory is O(r) for the result.

Tests:

assert allows(">=3.8,<3.12", "3.11") is True
assert allows(">=3.8,<3.12", "3.12") is False
assert allows(">=3.9", "3.12") is True                 # string comparison says False
assert allows("<3.10", "3.9.18") is True
assert allows("<3.10", "3.10") is False
assert allows("<=3.12", "3.12.0") is True              # 3.12 == 3.12.0
assert allows(">3.12", "3.12.0") is False
assert allows(">3.12", "3.12.1") is True
assert allows("!=3.12.0", "3.12") is False
assert allows("==3.11.*", "3.11.7") is True
assert allows("==3.11.*", "3.12") is False
assert allows("!=3.11.*", "3.12") is True
assert allows("==3.1.*", "3.12") is False              # prefix is by component, not by text
assert allows(" >= 3.10 ,  < 4 ", "3.12") is True
assert allows("", "3.12") is True
assert allows("   ", "3.12") is True

fleet = {
    "billing": ">=3.8,<3.12",
    "search": ">=3.10",
    "notifications": ">=3.9,!=3.12.0",
    "kyc-ocr": "==3.10.*",
    "growth": ">=3.11",
    "legacy-reports": "<3.9",
}
assert blocking_repos(fleet, "3.12") == ["billing", "kyc-ocr", "legacy-reports", "notifications"]
assert blocking_repos(fleet, "3.12.1") == ["billing", "kyc-ocr", "legacy-reports"]
assert blocking_repos({}, "3.12") == []

for bad in ["~=3.9", ">=3.x", "=>3.9", ">=3.9.*", "3.9"]:
    try:
        allows(bad, "3.12")
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted {bad!r}")

A common wrong solution (the tests catch it):

def allows(spec: str, version: str) -> bool:
    v = version.strip()
    for clause in spec.split(","):
        clause = clause.strip()
        if not clause:
            continue
        for symbol in (">=", "<=", "==", "!=", ">", "<"):
            if clause.startswith(symbol):
                bound = clause[len(symbol):].strip()
                break
        else:
            raise ValueError(f"unsupported clause: {clause!r}")
        if not bound.replace(".", "").replace("*", "").isdigit():
            raise ValueError(f"invalid version: {bound!r}")
        if bound.endswith(".*"):
            if symbol not in ("==", "!="):
                raise ValueError("wildcard only with == or !=")
            ok = v.startswith(bound[:-2])
            ok = ok if symbol == "==" else not ok
        else:
            ok = {">=": v >= bound, "<=": v <= bound, "==": v == bound,
                  "!=": v != bound, ">": v > bound, "<": v < bound}[symbol]
        if not ok:
            return False
    return True


def blocking_repos(repos: dict[str, str], target: str) -> list[str]:
    return sorted(name for name, spec in repos.items() if not allows(spec, target))

Alternatives and tradeoffs:

• packaging.specifiers.SpecifierSet is what you would use in the real fleet tool: it implements all of PEP 440 (~=, pre-releases, local versions, ===). Hand-rolling is only defensible for an exercise or a constrained subset; the Staff-level point is knowing which tool is authoritative (pip uses packaging).
• Comparing strings (the broken version) is the classic bug: '3.12' >= '3.9' is False, so most modern repos look "blocked", and startswith('3.1') makes ==3.1.* match 3.12.
• Regex parsing of clauses is fine as long as the result is still compared as integer tuples.
• At fleet scale the report should also read the lockfile / CI matrix, not only requires-python: a repo that allows 3.12 on paper may pin a dependency (an old numpy, pydantic 1) with no 3.12 wheels.

Follow-ups:

1. "This report says 60 repos block 3.12. How do you turn it into a migration plan?" Group blockers by cause (only the metadata cap vs a real dependency without 3.12 wheels vs code using removed modules like distutils/imp). Fix shared dependencies and internal libraries first, because product repos depend on them. Then run waves: low-risk internal tools, then services behind canaries, then the critical-path services, each with 3.12 added to the CI matrix before switching the runtime image. Track per-repo status on a dashboard, with a deadline tied to the 3.8/3.9 end-of-life date.
2. "Many repos say <3.12 only because someone copied a template. Is it safe to just drop the upper bound everywhere?" For applications, removing the cap after CI passes on 3.12 is fine. For libraries published to an index, upper bounds on requires-python are generally discouraged because resolvers backtrack to old releases instead of failing clearly. The change should be made per repo, gated on its CI matrix including 3.12, not by a blind fleet-wide edit.
3. "How would you keep this from regressing after the upgrade is done?" A central CI template that runs the supported versions matrix, a policy check that flags requires-python caps below the org's supported floor, and a dependency bot that keeps pins current so the next upgrade (3.13/3.14) starts from a healthy baseline.

Scoring: Strong: Parses versions into integer tuples with zero padding, handles the wildcard by component prefix, rejects unsupported operators, and mentions that packaging is the real answer. Turns the report into a dependency-ordered wave plan when asked. Acceptable: Correct tuple comparison and clause handling, but misses padding (3.12 vs 3.12.0) or the component-wise wildcard until a test fails, then fixes it. Red flag: Compares versions as strings or floats (float('3.12') < float('3.9')), or treats the report as the whole migration plan without considering dependencies.

ST-E-C02 · Multi-window burn-rate alerting for an SLO

Easy · Coding · Staff / Lead (8–12 years) · SLOs, error budgets, alerting, reliability, on-call · 20 min

Problem:

Your org is replacing threshold alerts ("page if error rate > 1% for 5 minutes") with SLO burn-rate alerts for every tier-1 Python service. You are writing the shared evaluation function that the alerting job calls once a minute per service.

Definitions:

• The error budget is 1 - slo (for slo = 0.999, 0.1% of requests may fail).
• The burn rate over a window is (errors / total) / (1 - slo). A burn rate of 1 spends exactly the 30-day budget in 30 days; 14.4 spends 2% of it in one hour. A window with total == 0 has burn rate 0.
• Each rule has a long window and a short window and fires only if both burn rates are at or above its threshold. The long window shows the problem is significant; the short window shows it is still happening, so the alert stops soon after recovery.

Implement alert_level(slo, counts), where counts maps each window name ("5m", "30m", "1h", "6h", "3d") to (errors, total). Return "page" if any page rule fires, otherwise "ticket" if the ticket rule fires, otherwise "ok". Raise ValueError if slo is not strictly between 0 and 1, if a window is missing, or if any count is negative or errors > total.

Signature: def alert_level(slo: float, counts: dict[str, tuple[int, int]]) -> str:

Constraints: Standard library only. The function is pure: it gets counts from the caller (who queries Prometheus or the log store) and must not do I/O.

Examples:

• alert_level(0.999, {'5m': (40, 2000), '30m': (200, 12000), '1h': (400, 24000), '6h': (500, 144000), '3d': (900, 1728000)}) → 'page' (1h burn ≈ 16.7 and 5m burn = 20, both ≥ 14.4)
• same, but '5m': (0, 2000) → 'ok' (the incident has stopped; no rule has both windows over threshold)

Hints:

1. Write a small burn(window) helper that handles total == 0.
2. Express the three rules as data (a list of tuples), not as nested ifs. The platform team will tune them.
3. Check all page rules before the ticket rule.

Reference solution:

WINDOWS = ("5m", "30m", "1h", "6h", "3d")

# (long window, short window, burn-rate threshold, result), most severe first
RULES = [
    ("1h", "5m", 14.4, "page"),
    ("6h", "30m", 6.0, "page"),
    ("3d", "6h", 1.0, "ticket"),
]


def alert_level(slo: float, counts: dict[str, tuple[int, int]]) -> str:
    if not 0 < slo < 1:
        raise ValueError("slo must be strictly between 0 and 1")
    budget = 1 - slo
    for w in WINDOWS:
        if w not in counts:
            raise ValueError(f"missing window {w}")
        errors, total = counts[w]
        if errors < 0 or total < 0 or errors > total:
            raise ValueError(f"bad counts for {w}: {counts[w]}")

    def burn(window: str) -> float:
        errors, total = counts[window]
        return 0.0 if total == 0 else (errors / total) / budget

    for long_w, short_w, threshold, result in RULES:
        if burn(long_w) >= threshold and burn(short_w) >= threshold:
            return result
    return "ok"

Complexity: O(1): five windows and three rules. The real cost is in the caller's queries; for 200 services evaluated every minute, precomputing per-window counts as recording rules (rather than raw range queries) keeps the alerting job cheap.

Tests:

def c(m5, m30, h1, h6, d3):
    return {"5m": m5, "30m": m30, "1h": h1, "6h": h6, "3d": d3}

quiet = (0, 100_000)
# fast burn: 1h at 2% errors (burn 20), 5m at 2% (burn 20)
assert alert_level(0.999, c((40, 2000), (200, 12000), (480, 24000), (500, 144000), (900, 1_728_000))) == "page"
# same hour, but the last 5 minutes are clean -> the incident is over, don't page
assert alert_level(0.999, c((0, 2000), (200, 12000), (480, 24000), (500, 144000), (900, 1_728_000))) == "ok"
# a short spike: 5m is terrible but the hour is fine -> not significant yet
assert alert_level(0.999, c((100, 2000), (100, 12000), (100, 24000), (100, 144000), (100, 1_728_000))) == "ok"
# slow burn: 6h at 0.8% (burn 8), 30m at 0.7% (burn 7), 1h only at 0.75% (below 14.4)
assert alert_level(0.999, c((14, 2000), (84, 12000), (180, 24000), (1152, 144000), (1300, 1_728_000))) == "page"
# budget leak: 3d at 0.15% (burn 1.5), 6h at 0.12% (burn 1.2)
assert alert_level(0.999, c((3, 2000), (18, 12000), (36, 24000), (173, 144000), (2592, 1_728_000))) == "ticket"
# the 3d window is leaking but the last 6h are clean -> ok
assert alert_level(0.999, c(quiet, quiet, quiet, (10, 144000), (2592, 1_728_000))) == "ok"
# a looser SLO changes the verdict for the same counts: 99% -> budget 1%, burn 2 -> ok
assert alert_level(0.99, c((40, 2000), (200, 12000), (480, 24000), (500, 144000), (900, 1_728_000))) == "ok"
# no traffic at all is not an error
assert alert_level(0.999, c((0, 0), (0, 0), (0, 0), (0, 0), (0, 0))) == "ok"

for bad_slo in (0, 1, 1.5, -0.1):
    try:
        alert_level(bad_slo, c(quiet, quiet, quiet, quiet, quiet))
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted slo={bad_slo}")
for bad in ({"5m": quiet}, c((5, 4), quiet, quiet, quiet, quiet), c((-1, 10), quiet, quiet, quiet, quiet)):
    try:
        alert_level(0.999, bad)
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted {bad}")

A common wrong solution (the tests catch it):

def alert_level(slo: float, counts: dict[str, tuple[int, int]]) -> str:
    if not 0 < slo < 1:
        raise ValueError("slo must be strictly between 0 and 1")
    budget = 1 - slo
    for w in ("5m", "30m", "1h", "6h", "3d"):
        if w not in counts:
            raise ValueError(f"missing window {w}")
        errors, total = counts[w]
        if errors < 0 or total < 0 or errors > total:
            raise ValueError(f"bad counts for {w}: {counts[w]}")

    def burn(window: str) -> float:
        errors, total = counts[window]
        return 0.0 if total == 0 else (errors / total) / budget

    # single-window rules: simpler, "the long window is what matters"
    if burn("1h") >= 14.4 or burn("6h") >= 6:
        return "page"
    if burn("3d") >= 1:
        return "ticket"
    return "ok"

Alternatives and tradeoffs:

• Single-window burn rate (the broken version): it keeps paging for up to an hour after the incident ends, because the 1h window still contains the bad minutes. That is exactly the alert fatigue the multi-window design removes.
• Plain error-rate thresholds ("> 1% for 5 minutes") ignore traffic volume and the SLO; they page on 3 failures out of 200 requests at night and miss slow leaks entirely.
• Prometheus/Alertmanager rules or a vendor SLO product can implement the same logic declaratively. A shared Python evaluator is useful when SLIs come from logs or a warehouse, or to unit-test the rules the platform publishes.
• The thresholds come from budget arithmetic: 14.4 = 0.02 × 30 days × 24 h / 1 h, i.e. spending 2% of a 30-day budget in an hour.

Follow-ups:

1. "A teammate adds a test exactly at the threshold, 144 errors in 10,000 requests with slo=0.999, and the rule doesn't fire. Why?" 1 - 0.999 is 0.0010000000000000009 in binary floating point, so the burn rate computes as 14.399999999999986, just under 14.4. Fix by doing the comparison without the subtraction (errors * 1000 >= 14.4 * total for a given SLO), with fractions.Fraction(str(slo)), or by expressing the SLO in parts per million as an integer. In practice nobody cares about the exact boundary, but a shared library's tests will.
2. "A low-traffic internal service gets 30 requests an hour. What goes wrong with these rules, and what do you do?" One failure is a 3.3% error rate, burn rate 33, so a single bad request pages someone at 3 a.m. Options: a minimum request count per window before a rule can fire, synthetic probes to add traffic, grouping low-traffic services into one SLO, or a looser SLO. It is a policy decision for the platform, written down, not each team's choice.
3. "How do you roll this standard out to 60 teams without drowning them in pages in week one?" Run it in shadow mode first: evaluate and log what would have paged next to the existing alerts for 2–4 weeks, review the diff with each team, fix SLI definitions (exclude 4xx caused by clients, health checks), then switch teams over in batches and delete the old threshold alerts so on-call isn't double-paged.

Scoring: Strong: Implements both-windows-must-agree rules as data, handles zero traffic and validation, explains why the short window prevents post-incident paging, and discusses low-traffic services or the floating-point boundary without prompting. Rollout plan includes shadow mode. Acceptable: Correct rules and ordering after reading the table carefully; needs a hint for zero-traffic windows or for why both windows are required. Red flag: Implements single-window checks or plain error-rate thresholds, or can't explain what a burn rate of 1 means in terms of the 30-day budget.

ST-E-C03 · Pod count from Little's law that survives losing a zone

Easy · Coding · Staff / Lead (8–12 years) · capacity planning, Little's law, availability zones, cost, gunicorn workers · 20 min

Problem:

Before the Big Billion Days-style sale, every team running a Python web service must submit a capacity number for its peak. You are writing the shared calculator that the platform team will use, so teams stop guessing.

Each service runs synchronous workers (for example gunicorn sync workers): one worker handles one request at a time. Implement pods_needed(peak_rps, mean_latency_ms, workers_per_pod, target_util_pct, zones):

1. Little's law: requests in flight at peak = peak_rps × mean_latency_ms / 1000. That is the number of busy workers.
2. Workers must run at no more than target_util_pct percent busy, so workers needed = ceil(busy / (target_util_pct / 100)).
3. Pods needed to carry the peak = ceil(workers / workers_per_pod), and at least 1.
4. Pods are spread evenly across zones availability zones, and the service must still carry the full peak after losing any one zone. Return the total pod count (a multiple of zones). With zones == 1 there is no zone redundancy; return the step 3 number.

Raise ValueError if peak_rps < 0, mean_latency_ms <= 0, workers_per_pod <= 0, target_util_pct is not in 1..100, or zones < 1. All inputs are integers.

Signature: def pods_needed(peak_rps: int, mean_latency_ms: int, workers_per_pod: int, target_util_pct: int, zones: int) -> int:

Constraints: Integers in, integer out. Avoid float rounding in the ceilings: 1000 * 0.12 style products can land a hair above an integer and add a pod.

Examples:

• pods_needed(1200, 250, 4, 60, 3) → 189 (300 busy → 500 workers → 125 pods → 63 per zone so 2 zones carry 126)
• pods_needed(1200, 250, 4, 60, 1) → 125
• pods_needed(0, 100, 4, 60, 3) → 3 (at least one pod, which still has to survive a zone loss)

Hints:

1. Little's law: L = λ × W. Mind the units: requests per second times seconds.
2. Integer ceiling division: -(-a // b).
3. After losing one of z zones, z - 1 zones' worth of pods must carry the peak. How many pods per zone does that need?

Reference solution:

def _ceil_div(a: int, b: int) -> int:
    return -(-a // b)


def pods_needed(peak_rps: int, mean_latency_ms: int, workers_per_pod: int,
                target_util_pct: int, zones: int) -> int:
    if peak_rps < 0:
        raise ValueError("peak_rps must be >= 0")
    if mean_latency_ms <= 0 or workers_per_pod <= 0:
        raise ValueError("latency and workers_per_pod must be positive")
    if not 1 <= target_util_pct <= 100:
        raise ValueError("target_util_pct must be in 1..100")
    if zones < 1:
        raise ValueError("zones must be >= 1")

    # busy workers = rps * latency_ms / 1000; workers = busy / (pct / 100)
    workers = _ceil_div(peak_rps * mean_latency_ms * 100, 1000 * target_util_pct)
    pods_for_peak = max(1, _ceil_div(workers, workers_per_pod))
    if zones == 1:
        return pods_for_peak
    per_zone = _ceil_div(pods_for_peak, zones - 1)      # survive the loss of any one zone
    return per_zone * zones

Complexity: O(1) time and space. The interesting cost is the answer: here, surviving a zone loss with 3 zones costs 189 pods instead of 126 (+50%), which is the number to put in front of whoever owns the cloud bill.

Tests:

assert pods_needed(1200, 250, 4, 60, 3) == 189
assert pods_needed(1200, 250, 4, 60, 1) == 125
assert pods_needed(1200, 250, 4, 60, 2) == 250        # two zones: each must carry everything
assert pods_needed(0, 100, 4, 60, 3) == 3
assert pods_needed(0, 100, 4, 60, 1) == 1
# exact boundaries must not round up: 1000 rps * 120 ms = 120 busy, at 60% -> 200 workers -> 50 pods
assert pods_needed(1000, 120, 4, 60, 1) == 50
# 100% target utilisation: 50 busy workers, 5 per pod -> 10 pods; 3 zones -> 5 per zone
assert pods_needed(500, 100, 5, 100, 3) == 15
# one extra in-flight request needs a whole extra worker, and maybe a pod
assert pods_needed(1001, 120, 4, 60, 1) == 51
# more zones need less headroom: 125 pods for peak
assert pods_needed(1200, 250, 4, 60, 4) == 4 * 42
assert pods_needed(1200, 250, 4, 60, 6) == 6 * 25

for bad in [(-1, 100, 4, 60, 3), (10, 0, 4, 60, 3), (10, 100, 0, 60, 3),
            (10, 100, 4, 0, 3), (10, 100, 4, 101, 3), (10, 100, 4, 60, 0)]:
    try:
        pods_needed(*bad)
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted {bad}")

A common wrong solution (the tests catch it):

import math


def pods_needed(peak_rps: int, mean_latency_ms: int, workers_per_pod: int,
                target_util_pct: int, zones: int) -> int:
    if peak_rps < 0:
        raise ValueError("peak_rps must be >= 0")
    if mean_latency_ms <= 0 or workers_per_pod <= 0:
        raise ValueError("latency and workers_per_pod must be positive")
    if not 1 <= target_util_pct <= 100:
        raise ValueError("target_util_pct must be in 1..100")
    if zones < 1:
        raise ValueError("zones must be >= 1")
    busy = peak_rps * mean_latency_ms / 1000
    workers = math.ceil(busy / (target_util_pct / 100))
    pods = max(1, math.ceil(workers / workers_per_pod))
    return math.ceil(pods / zones) * zones          # spread evenly across zones

Alternatives and tradeoffs:

• Spreading evenly without zone-loss headroom (the broken version) gives 126 pods for 3 zones: fine until a zone fails during the sale, when 84 pods face the full peak at about 90% utilisation and latency climbs. It also uses floats, which adds a pod at exact boundaries such as 1000 rps × 120 ms.
• Autoscaling instead of a fixed number: still needs this calculation for the maximum and for pre-scaling, because scale-up takes minutes while a sale's traffic jump takes seconds, and the cluster needs quota and node capacity in advance.
• Async workers (uvicorn with an async stack) change step 1: in-flight requests are no longer bounded by worker count but by connection pools, CPU and downstream limits, so the binding constraint has to be measured, not assumed.
• Use mean latency in Little's law, not p99; p99 drives timeouts and SLOs, mean drives concurrency.

Follow-ups:

1. "The DB behind this service allows 2,000 connections, and each worker holds one. Does 189 pods × 4 workers fit, and what do you do if it doesn't?" 756 connections fit, but check every service sharing that database and the replica count during deploys (a rolling update can briefly run both old and new pods). If it doesn't fit, put a pooler (PgBouncer) in transaction mode between the pods and the DB, reduce per-pod connections, or move read traffic to replicas. Capacity planning has to include the shared dependencies, not just the stateless tier.
2. "Finance asks why the bill for this service is 50% higher than 'what we need'. How do you explain and decide?" The extra third of a zone per zone is the price of N+1 zone redundancy: without it a zone outage during peak becomes a customer-facing outage. Present it as a choice with the cost per hour (pods × hourly rate in ₹) against the revenue at risk during a sale hour, and offer middle grounds: full headroom only during sale windows, or graceful degradation (shed recommendations, keep checkout) instead of full headroom for every service.
3. "Where do the inputs come from, and which one is most often wrong?" Peak RPS from last year's sale plus the growth forecast; mean latency and utilisation from production metrics under load, ideally a load test at the target peak. Latency is the one most often wrong: it rises with load (queueing, cache misses, DB contention), so a mean measured at normal traffic underestimates in-flight requests at peak.

Scoring: Strong: Applies Little's law with correct units, integer ceilings, and the ceil(pods / (z - 1)) × z zone-loss rule; immediately quantifies the headroom cost and brings up shared dependencies (DB connections) and measuring latency under load. Acceptable: Gets Little's law and the per-pod maths right, but spreads evenly across zones until asked what happens when a zone fails, then fixes it. Red flag: Sizes from CPU averages or from p99 latency, ignores zone failure entirely, or can't connect workers in flight to requests per second.

ST-E-D01 · Moving 300 repositories from Python 3.8/3.9 to 3.12

Easy · Design · Staff / Lead (8–12 years) · migration strategy, Python upgrades, dependency management, CI, standards, program management · 40 min

Scenario:

A Pune-based lending company has about 300 Python repositories owned by 35 teams: FastAPI and Django services, Airflow DAGs, Spark/pandas jobs, ML training code and internal libraries. Roughly 60% run on Python 3.8, 30% on 3.9 and the rest on 3.10 or 3.11. Base images are built per team; there is no single dependency policy. Python 3.8 and 3.9 are both past end-of-life and no longer get security fixes; the security team has flagged them for the next RBI-driven audit.

You are the Staff engineer asked to own the upgrade to Python 3.12 across the fleet within two quarters, without a feature freeze and without causing incidents in the loan-disbursal path. Design the programme: technical approach, sequencing, tooling, and how you will know you're done.

Clarifying questions:

• Why 3.12 rather than the newest release? (The data platform's key dependencies and the vendor-supplied OCR library are certified on 3.12; the platform team will move to a newer version in the following cycle.)
• {'Do we know what runs where? Is there an inventory of repos, their owners, Python versions, base images and deploy targets? (Partly': 'a service catalogue lists services but not batch jobs or notebooks.)'}
• {'Is there a shared CI template or does every repo have its own pipeline? (Mixed': 'about half use a shared GitHub Actions workflow.)'}
• Which internal libraries are imported by many repos? (A lending-common package used by ~180 repos, a Kafka client wrapper and an internal auth SDK.)
• What's the test coverage of the critical path (disbursal, repayments, KYC)? Are there staging environments with production-like data?
• Do any repos depend on native extensions or old pinned versions (numpy 1.19, pandas 1.1, pydantic 1, an old cryptography)?
• What is the appetite for central tooling (a platform team, a dependency bot) vs asking every team to do the work?
• Is "done" defined as all production workloads on 3.12, or also every dev environment and notebook?

Reference design:

Principle: make the upgrade a platform capability, not 300 one-off projects. Most of the effort is shared (base images, CI, internal libraries, common dependency bumps); do that once, centrally, and leave teams with a small, well-scripted change.

1. Inventory (weeks 1–2). A script that scans every repo: requires-python, Python version in Dockerfiles/CI, lockfile contents, imports of removed or changed stdlib modules (distutils, imp, asynchat/asyncore, smtpd), deprecated APIs (datetime.utcnow, pkg_resources, asyncio.get_event_loop without a running loop), and dependency pins without 3.12 wheels. Output: a dashboard per repo with owner, criticality tier and blockers. This also fixes the "we don't know what runs where" problem permanently.

2. Unblock the dependency graph first (weeks 2–6). - Upgrade internal libraries (lending-common, Kafka wrapper, auth SDK) to support 3.8–3.12 simultaneously, tested in a CI matrix. Product repos can't move until their imports can. - Publish golden base images for 3.12 (slim, with the same OS packages and CA bundle) and a shared CI workflow that runs the test suite on the current version and 3.12. - Pick a supported set of key third-party versions (numpy 1.26+/2.x, pandas 2.x, pydantic 2, SQLAlchemy 2) and decide which big upgrades ride along. Rule of thumb: do not bundle pydantic 1→2 or SQLAlchemy 1.4→2 into the interpreter upgrade unless the old version blocks 3.12; separate changes, separate rollbacks.

3. Make each repo's change small and boring. A codemod/bot opens a PR per repo: bump base image, CI matrix, requires-python, lockfile refresh, automated fixes (pyupgrade/ruff rules for removed APIs). Teams review, run their tests, deploy. Provide a migration guide with known issues and fixes, and office hours.

4. Waves by risk, not by team. Wave 0: internal tools and libraries. Wave 1: batch jobs and low-tier services. Wave 2: tier-2 services. Wave 3: disbursal/repayment/KYC paths, each with a canary deployment (one pod or 5% of traffic on 3.12) and comparison of error rate, latency, memory and outputs. Data jobs get a shadow run that compares outputs on the old and new interpreter (row counts, aggregates, model scores within tolerance) before switching.

5. Guardrails and done-ness. Track per-repo status on the dashboard; report weekly to engineering leadership. Once a repo is on 3.12, CI fails if the version regresses. Turn DeprecationWarnings into errors in test runs (-W error::DeprecationWarning scoped to first-party code) so the next upgrade starts clean. "Done" = all production workloads on 3.12 images, old images removed from the registry, 3.8/3.9 blocked in CI templates. Exceptions need a named owner and an expiry date.

6. Make the next upgrade cheaper. A standing policy: support the latest two minor versions, an annual upgrade window tied to the Python release calendar (October), the dependency bot always on. The real deliverable is that 3.13/3.14 is a two-week exercise, not a two-quarter one.

Trade-offs:

• Central platform team does the work vs each team does its own. Central work on shared pieces (images, CI, libraries, codemods) is far cheaper than 35 teams rediscovering the same issues; but each team must own testing and deploying its service, since only they know its behaviour. A hybrid with a central "migration squad" for the long tail of unowned repos works best.
• Jump straight to 3.12 vs step through 3.10/3.11. Going directly avoids paying deployment and validation costs twice; the changes between 3.8 and 3.12 are well documented. Stepping only makes sense if a critical dependency supports 3.11 but not yet 3.12.
• Bundle library upgrades vs keep them separate. Bundling reduces the number of deploys but makes failures hard to attribute and rollbacks coarse. Keep the interpreter change as close to a no-op as possible.
• Deadline pressure vs safety. A hard date (audit) motivates teams; a hard date without a canary/rollback path invites incidents. Make rollback a one-line image revert and rehearse it.
• Monorepo-style central upgrade vs per-repo PRs. Per-repo PRs respect ownership but take longer; a central bot generating them gets most of the speed without taking ownership away.

Deep-dive probes:

• "lending-common pins pydantic 1, and 120 services rely on its v1 models." Look for: decoupling the interpreter upgrade from pydantic 2 (pydantic 1.10 supports 3.12), so the 3.12 move isn't blocked; a separate plan for pydantic 2 using the pydantic.v1 compatibility namespace and gradual migration of models.
• "A model-scoring job gives slightly different scores on 3.12." Look for: is it the interpreter or a dependency bump (numpy, scikit-learn, BLAS)? Shadow comparison with tolerances, pinning versions to isolate the change, deciding with the model owners whether the difference is acceptable, and documenting it for model governance.
• "Thirty repos have no owner and no tests." Look for: identify consumers from deploy logs and imports; decommission what's unused (often a third); for the rest, the migration squad adds smoke tests and upgrades; record ownership in the catalogue. Don't let orphans block the 'done' definition indefinitely, but don't silently leave them on an EOL interpreter either.
• "How do you know an upgraded service is actually healthy?" Look for: canary with automated comparison (error rate, p99, memory, CPU; Python 3.12 often changes memory and performance profiles), business metrics on critical paths (disbursals per minute), and a defined bake time before full rollout.

Rubric: Strong: Treats it as a programme with an inventory, a dependency-ordered plan (internal libraries and base images first), automation (codemods, CI matrix, dashboard), risk-based waves with canaries and shadow runs, clear done criteria, and a policy that makes the next upgrade cheap. Separates the interpreter change from big library migrations and can explain why. Acceptable: Proposes a sensible per-repo plan with CI testing and staged rollout, but misses the dependency ordering through internal libraries or the need for an inventory and central tooling. Red flag: "Ask each team to upgrade by the deadline", a big-bang switch of all base images, or bundling every library upgrade into one change with no canary or rollback plan.

ST-E-D02 · An internal Python platform SDK used by 40 teams

Easy · Design · Staff / Lead (8–12 years) · SDK design, API design, versioning, deprecation, observability, platform engineering · 40 min

Scenario:

A Hyderabad-based logistics company has 40 teams writing Python services and jobs. Every team has its own code for the same cross-cutting concerns: reading config and secrets, structured logging, metrics and tracing, calling internal services over HTTP (auth, retries, timeouts), and publishing events to Kafka. Incidents keep coming from the differences: one team's client retries non-idempotent POSTs, another logs full customer addresses and phone numbers, a third has no timeouts at all.

The CTO asks you to design an internal Python SDK (lp-platform) that all teams will use. Describe its scope, API shape, packaging and versioning, how it evolves without breaking 40 teams, and how you drive adoption.

Clarifying questions:

• What Python versions and frameworks are in use? (3.10–3.12; FastAPI, Django, Celery, Airflow; a few asyncio-heavy services.)
• Is there a common deployment platform (Kubernetes, a standard base image), and is there already an observability stack (OpenTelemetry collector, Prometheus, a log pipeline)?
• Which of these concerns cause the most incidents or audit findings today? (Missing timeouts and PII in logs.)
• Is there a platform team that will own and support the SDK, with on-call? How many engineers?
• Are there non-Python services that must interoperate (same headers, same event envelope)?
• Can the SDK be mandatory for new services? For existing ones?

Reference design:

Scope: small, opinionated, and only cross-cutting concerns. The SDK owns the things every service needs and that are dangerous to get wrong: config/secrets, logging (with PII redaction), telemetry, an HTTP client with safe defaults, and an event publisher with a standard envelope. It does not own business logic, a web framework, or an ORM. "Batteries for platform concerns, not a framework."

Package structure. One distribution lp-platform with optional extras so services don't pull what they don't use: lp-platform[fastapi], [django], [kafka], [celery]. Submodules with a narrow public surface (lp_platform.config, .log, .http, .events, .telemetry); everything else is underscored and documented as private. Type hints throughout and py.typed so mypy/pyright users get checking.

API shape, with safe defaults. - http.Client(service="pricing"): built on httpx, with mandatory timeouts, retries only for idempotent methods or when an idempotency key is passed, retry budgets, propagation of trace and request IDs, standard metrics. Sync and async variants share configuration. - log.get_logger(__name__): structured JSON, request/trace IDs from contextvars, a redaction filter for known PII fields (phone, Aadhaar, address) that is on by default. - events.Publisher: standard envelope (event ID, type, schema version, produced_at in UTC, trace ID), schema validation with pydantic models, and delivery guarantees spelled out. - config.load(Settings): pydantic-settings models, secrets from the vault, fail fast at startup on missing values. - platform.init(app): one call that wires middleware, telemetry and logging for FastAPI/Django; no import-time side effects: importing the SDK must not configure logging, start threads or touch the network.

Versioning and evolution. Semantic versioning with a written compatibility policy: no breaking changes in minor releases; deprecations emit DeprecationWarning with the replacement and removal version, stay for at least two minor releases or 6 months; a changelog that leads with "action needed". Support a window of Python versions that matches the org policy. Wide dependency ranges (not exact pins) for things services also use (httpx, pydantic), with CI testing against the lowest and highest supported versions, so the SDK never becomes the reason a team can't upgrade a library.

Quality bar. Contract tests for public APIs, a test matrix across Python versions and frameworks, and a set of consumer canaries: a handful of real services that run SDK release candidates in staging before release. Telemetry on SDK version per service so you know who is on what.

Adoption. Make the SDK the easiest path: service templates (cookiecutter) that start with it, a migration guide per concern, and the platform team doing the first migrations with two or three pilot teams to learn the rough edges. Mandatory for new services; for existing ones, prioritise by incident risk (services without timeouts first). Track adoption on a dashboard by team. A dependency bot keeps everyone within the supported window (e.g. last two minor versions).

Ownership. A platform team with documented SLAs for bugs, office hours, an RFC process for new features, and a contribution model so product teams can add things without waiting for the platform team.

Trade-offs:

• Thin library vs framework. A framework (base classes, its own app structure) gives more consistency but couples every team to platform release cycles and makes leaving expensive. A thin library of composable pieces is easier to adopt incrementally and to replace.
• One package vs many small packages. One package with extras gives a single version to reason about and test; many packages allow independent releases but create a compatibility matrix nobody can test. Start with one.
• Pinning dependencies vs wide ranges. Pins make the SDK reproducible but force every consumer onto the same versions and block upgrades; ranges with min/max CI testing are the library norm.
• Mandate vs incentives. Mandates speed adoption but generate resentment if the SDK is immature; build credibility with pilot teams first, then make it the default via templates and guardrails.
• Sidecar/service mesh vs library. Retries, mTLS and some telemetry can live in a mesh (language-agnostic, no code change), but PII redaction, event envelopes and config are application-level. Usually both, with a clear split of responsibilities.

Deep-dive probes:

• "Version 3.0 needs to change the event envelope." Look for: support both envelope versions in consumers first, publish v2 behind a flag, a migration window with telemetry showing which producers are on v1, and removing v1 only when it reaches zero. Never a coordinated big-bang release across 40 teams.
• "The SDK's HTTP client added a default 10 s timeout, and a team's 30 s report endpoint broke." Look for: the change was technically correct but behaviour-changing, so it belonged in a major release or behind an opt-in with a deprecation warning; per-call overrides; and release notes plus consumer canaries that would have caught it.
• "How do you know whether the SDK is helping?" Look for: incident counts by category before and after (missing timeouts, PII leaks), time to bootstrap a new service, adoption by version, and support ticket volume. Outcome metrics, not just download counts.
• "A team says the SDK makes their async service slower." Look for: profiling (e.g. a sync redaction filter on every log line, blocking secret fetches), async-native code paths, benchmarks in CI, and treating SDK overhead as a budget with a number.

Rubric: Strong: Keeps the scope to cross-cutting concerns with safe defaults, designs a small typed public API with no import-time side effects, has a concrete versioning and deprecation policy, wide dependency ranges tested in CI, consumer canaries, and an adoption plan driven by risk and templates. Talks about ownership and support, not just code. Acceptable: A reasonable set of modules and semantic versioning, but thin on deprecation mechanics, dependency ranges or how to roll out breaking changes across 40 teams. Red flag: A large framework that owns business logic, exact pins on common dependencies, breaking changes shipped in minor versions, or 'teams will just upgrade when we tell them'.

ST-E-D03 · Build or buy an ML platform for eight data-science teams

Easy · Design · Staff / Lead (8–12 years) · ML platform, build vs buy, model registry, feature store, model serving, cost, governance · 40 min

Scenario:

An insurance company in Mumbai has 8 data-science teams (about 45 people) building Python models: claim-fraud scoring, motor-claim photo triage, renewal propensity, call-centre routing. Today each team trains on its own VMs or notebooks, tracks experiments in spreadsheets, hands a pickle file to a backend team, and that team wraps it in a Flask app. Releasing a new model version takes 3–6 weeks. Last quarter a fraud model ran for a month with a feature computed differently in training and in production, and nobody could say which training data a given production model came from, which the IRDAI audit asked about.

The CTO has a ₹2.5 crore annual budget and asks you: should we buy a managed ML platform from our cloud provider, adopt open-source components, or build our own? Design the target platform and make the recommendation.

Clarifying questions:

• How many models are in production, how often are they retrained, and what serving patterns do we need? (About 25 models; most are batch-scored nightly, 6 are online with p99 < 100 ms; retraining monthly.)
• What's the cloud situation? (Single cloud, with data in a warehouse and object storage; some data must stay in India-region by regulation.)
• What skills exist? Is there a platform/MLOps team? (Two MLOps engineers, a strong data-engineering team, Kubernetes already run by the infra team.)
• What are the audit requirements? (Lineage from production prediction back to model version, code version and training data snapshot; approval records for model changes; explainability for underwriting decisions.)
• {'Which pain hurts most': 'time to production, training/serving skew, or cost of GPUs for the photo model?'}
• Are there constraints on sending data to SaaS vendors outside the cloud account?

Reference design:

Frame the decision around capabilities, not products. The platform needs, in order of value for this company:

1. Reproducible training and lineage: experiment tracking and a model registry recording, for every version, the code commit, the training data snapshot (a warehouse snapshot or a versioned dataset path), parameters, metrics and the approval. This alone answers the audit question.
2. One definition of features for training and serving, to remove the skew that caused the fraud incident: features defined once in Python/SQL, computed in batch for training and served from a low-latency store for the 6 online models (a feature store, or at minimum a shared feature library with tests that compare batch and online values).
3. Standard deployment paths: batch scoring as an orchestrated job (Airflow already exists) and online serving as a standard containerised FastAPI or model-server image with health checks, autoscaling, request/response logging and shadow/canary deployment.
4. Monitoring: input drift, prediction distribution, latency, and business outcome joins (fraud confirmed later), with alerts owned by the model team.
5. Governance: approvals tied to registry stages, model cards, access control.

Recommendation: assemble, don't build. Buy or adopt commodity pieces and build only the thin layer that is specific to the company: - Managed or open-source for commodity capabilities. Experiment tracking and registry (MLflow, self-hosted or managed by the cloud provider), training compute (the cloud's managed training jobs or Kubernetes jobs, with GPU instances only for the photo model), and model serving (a managed endpoint or KServe/BentoML on the existing Kubernetes). - Build the "paved road": a Python package and project template (insure-ml) that wraps those components: train() logs to the registry with the data snapshot automatically, a feature library with shared definitions, a standard serving image, CI that runs training smoke tests and a skew check, and a promotion workflow (staging → shadow → production) with approval records. - Don't build a custom orchestrator, a custom model server or a home-grown feature store from scratch with two MLOps engineers. The scarce resource is people, not licence money.

Phasing. Quarter 1: registry, lineage, the template and batch scoring for two pilot teams (fraud, renewal). Quarter 2: online serving path and the feature library with skew tests; migrate the 6 online models. Quarter 3: monitoring and governance workflows; migrate the rest. Measure: time from "model approved" to "in production" (target under 1 week), number of models with full lineage (target 100%), incidents from skew.

Cost check. Managed services charge for compute plus a premium; at ~25 models with mostly batch scoring, the premium is small compared with the cost of two or three more engineers (₹40–60 lakh/year each fully loaded) needed to run everything self-hosted. Revisit when the scale changes (hundreds of models, heavy GPU use), where self-hosting on Kubernetes can save real money.

Trade-offs:

• Full managed platform (one vendor) vs open-source components vs build. One vendor gives the fastest start and one support contract, but lock-in, opinionated workflows that may not fit, and costs that grow with usage. Open-source components avoid lock-in but need people to run and upgrade them. Building gives the best fit and the highest cost; justified only for the parts that differentiate the business.
• Feature store now vs later. A full feature store is heavy for 6 online models; a shared feature library with batch/online parity tests may solve the skew problem at a fraction of the cost. Buy or adopt a store when online models and shared features grow.
• Standardisation vs team freedom. A paved road with escape hatches: teams may deviate but then own the lineage and monitoring obligations themselves.
• Notebook-first vs code-first. Data scientists are productive in notebooks; production training must be code in a repo with tests. Support notebooks for exploration and make "notebook to pipeline" a templated step.

Deep-dive probes:

• "The vendor's managed platform demo looked great. Why not just buy it all?" Look for: an evaluation against the five capabilities with a pilot on a real model, data-residency checks, exit costs (can models and metadata be exported?), pricing at the projected scale, and how it integrates with the existing warehouse, Airflow and Kubernetes. Buying is a valid answer if it survives that evaluation.
• "How do you guarantee the fraud incident (training/serving skew) can't recur?" Look for: single feature definitions used in both paths, a parity test that computes features for sampled entities through batch and online code and compares, logging of served features so training can use exactly what production saw, and alerting on distribution differences.
• "The auditor picks a claim decision from March and asks which model and data produced it." Look for: prediction logs carrying model version and request ID; registry entries linking version to code commit, data snapshot and approval; retention aligned to the regulation.
• "Teams don't want to migrate; their notebooks work." Look for: start with teams that feel the pain, make the template faster than the old way, migrate models when they next retrain rather than as a separate project, and make the audit requirement the forcing function for production models only.

Rubric: Strong: Starts from capabilities and the company's actual pains (lineage, skew, time to production), recommends buying or adopting commodity parts and building a thin paved road, phases the rollout with pilots and outcome metrics, and reasons about cost in terms of people as well as licences. Handles regulation (lineage, approvals, residency) concretely. Acceptable: Lists sensible components (registry, feature store, serving, monitoring) and makes a plausible build/buy call, but without tying it to the specific pains, phasing, or team capacity. Red flag: Proposes building a custom platform end to end with two engineers, or buys a platform without evaluating fit, lock-in and data residency; ignores the audit and skew problems that triggered the work.

ST-E-D04 · Drawing boundaries in a Django monolith shared by 120 engineers

Easy · Design · Staff / Lead (8–12 years) · architecture, modular monolith, service boundaries, Django, ownership, Conway's law, import contracts · 40 min

Scenario:

A grocery-delivery company in Bengaluru runs its core backend as one Django monolith (about 900k lines, 160 Django apps, one PostgreSQL database with 1,400 tables). It has grown from 15 to 120 engineers in 12 teams (catalogue, pricing, cart & checkout, payments, orders, delivery, partner apps, promotions, support...). Symptoms:

• The CI suite takes 75 minutes; deploys happen twice a day and one team's bad migration blocks everyone.
• Any change to orders/models.py breaks something in promotions or support; nobody knows who owns what.
• Teams want to "move to microservices"; one team has already split out notifications and now runs a service with its own outages and no on-call.

As the Staff engineer for the platform, design the target architecture for the next 18 months and the path to get there.

Clarifying questions:

• {'What hurts most, measured': 'deploy frequency, lead time, incidents caused by cross-team changes, CI time? (CI time and cross-team breakage.)'}
• Do we have different scaling or availability needs per domain? (Search/catalogue reads are 20× everything else; payments needs stricter change control.)
• How tangled is the code? Do we have an import graph between apps and a map of which apps read or write which tables? (No; we'd need to build it.)
• {"What's the operational maturity": 'standard service templates, observability, on-call per team, a deployment platform? (Kubernetes exists; observability is partial; on-call is a single rota.)'}
• Are there hard constraints, such as a PCI scope for payments, that favour isolating a domain?
• What is the business roadmap? (Quick-commerce in 10 new cities; a marketplace for third-party sellers next year.)

Reference design:

Target: a modular monolith first, with a few services extracted where there is a clear reason. The problems listed (ownership, cross-team breakage, slow CI, coupled deploys) are mostly about boundaries and ownership, which microservices don't create by themselves. Splitting a tangled monolith into services produces a distributed monolith: the same coupling, plus network failures.

1. Make the current structure visible. Generate the import graph between Django apps and the table-access map (which app reads/writes which table, from code analysis and DB query logs). Cluster into domains that match teams: catalogue, pricing, cart/checkout, payments, orders, fulfilment, promotions, partner, support. Record owners in a CODEOWNERS-style file and the service catalogue.

2. Define module boundaries and enforce them. - Each domain is a top-level package (domains/orders/...) with a public API module (domains/orders/api.py: functions and pydantic/dataclass DTOs) and everything else private. - Import contracts checked in CI (e.g. import-linter): other domains may import only orders.api, not orders.models; no cycles between domains. Start with the current violations recorded as an allowlist and burn it down per team each quarter. - Data ownership: each table has one owning domain; others read through the owner's API or a read model, never by joining the owner's tables. Add a check that flags cross-domain ORM access. - Cross-domain side effects go through domain events published after commit (outbox table), so promotions reacts to OrderPlaced instead of being called inline.

3. Fix the engineering-system pain directly. - Test selection: run tests for changed domains and their declared dependents, with the full suite nightly; this becomes possible once boundaries exist. Target CI under 15 minutes. - Migrations: a migration policy (expand/contract, no locking migrations on hot tables, review by the owning team), and decoupling migration rollout from code deploys. - Per-domain on-call and dashboards, so ownership is real.

4. Extract services only where it pays. Criteria: a distinct scaling profile (catalogue/search reads), a compliance boundary (payments for PCI scope), a different release cadence, or a team that is blocked by the monolith's release process, and a boundary that is already clean in the modular monolith. Extract by putting the module's public API behind a network interface (same contract), migrating its tables to a separate database, and keeping the events. The strangler pattern: route traffic gradually, keep a rollback.

5. Fix the notifications service by giving it an owner, an on-call rota and the platform's service template, or folding it back in. Extraction without operational ownership is the anti-pattern to stop.

18-month shape: one deployable monolith with 9 enforced domains and CI under 15 minutes; catalogue/search and payments as services; a platform template for any future extraction; decision records (ADRs) for each boundary.

Trade-offs:

• Modular monolith vs microservices. The monolith keeps transactions, refactoring and debugging simple and avoids network failure modes, but scales only as one unit and shares one database. Microservices give independent deploys and scaling at the cost of distributed data, eventual consistency, more infrastructure and more on-call. Microservices pay off once boundaries are stable and teams can operate services.
• Shared database vs database per domain. One database keeps joins and transactions; it couples schema changes and capacity. Logical ownership inside one database is a cheap intermediate step; physical separation follows for extracted services.
• Synchronous API calls vs events between domains. Sync calls are simple and consistent but couple availability; events decouple but require idempotent consumers and handling eventual consistency in the UX.
• Enforcement vs autonomy. CI-enforced import contracts slow down quick fixes; without enforcement, boundaries erode within months.

Deep-dive probes:

• "Checkout needs order, pricing, promotions and inventory in one transaction today. How does that survive boundaries?" Look for: keep it in one database transaction while in the monolith, but behind the domains' APIs; identify which parts truly need atomicity (stock reservation, payment intent) and which can be eventual (loyalty points, notifications via events); a saga only for extracted services.
• "Promotions reads orders_order directly in 140 places." Look for: provide orders.api query functions or a read model, codemod the call sites, record remaining violations in the allowlist with an owner and a deadline, and measure the count going down.
• "A team insists on extracting their domain now." Look for: apply the extraction criteria; check that the boundary is clean (no cross-domain table access, a stable API), that they have on-call and observability, and what problem extraction solves that the modular monolith doesn't. Agree on a written decision either way.
• "How will you know the architecture is working?" Look for: DORA-style metrics per team (deploy frequency, lead time, change-failure rate), CI time, count of boundary violations, and incidents caused by cross-domain changes, reviewed quarterly.

Rubric: Strong: Diagnoses the problems as boundary and ownership issues, proposes a modular monolith with enforced import contracts and data ownership, fixes CI and migrations directly, and extracts services only by explicit criteria with operational readiness. Uses data (import graph, table access) and measurable outcomes. Acceptable: Recognises that a big-bang microservices rewrite is risky and proposes modularisation, but is vague on enforcement, data ownership or how to decide which services to extract. Red flag: Splits into 12 microservices (one per team) with a shared database, or proposes a rewrite; ignores ownership, on-call and data coupling.

ST-ET-O01 · A platform SDK calls basicConfig at import and the service's logging config vanishes

Easy & Tricky · Output · Staff / Lead (8–12 years) · logging, SDK design, platform standards, import side effects, observability · 5 min

Question: The payments platform team ships payments_sdk to 40 product teams. Version 4.2 added a logging.basicConfig(...) call at import time "so warnings are visible". The orders team's main.py imports the SDK first and then configures logging for its own service. Both pieces are shown inline below, in import order. What does this print?

import logging
import sys

# --- inside payments_sdk/__init__.py (a platform SDK, version 4.2) ---
logging.basicConfig(level=logging.WARNING, stream=sys.stdout,
                    format="%(levelname)s:%(name)s:%(message)s")
sdk_log = logging.getLogger("payments_sdk")
sdk_log.warning("using default region ap-south-1")

# --- inside the team's service, main.py ---
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout,
                    format="%(asctime)s %(levelname)s [orders] %(message)s")
log = logging.getLogger("orders")
log.debug("cart has 3 items")
log.info("order ORD-981 placed for ₹1,499")
log.warning("inventory low for SKU-17")
print("root handlers:", len(logging.getLogger().handlers))

Expected output:

WARNING:payments_sdk:using default region ap-south-1
WARNING:orders:inventory low for SKU-17
root handlers: 1

The trap: Candidates expect the service's configuration to win because it runs last: three orders lines (debug, info, warning) in the [orders] format with timestamps, and possibly 2 root handlers. It looks like ordinary "last assignment wins" configuration.

Explanation:

logging.basicConfig does nothing if the root logger already has handlers (unless force=True is passed, available since 3.8). The SDK's import-time call installed a stream handler with the SDK's format and set the root level to WARNING. The service's call, which runs later, is silently a no-op: no error, no warning.

So the root logger keeps level WARNING and the SDK's format. The orders logger has no level of its own, so its effective level is the root's WARNING: debug and info are dropped, and the warning is printed as WARNING:orders:.... There is still exactly one root handler.

In production this looks like "our service's logs disappeared after a dependency bump": INFO-level audit lines (orders placed, payments captured) stop reaching the log pipeline, JSON formatting the log shipper relies on is replaced by plain text, and nobody's code changed in the affected repo. With 40 consuming teams, one line in a shared SDK becomes an org-wide observability incident.

The standard for libraries: never configure logging. Use logging.getLogger(__name__), optionally add a logging.NullHandler() to the library's top-level logger, and leave handlers, levels and formats to the application's entry point. More generally, a platform SDK must not have import-time side effects on process-global state (logging, warnings filters, signal handlers, sys.path, the default event loop policy, locale).

Takeaway: basicConfig only configures an unconfigured root logger, so whoever calls it first wins; libraries must not call it.

Follow-ups:

1. "As the SDK's owning team lead, how do you roll this back across 40 consumers, and how do you stop it recurring?" Ship 4.2.1 removing the call (with a changelog entry naming the symptom) and pin-bump through the dependency bot; tell teams they can call basicConfig(force=True) meanwhile. To stop recurrence: a lint/CI rule in the SDK repo banning basicConfig, handler creation and other global mutation at module level, plus an import-side-effect test that compares logging/warnings state before and after import payments_sdk.
2. "Would force=True in main.py be the right fix?" It works as a stopgap (it removes and closes existing root handlers, then configures), but it treats the symptom in every consumer. It also depends on import order: if the SDK configured logging lazily on first use, after main had configured it, force=True in main wouldn't help. The fix belongs in the library.
3. "The SDK team says users never saw its warnings without that line. What do you tell them?" With no handlers configured, Python's last-resort handler already prints WARNING and above to stderr, so warnings are visible in unconfigured scripts. In configured services, visibility is the application's choice. If a message is important enough to force, it should probably be an exception or a warnings.warn (DeprecationWarning/UserWarning), not a log line.

Scoring: Strong: Exact output, explains that basicConfig is a no-op once root has handlers and that the root level is WARNING, then frames it as a library-standards problem (no import-time global side effects) with a concrete rollout and a CI guard. Acceptable: Predicts the configured service wins, but when asked to look again at basicConfig's documented behaviour, works out the output and suggests force=True or removing the SDK's call. Red flag: Says the second call 'adds a second handler' or 'overrides the first', or proposes that every team call force=True as the standard.

ST-ET-O02 · A naive UTC watermark becomes an epoch 5.5 hours early on IST hosts

Easy & Tricky · Output · Staff / Lead (8–12 years) · datetime, timezones, epoch time, Python upgrades, data platform, cross-service contracts · 5 min

Question: An export job stores its high-water mark as datetime.utcnow() returns it and sends it to a consumer service as epoch seconds. The batch hosts are configured in IST; the consumer runs in UTC. To keep the output fixed, the snippet uses midnight UTC on 1 April 2026 instead of calling utcnow(). What does this print?

import os
import time
from datetime import datetime, timezone

os.environ["TZ"] = "Asia/Kolkata"       # the batch hosts are configured in IST
time.tzset()

# The export job stores its watermark as datetime.utcnow() returned it: a naive UTC value.
watermark = datetime(2026, 4, 1, 0, 0)  # midnight UTC

epoch = int(watermark.timestamp())      # sent to the consumer service as epoch seconds
print(epoch)
print(datetime.fromtimestamp(epoch))                     # read back on the same host
print(datetime.fromtimestamp(epoch, tz=timezone.utc))    # read by a consumer that thinks in UTC
print(int(watermark.replace(tzinfo=timezone.utc).timestamp()))

Expected output:

1774981800
2026-04-01 00:00:00
2026-03-31 18:30:00+00:00
1775001600

The trap: "1775001600, then midnight twice, then 1775001600 again." The value came from utcnow(), so it feels like a UTC timestamp, and the round trip on the same host (line 2) looks perfect, which seems to confirm it.

Explanation:

A naive datetime has no timezone, and .timestamp() interprets a naive value as local time. On a host whose zone is IST (UTC+5:30), "2026-04-01 00:00" is read as midnight IST, which is 2026-03-31 18:30 UTC, epoch 1774981800. That is 19,800 seconds (5.5 hours) earlier than the intended 1775001600.

Line 2 is the dangerous one: fromtimestamp without a zone converts back to the same local time, so a round trip on the same host looks correct and unit tests on that host pass. The consumer, which thinks in UTC, sees 18:30 on 31 March and re-exports (or skips, depending on the direction) 5.5 hours of data every run. If dev laptops run in IST and CI runs in UTC, the same code behaves differently in each place.

This is a large part of why datetime.utcnow() and utcfromtimestamp() are deprecated in Python 3.12: they return naive values that look like UTC but are treated as local time by everything else in the module. The replacement is datetime.now(timezone.utc), which is timezone-aware, so .timestamp() is correct on any host. A fleet-wide 3.12 upgrade is a good moment to make "aware datetimes only, UTC at boundaries" a lint-enforced standard (ruff's DTZ rules flag naive calls).

Takeaway: naive datetimes mean "local time" to .timestamp(); anything that crosses a process or service boundary must be timezone-aware.

Follow-ups:

1. "You are planning the 3.12 upgrade across 300 repos and utcnow() appears 1,800 times. How do you approach it?" Don't search-and-replace blindly: datetime.now(timezone.utc) returns an aware value, ordering comparisons between aware and naive values raise TypeError (and == is silently False), and naive values stored in DB columns or pickles will start mixing with aware ones. Codemod the call sites per repo, fix comparisons and serialisation together, add the DTZ lint rule to the shared config so new naive calls fail CI, and turn the DeprecationWarning into an error in test runs once a repo is clean.
2. "Why might this bug never show up in a Kubernetes deployment, then appear on one VM-based job?" Container images usually run with TZ unset, which means UTC, so local time equals UTC and the bug is invisible. A legacy VM or a developer laptop configured in Asia/Kolkata exposes it. The code is wrong everywhere; only the environment hides it, which is why environment-dependent behaviour belongs in the standards, not in tribal knowledge.
3. "What should the contract between the export job and the consumer say about time?" Timestamps cross the boundary as UTC: ISO 8601 with an explicit offset (2026-04-01T00:00:00+00:00) or integer epoch with documented units (seconds vs milliseconds). Business-local concepts such as 'the IST business day' are computed explicitly with zoneinfo.ZoneInfo('Asia/Kolkata') where needed, never implied by the host's zone.

Scoring: Strong: Exact output, explains naive-as-local in .timestamp(), points out the same-host round trip that hides the bug, links it to the 3.12 deprecation, and proposes an enforceable org standard (aware UTC at boundaries, lint rule, migration plan). Acceptable: Expects the UTC epoch at first, but once reminded that the datetime is naive, works out the 5.5-hour shift and the fix with datetime.now(timezone.utc). Red flag: Insists utcnow() values are UTC 'because of the name', or proposes fixing it by adding timedelta(hours=5, minutes=30) in the consumer.

ST-ET-C01 · Composite availability of a request path, and the SLO you can actually promise

Easy & Tricky · Coding · Staff / Lead (8–12 years) · availability, SLOs, dependencies, redundancy, recursion, architecture review · 20 min

Problem:

A business head wants to promise merchants a 99.95% monthly availability SLA for checkout. You are reviewing the architecture, and you want a small tool that computes the best availability the current dependency graph can deliver.

A request path is described by a nested structure:

• a float in [0, 1]: one component's availability, e.g. 0.999;
• ("serial", [node, ...]): the request needs every child to be up (a call chain);
• ("parallel", [node, ...]): redundant replicas; the request succeeds if any child is up.

Assume failures are independent. Implement availability(node) -> float, and downtime_minutes(a, days=30) -> float (expected minutes of downtime in a month of days days at availability a).

Raise ValueError for an availability outside [0, 1], an unknown node kind, or an empty child list.

Signature: Node = float | tuple[str, list["Node"]] def availability(node: Node) -> float: ... def downtime_minutes(a: float, days: int = 30) -> float: ...

Constraints: Standard library only. Nesting depth is small (under 10), so recursion is fine.

Examples:

• availability(('serial', [0.999, 0.999, 0.999])) → 0.997002999 (not 0.999)
• availability(('parallel', [0.99, 0.99])) → 0.9999
• availability(('serial', [0.9999, ('parallel', [0.995] * 3), 0.999, 0.9995, 0.999])) → ≈ 0.997402, about 112 minutes of downtime a month; a 99.95% SLA allows about 21.6

Hints:

1. For a call chain, what is the probability that all of the components are up at the same moment?
2. For replicas, it's easier to compute the probability that all of them are down.
3. Recurse on the node kind; leaves are floats.

Reference solution:

import math

Node = float | tuple[str, list["Node"]]


def availability(node: Node) -> float:
    if isinstance(node, (int, float)) and not isinstance(node, bool):
        if not 0.0 <= node <= 1.0:
            raise ValueError(f"availability out of range: {node}")
        return float(node)
    kind, children = node
    if not children:
        raise ValueError(f"{kind!r} node has no children")
    values = [availability(child) for child in children]
    if kind == "serial":
        return math.prod(values)                       # every hop must be up
    if kind == "parallel":
        return 1.0 - math.prod(1.0 - v for v in values)  # down only if all replicas are down
    raise ValueError(f"unknown node kind: {kind!r}")


def downtime_minutes(a: float, days: int = 30) -> float:
    if not 0.0 <= a <= 1.0:
        raise ValueError(f"availability out of range: {a}")
    return (1.0 - a) * days * 24 * 60

Complexity: O(n) over the n nodes of the tree, O(depth) recursion stack.

Tests:

from math import isclose

assert isclose(availability(0.999), 0.999)
assert isclose(availability(("serial", [0.999, 0.999, 0.999])), 0.997002999)
assert isclose(availability(("parallel", [0.99, 0.99])), 0.9999)
assert isclose(availability(("parallel", [0.999, 0.999])), 0.999999)

checkout = ("serial", [
    0.9999,                           # CDN / load balancer
    ("parallel", [0.995] * 3),        # three API pods
    0.999,                            # auth service
    0.9995,                           # Postgres primary
    0.999,                            # payment gateway
])
a = availability(checkout)
assert isclose(a, 0.997402124624769, rel_tol=1e-12), a
assert a < 0.9995                     # the 99.95% SLA is not achievable on this path
assert isclose(downtime_minutes(a), 112.228216, rel_tol=1e-6)
assert isclose(downtime_minutes(0.9995), 21.6)

# nested: two regions, each a chain of LB + app, behind a global LB
region = ("serial", [0.999, 0.995])
assert isclose(availability(("serial", [0.9999, ("parallel", [region, region])])), 0.9998640635690026)

# a serial chain is never better than its weakest link, and usually much worse
chain = ("serial", [0.9999] * 20)
assert availability(chain) < 0.9999
assert isclose(availability(chain), 0.9999 ** 20)

for bad in [1.2, -0.1, ("serial", []), ("mesh", [0.99]), ("parallel", [0.99, 2.0])]:
    try:
        availability(bad)
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted {bad}")

A common wrong solution (the tests catch it):

Node = float | tuple[str, list["Node"]]


def availability(node: Node) -> float:
    if isinstance(node, (int, float)) and not isinstance(node, bool):
        if not 0.0 <= node <= 1.0:
            raise ValueError(f"availability out of range: {node}")
        return float(node)
    kind, children = node
    if not children:
        raise ValueError(f"{kind!r} node has no children")
    values = [availability(child) for child in children]
    if kind == "serial":
        return min(values)                 # "a chain is as strong as its weakest link"
    if kind == "parallel":
        return max(values)                 # "as good as the best replica"
    raise ValueError(f"unknown node kind: {kind!r}")


def downtime_minutes(a: float, days: int = 30) -> float:
    if not 0.0 <= a <= 1.0:
        raise ValueError(f"availability out of range: {a}")
    return (1.0 - a) * days * 24 * 60

Alternatives and tradeoffs:

• Weakest link (min) is the obvious answer and is wrong: it says the checkout path is 99.9% available, when independent failures of five components add up to about 99.74%: 112 minutes of downtime a month instead of 43, about 2.6 times as much. Using max for replicas similarly ignores that redundancy multiplies unavailability, which is why two 99% replicas give 99.99%.
• Independence is optimistic. Replicas in one zone, one shared database, one config push or one bad deploy fail together. Real reviews model shared fate explicitly (a zone or the deploy pipeline is itself a serial component of the replica set) and check against incident history.
• Soft dependencies don't belong in the serial chain at all if the code degrades gracefully (recommendations, reviews). Turning hard dependencies into soft ones, with timeouts and fallbacks, is usually the cheapest availability improvement.

Follow-ups:

1. "The business still wants to promise 99.95%. What are your options as the architect?" Raise availability of the weakest hard dependencies (multi-zone Postgres with automatic failover, a second payment gateway with routing), remove hard dependencies (cache auth tokens so auth is soft for a few minutes, queue non-critical writes), or narrow the SLA's definition (exclude scheduled maintenance, measure on successful-payment flows only). Or promise less: the honest answer may be 99.9% with credits, and that is a business decision you frame with the downtime numbers.
2. "Where do you get the component numbers from, and how much do you trust a vendor's 99.99%?" Use measured availability from your own SLIs (synthetic checks and request success rates over the last 6–12 months), not the vendor's marketing SLA, which describes credits, not reality, and is measured at their edge, not on your path. Your network path to them, DNS and TLS also belong in the chain.
3. "What does this model ignore that matters for an SLA measured per month?" Time to detect and recover (MTTR): availability is MTBF/(MTBF+MTTR), and one 3-hour outage blows a monthly 99.95% budget even if the yearly average is fine. It also ignores partial failures (slow but up), correlated failures and planned maintenance windows.

Scoring: Strong: Products for serial and complement-products for parallel, validation, and immediately states the business consequence (the 99.95% SLA is not achievable on this path). Raises shared fate and soft vs hard dependencies without prompting and offers concrete architectural options. Acceptable: Gets the formulas right, perhaps after a first min instinct, and explains the downtime numbers; needs prompting on correlated failures. Red flag: Defends min for chains or max for replicas, or tells the business head 99.95% is fine because every component is at least 99.9%.

ST-ET-C02 · Fleet-wide p99 from per-host latency histograms

Easy & Tricky · Coding · Staff / Lead (8–12 years) · observability, percentiles, histograms, SLOs, aggregation, metrics design · 20 min

Problem:

Each of the hosts behind the order API exports a latency histogram: a shared list of bucket upper bounds in milliseconds, and per host the number of requests that fell into each bucket. The SLO dashboard needs the fleet-wide p99 (and other percentiles) across all hosts.

Implement fleet_percentile(bounds, host_counts, pct):

• bounds is strictly increasing; the last bound may be math.inf. counts[i] is the number of requests with latency in (bounds[i-1], bounds[i]] (not cumulative).
• Return the smallest bound b such that at least pct percent of all requests, across all hosts, took ≤ b ms. Formally, with total requests, the target rank is ceil(pct × total / 100), and you return the bound of the first bucket where the running count reaches that rank. Do not interpolate inside buckets.
• 0 < pct <= 100. Raise ValueError for a pct outside that range, a host whose counts list has the wrong length, a negative count, or no requests at all.

Signature: def fleet_percentile(bounds: list[float], host_counts: dict[str, list[int]], pct: float) -> float:

Constraints: Standard library only. Up to a few hundred hosts and a few dozen buckets per evaluation.

Examples:

• bounds = [50, 100, 200, 500, 1000, 2000, math.inf] host_counts = {"api-1": [8000, 900, 90, 10, 0, 0, 0], # busy, fast "api-2": [100, 300, 300, 200, 80, 20, 0]} # quiet, slow fleet_percentile(bounds, host_counts, 99) → 500 (api-1's p99 is 200 and api-2's is 2000; neither their average, 1100, nor their max is the fleet p99)

Hints:

1. A percentile of a combined population is not a function of the per-host percentiles. What can you add across hosts?
2. Histograms with the same buckets merge by adding counts bucket by bucket.
3. Compute the rank with exact arithmetic: 99.9 / 100 * total in floats can land a hair above an integer.

Reference solution:

from decimal import Decimal, ROUND_CEILING


def fleet_percentile(bounds: list[float], host_counts: dict[str, list[int]], pct: float) -> float:
    if not 0 < pct <= 100:
        raise ValueError("pct must be in (0, 100]")
    merged = [0] * len(bounds)
    for host, counts in host_counts.items():
        if len(counts) != len(bounds):
            raise ValueError(f"{host}: expected {len(bounds)} buckets, got {len(counts)}")
        for i, c in enumerate(counts):
            if c < 0:
                raise ValueError(f"{host}: negative count")
            merged[i] += c                       # histograms merge by adding counts
    total = sum(merged)
    if total == 0:
        raise ValueError("no requests recorded")

    rank = int((Decimal(str(pct)) * total / 100).to_integral_value(rounding=ROUND_CEILING))
    running = 0
    for bound, c in zip(bounds, merged):
        running += c
        if running >= rank:
            return bound
    raise AssertionError("unreachable: running count reaches total")

Complexity: O(h × b) to merge h hosts with b buckets, O(b) to scan; O(b) extra space. Crucially, the per-host data you need to ship and store is O(b) regardless of request volume, which is why histograms (and sketches) are the standard for fleet percentiles.

Tests:

import math

bounds = [50, 100, 200, 500, 1000, 2000, math.inf]
hosts = {
    "api-1": [8000, 900, 90, 10, 0, 0, 0],
    "api-2": [100, 300, 300, 200, 80, 20, 0],
}
assert fleet_percentile(bounds, hosts, 99) == 500
assert fleet_percentile(bounds, hosts, 99.9) == 2000
assert fleet_percentile(bounds, hosts, 50) == 50
assert fleet_percentile(bounds, hosts, 100) == 2000
assert fleet_percentile(bounds, hosts, 81) == 50          # rank 8100 exactly: still the first bucket
# one host only: the fleet percentile is that host's percentile
assert fleet_percentile(bounds, {"api-1": hosts["api-1"]}, 99) == 200
assert fleet_percentile(bounds, {"api-2": hosts["api-2"]}, 99) == 2000
# hosts with no traffic don't distort anything
assert fleet_percentile(bounds, {**hosts, "api-3": [0] * 7}, 99) == 500
# three equal hosts, one of them slow
same = {"a": [99, 1, 0, 0, 0, 0, 0], "b": [99, 1, 0, 0, 0, 0, 0], "c": [0, 0, 0, 0, 0, 0, 100]}
assert fleet_percentile(bounds, same, 50) == 50
assert fleet_percentile(bounds, same, 66) == 50           # rank 198 of 300
assert fleet_percentile(bounds, same, 66.5) == 100        # rank 200
assert fleet_percentile(bounds, same, 67) == math.inf     # beyond the largest finite bound

for args in [(bounds, hosts, 0), (bounds, hosts, 101), (bounds, {"x": [1, 2]}, 99),
             (bounds, {"x": [0] * 7}, 99), (bounds, {"x": [-1, 2, 0, 0, 0, 0, 0]}, 99)]:
    try:
        fleet_percentile(*args)
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted {args[1]}, {args[2]}")

A common wrong solution (the tests catch it):

from decimal import Decimal, ROUND_CEILING


def _host_percentile(bounds, counts, pct):
    total = sum(counts)
    rank = int((Decimal(str(pct)) * total / 100).to_integral_value(rounding=ROUND_CEILING))
    running = 0
    for bound, c in zip(bounds, counts):
        running += c
        if running >= rank:
            return bound


def fleet_percentile(bounds: list[float], host_counts: dict[str, list[int]], pct: float) -> float:
    if not 0 < pct <= 100:
        raise ValueError("pct must be in (0, 100]")
    per_host = []
    for host, counts in host_counts.items():
        if len(counts) != len(bounds):
            raise ValueError(f"{host}: expected {len(bounds)} buckets, got {len(counts)}")
        if any(c < 0 for c in counts):
            raise ValueError(f"{host}: negative count")
        if sum(counts):
            per_host.append(_host_percentile(bounds, counts, pct))
    if not per_host:
        raise ValueError("no requests recorded")
    return sum(per_host) / len(per_host)      # "fleet p99 = average of host p99s"

Alternatives and tradeoffs:

• Averaging per-host p99s (the broken version) gives 1100 ms here: it weights a quiet host's tail the same as a busy host's, and the result is not a percentile of anything. Taking the max (2000) is equally wrong in the other direction; a traffic-weighted average (380) is still wrong. Percentiles do not compose; counts do.
• Shipping raw latencies to a central store gives exact percentiles but costs storage and network proportional to traffic; fine for sampled traces, not for every request on every host.
• Mergeable sketches (t-digest, DDSketch, HDR histograms, Prometheus native histograms) give bounded relative error with adaptive buckets; the same "merge, then query" rule applies.
• Interpolating inside the bucket (what Prometheus' histogram_quantile does) gives a smoother number but can report values that never occurred; returning the bucket bound is conservative and honest about resolution.

Follow-ups:

1. "Teams currently export a p99_ms gauge per pod and the dashboard averages them. What standard do you set, and how do you migrate?" Standard: services export latency as histograms (or a mergeable sketch) with org-wide default buckets aligned to SLO thresholds, and all percentile panels and alerts compute from merged buckets. Migrate by adding the histogram in the shared HTTP middleware of the platform SDK (one change, every service gets it), running both for a few weeks, then removing the gauge from dashboards and alerts. Communicate why the numbers change: the 'new' p99 is not a regression.
2. "How should bucket bounds be chosen?" Put bounds at the SLO thresholds you alert on (e.g. 300 ms if the SLO is '99% under 300 ms'), so 'fraction under threshold' is exact, with roughly exponential spacing elsewhere. Too few buckets lose resolution; too many multiply time-series cardinality across hosts × endpoints × status codes, which is a real cost line in the metrics bill.
3. "Why did the reference solution compute the rank with Decimal(str(pct)) instead of math.ceil(pct / 100 * total)?" 81 / 100 * 10000 is 8100.000000000001 in binary floating point, so ceil gives 8101 and the answer moves to the next bucket. With decimal or integer arithmetic the rank is exactly 8100. A shared library will be called with every possible pct, so boundary correctness matters more than in a one-off script.

Scoring: Strong: Merges counts first and takes the percentile once; explains immediately why per-host percentiles cannot be averaged, gives the numbers for this example, and turns it into an org-wide metrics standard (histograms in the shared middleware, bucket choice tied to SLOs, cardinality cost). Acceptable: Starts from per-host percentiles, but when shown the busy-fast vs quiet-slow example, switches to merging counts and implements it correctly. Red flag: Defends averaging (or max of) per-host p99s for the SLO dashboard, or proposes shipping every raw latency to compute it exactly at full traffic.

ST-ET-C03 · When will the Kafka backlog clear? The hot partition decides

Easy & Tricky · Coding · Staff / Lead (8–12 years) · incident leadership, Kafka, partitions, consumer lag, capacity planning, data skew · 20 min

Problem:

During an incident, the UPI payment-notification consumer fell behind. Consumers are healthy again and the head of support asks you, as incident lead, when notifications will be back to real time. You want a function the on-call dashboard can use instead of mental arithmetic.

The topic has P partitions. The consumer group has at least one consumer per partition, so each partition is processed by its own consumer, sequentially, at up to consumer_rate messages/s. Producers keep writing to each partition at its own rate while you drain. Partitions are keyed by merchant, so traffic is not evenly spread.

Implement:

• drain_eta_seconds(lags, produce_rates, consumer_rate) -> float | None: seconds until every partition's lag is zero. A partition with lag 0 whose produce rate is at most consumer_rate takes 0 s. A partition whose produce rate is at least consumer_rate and has lag (or whose produce rate exceeds consumer_rate) never drains: return None for the whole topic.
• required_consumer_rate(lags, produce_rates, deadline_s) -> float: the per-consumer throughput needed to drain every partition within deadline_s seconds.

lags[i] and produce_rates[i] describe partition i. Raise ValueError for empty or mismatched lists, negative values, a non-positive consumer_rate or deadline_s.

Signature: def drain_eta_seconds(lags: list[int], produce_rates: list[float], consumer_rate: float) -> float | None: ... def required_consumer_rate(lags: list[int], produce_rates: list[float], deadline_s: float) -> float: ...

Constraints: Standard library only. Rates are messages per second; treat them as constant during the drain.

Examples:

• drain_eta_seconds([36000] * 4, [100] * 4, 130) → 1200.0 (20 minutes; with even traffic, aggregate maths happens to agree)
• drain_eta_seconds([60000, 10000, 10000, 10000], [160, 80, 80, 80], 200) → 1500.0 (the hot partition; total lag / total net rate would say 225 s)
• drain_eta_seconds([5000, 0, 0, 0], [250, 50, 50, 50], 200) → None (partition 0 receives more than one consumer can process)

Hints:

1. Can a second consumer help with partition 0's backlog?
2. Each partition drains independently at consumer_rate - produce_rate. Which partition finishes last?
3. For the deadline version, solve lag / (r - produce) <= deadline for r, per partition.

Reference solution:

def _validate(lags: list[int], produce_rates: list[float]) -> None:
    if not lags or len(lags) != len(produce_rates):
        raise ValueError("need one lag and one produce rate per partition")
    if any(x < 0 for x in lags) or any(r < 0 for r in produce_rates):
        raise ValueError("lags and rates must be non-negative")


def drain_eta_seconds(lags: list[int], produce_rates: list[float], consumer_rate: float) -> float | None:
    _validate(lags, produce_rates)
    if consumer_rate <= 0:
        raise ValueError("consumer_rate must be positive")
    worst = 0.0
    for lag, produced in zip(lags, produce_rates):
        net = consumer_rate - produced          # one consumer per partition, processed in order
        if net < 0 or (net == 0 and lag > 0):
            return None                         # this partition's lag never reaches zero
        if lag:
            worst = max(worst, lag / net)
    return worst


def required_consumer_rate(lags: list[int], produce_rates: list[float], deadline_s: float) -> float:
    _validate(lags, produce_rates)
    if deadline_s <= 0:
        raise ValueError("deadline_s must be positive")
    return max(produced + lag / deadline_s for lag, produced in zip(lags, produce_rates))

Complexity: O(P) time and O(1) extra space for P partitions. The numbers come from the consumer group's lag per partition and the topic's per-partition produce rate, both of which the monitoring stack already has.

Tests:

from math import isclose

# even traffic: per-partition and aggregate answers agree
assert isclose(drain_eta_seconds([36000] * 4, [100] * 4, 130), 1200.0)
# skew: the hot partition takes 1500 s; aggregate maths would say 225 s
assert isclose(drain_eta_seconds([60000, 10000, 10000, 10000], [160, 80, 80, 80], 200), 1500.0)
# a partition receiving more than one consumer can handle never drains
assert drain_eta_seconds([5000, 0, 0, 0], [250, 50, 50, 50], 200) is None
# break-even with a backlog never drains; break-even with no backlog is fine
assert drain_eta_seconds([100, 0], [200, 50], 200) is None
assert drain_eta_seconds([0, 0], [200, 50], 200) == 0.0
# nothing to drain
assert drain_eta_seconds([0, 0, 0], [10, 10, 10], 100) == 0.0
# an idle but backlogged partition
assert isclose(drain_eta_seconds([1000, 0], [0, 0], 50), 20.0)

assert isclose(required_consumer_rate([60000, 10000, 10000, 10000], [160, 80, 80, 80], 600), 260.0)
assert isclose(required_consumer_rate([36000] * 4, [100] * 4, 1200), 130.0)
# consistency: at the required rate, the ETA equals the deadline
r = required_consumer_rate([60000, 10000, 10000, 10000], [160, 80, 80, 80], 600)
assert isclose(drain_eta_seconds([60000, 10000, 10000, 10000], [160, 80, 80, 80], r), 600.0)

for args in [([], [], 10), ([1, 2], [1], 10), ([-1], [1], 10), ([1], [-1], 10), ([1], [1], 0)]:
    try:
        drain_eta_seconds(*args)
    except ValueError:
        pass
    else:
        raise AssertionError(f"accepted {args}")
try:
    required_consumer_rate([1], [1], 0)
except ValueError:
    pass
else:
    raise AssertionError("accepted deadline 0")

A common wrong solution (the tests catch it):

def _validate(lags, produce_rates):
    if not lags or len(lags) != len(produce_rates):
        raise ValueError("need one lag and one produce rate per partition")
    if any(x < 0 for x in lags) or any(r < 0 for r in produce_rates):
        raise ValueError("lags and rates must be non-negative")


def drain_eta_seconds(lags, produce_rates, consumer_rate):
    _validate(lags, produce_rates)
    if consumer_rate <= 0:
        raise ValueError("consumer_rate must be positive")
    total_lag = sum(lags)
    net = consumer_rate * len(lags) - sum(produce_rates)    # whole-group throughput
    if total_lag == 0:
        return 0.0
    if net <= 0:
        return None
    return total_lag / net


def required_consumer_rate(lags, produce_rates, deadline_s):
    _validate(lags, produce_rates)
    if deadline_s <= 0:
        raise ValueError("deadline_s must be positive")
    return (sum(produce_rates) + sum(lags) / deadline_s) / len(lags)

Alternatives and tradeoffs:

• Aggregate maths (the broken version): total lag divided by total net throughput. It is correct only when load is even. Under merchant-keyed skew it promised 225 s when the real answer was 1500 s, and "a few seconds" when the hot partition will never drain. In an incident, that is the difference between a credible update and a missed one.
• Adding consumers beyond the partition count does nothing: extra consumers in a group sit idle. Adding partitions spreads new keys but changes the key → partition mapping (breaking per-merchant ordering during the transition) and does not move the existing backlog.
• Speeding up the hot partition's consumer: batch writes, raise max.poll.records, move slow side effects (an SMS vendor call) to a separate queue, or process a partition's messages concurrently per key while preserving per-key order. These change consumer_rate for the partition that matters.
• Shedding: if the notifications are stale (an SMS for a payment 3 hours ago), skip or summarise old messages instead of draining them all. A product decision the incident lead should put on the table.

Follow-ups:

1. "Leadership asks you for an ETA every 30 minutes. What do you report, and how?" The ETA from the slowest partition with its assumption (current produce rates and consumer throughput), the trend (is the hot partition's lag actually falling?), and what you're doing to change it. Give a range, not a single number, and update when the trend breaks. Never give the aggregate number when partitions are skewed.
2. "Why is partition 0 hot in the first place, and what is the long-term fix?" Merchant-keyed partitioning plus one very large merchant (a big e-commerce or utility biller) puts a big share of the traffic on one partition. Options: a compound key (merchant + shard suffix) for high-volume merchants where per-merchant global order isn't required, or ordering only per payment ID; a dedicated topic for the largest merchants; and capacity planning per partition, not per topic. The key choice is an architecture decision with an ordering contract attached, so it needs owners' agreement.
3. "What alerting would have caught this before support did?" Alert on per-partition lag expressed as time (lag / consume rate, or the age of the oldest unconsumed message), not on total message count, with thresholds tied to the notification SLO. Total-lag alerts hide one hot partition among many quiet ones.

Scoring: Strong: Computes the ETA per partition and takes the max, handles never-drains cases, explains why extra consumers don't help and why aggregate maths misleads under skew, and talks about how to communicate the ETA and fix the key design afterwards. Acceptable: Starts with total lag over total rate, then corrects to per-partition maths once asked whether a second consumer can help partition 0. Red flag: Insists the aggregate ETA is right, or proposes 'scale consumers to 50' as the fix for a 4-partition topic.

ST-H-O01 · A pre-fork server copies the SDK's client and its idempotency-key counter into every worker

Hard · Output · Staff / Lead (8–12 years) · fork safety, pre-fork servers, gunicorn preload, idempotency keys, SDK design, module-level state · 6 min

Question: The platform team's payments_sdk keeps one lazily created client per process and generates idempotency keys from a module-level counter (host name plus sequence number). The checkout service runs under Gunicorn with --preload: the app is imported once in the master, which makes a warm-up call, and then the master forks the workers. The snippet below reproduces that with os.fork() (Linux/macOS). What does it print?

import itertools
import os
import sys

# --- payments_sdk/client.py (platform SDK) ---
_seq = itertools.count(1)
_client = None


class Client:
    def __init__(self):
        self.owner_pid = os.getpid()      # stands in for an open TLS connection

    def idempotency_key(self):
        return f"web-01-{next(_seq)}"     # host name + per-process sequence


def get_client():
    global _client
    if _client is None:
        _client = Client()
    return _client


# --- app.py, imported once by the pre-fork master (gunicorn --preload) ---
print("master warm-up key:", get_client().idempotency_key())
sys.stdout.flush()

for worker in (1, 2):
    pid = os.fork()
    if pid == 0:                          # inside a worker process
        c = get_client()
        keys = [c.idempotency_key(), c.idempotency_key()]
        print(f"worker {worker}: own client={c.owner_pid == os.getpid()}, keys={keys}")
        sys.stdout.flush()
        os._exit(0)
    os.waitpid(pid, 0)

print("master next key:", get_client().idempotency_key())

Expected output:

master warm-up key: web-01-1
worker 1: own client=False, keys=['web-01-2', 'web-01-3']
worker 2: own client=False, keys=['web-01-2', 'web-01-3']
master next key: web-01-2

Explanation:

fork() gives the child a copy of the parent's whole address space at that instant, copy-on-write. Nothing in Python runs to "re-initialise" module state, so every worker starts with:

• the same _client object the master created during warm-up (owner_pid is the master's PID, hence own client=False). In production that object holds a socket or TLS session, and now N processes share one file descriptor. Interleaved writes and reads on one TLS connection produce errors such as SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC or responses delivered to the wrong worker, which is the classic symptom of a database or HTTP pool created before the fork.
• a copy of the counter at value 2. Each worker continues from there independently, so both hand out web-01-2, web-01-3, ... The master's own counter never moved either (its next key is also web-01-2), because the children advanced their own copies, not the parent's.

The second effect is the dangerous one for a payments platform. Idempotency keys exist so the provider can collapse retries of the same request. Two workers sending web-01-2 for two different customers' payments make the provider treat the second as a retry of the first: it returns the first payment's response and never charges the second customer, or rejects it with a key-reuse error. Nothing fails in testing, because tests and gunicorn without --preload (or a single worker) never share the counter.

Fixes, from the SDK owner's side:

• Never derive uniqueness from process-local counters. Use uuid.uuid4() (from os.urandom, safe across fork) or include PID plus a random component; better still, let the caller supply a business key such as order_id + attempt.
• Make the client fork-aware: record os.getpid() at creation and rebuild when it differs, or register os.register_at_fork(after_in_child=_reset) to drop the client and reseed any state. The stdlib random module does exactly this (it reseeds in the child via register_at_fork).
• Document "create after fork" and give servers a hook (post_fork in Gunicorn, worker_process_init in Celery).

Takeaway: module-level state created before fork() is duplicated, not shared and not reset; an SDK must not assume one process per import.

Follow-ups:

1. "Why did the master print web-01-2 as its next key, not web-01-6?" After fork the processes have separate memories. The children advanced their copies of the counter; the parent's copy is still at 2. There's no shared state unless you explicitly use shared memory, a pipe or an external store.
2. "You own this SDK and 40 services use it, some with --preload. What do you ship, and how do you find out who was affected?" Ship a patch release that (1) switches keys to uuid4 or a caller-supplied key, and (2) adds a PID check or os.register_at_fork(after_in_child=...) to rebuild the client. Add a test in the SDK that forks and asserts distinct keys and a fresh connection. To find impact: query the provider for idempotency-key collisions or 'key reused with different payload' errors, join with service deploy configs that use --preload or multiple workers, and reconcile affected payments. Communicate it as an incident, not a routine upgrade.
3. "Python 3.12 started warning about fork(). Which situation does that warning target, and does it apply here?" 3.12 emits a DeprecationWarning when os.fork() is called while the process has more than one thread, because only the calling thread survives and locks held by other threads (logging, an SDK's pool lock, a background metrics flusher) can stay locked forever in the child. This snippet is single-threaded, so it isn't triggered, but an SDK that starts a background thread at import (metrics, token refresh) would make every preloaded app hit it and risk deadlocks. Separately, the default multiprocessing start method on Linux moves away from plain fork in 3.14, which changes these assumptions again.

Scoring: Strong: Exact output, explains copy-on-write duplication of both the client and the counter, connects duplicate idempotency keys to silently dropped payments, and proposes an SDK-level fix (uuid4 or caller keys, register_at_fork or PID check) plus a fork test and an impact investigation. Acceptable: Sees that workers share the master's client and gets the key sequence after a short trace, proposes creating clients in post_fork, but does not connect the duplicated keys to a business impact until prompted. Red flag: Expects workers to continue the master's sequence (keys 2,3 then 4,5) as if memory were shared, or thinks each worker re-imports the module and starts from 1.

ST-H-O02 · An unbounded asyncio.Queue between a Kafka poller and slow workers

Hard · Output · Staff / Lead (8–12 years) · backpressure, asyncio, queues, capacity, memory, Kafka consumers, load shedding · 7 min

Question: A Python service reads events from a Kafka partition and hands them to four async workers that call a downstream API. The design review shows an asyncio.Queue() between the poller and the workers. The simulation below counts scheduler ticks instead of real time: each await asyncio.sleep(0) is one yield to the event loop. Case A is the original code; B is after someone added an await in the poller "so the workers get a turn", during a period when the downstream API is slow; C bounds the queue. What does it print?

import asyncio


async def run(label, maxsize, poller_yields, ticks_per_call, events=10_000):
    q = asyncio.Queue(maxsize=maxsize)
    stats = {"puts": 0, "peak": 0, "first_get_after": None}

    async def poller():                        # reads offsets from a Kafka partition
        for offset in range(events):
            await q.put(offset)
            stats["puts"] += 1
            stats["peak"] = max(stats["peak"], q.qsize())
            if poller_yields:
                await asyncio.sleep(0)         # the next poll awaits the network
        for _ in range(4):
            await q.put(None)

    async def worker():                        # calls the downstream API
        while (item := await q.get()) is not None:
            if stats["first_get_after"] is None:
                stats["first_get_after"] = stats["puts"]
            for _ in range(ticks_per_call):
                await asyncio.sleep(0)         # the awaited HTTP call

    await asyncio.gather(poller(), *(worker() for _ in range(4)))
    print(f"{label}: peak={stats['peak']}, first get after {stats['first_get_after']} puts")


asyncio.run(run("A unbounded, fast API", maxsize=0, poller_yields=False, ticks_per_call=1))
asyncio.run(run("B unbounded, slow API", maxsize=0, poller_yields=True, ticks_per_call=8))
asyncio.run(run("C maxsize=100, slow API", maxsize=100, poller_yields=True, ticks_per_call=8))

Expected output:

A unbounded, fast API: peak=10000, first get after 10000 puts
B unbounded, slow API: peak=5000, first get after 1 puts
C maxsize=100, slow API: peak=100, first get after 1 puts

Explanation:

A. await q.put(x) on an unbounded queue never suspends: put only waits while the queue is full, and an unbounded queue is never full. An await that doesn't suspend doesn't give the event loop a turn, so the poller runs its whole loop in one go. All 10,000 items are queued before any worker starts. In a real service the poller keeps going until the partition is drained or the pod runs out of memory, and the first event waits behind the entire backlog, so latency grows with lag.

B. Yielding fixes the starvation (the first get happens after 1 put) but not the capacity problem. The poller adds one item per tick; four workers each need 8 ticks per call, so together they finish 0.5 items per tick. Over 10,000 ticks the backlog grows by about 0.5 per tick, to 5,000. This is plain arithmetic: when arrival rate exceeds service rate, an unbounded buffer grows without limit, and by Little's law the time an item spends in the queue grows with it.

C. With maxsize=100, put suspends when the queue is full, so the poller can only go as fast as the workers. That is backpressure: the slowdown propagates upstream to the poller, and from there to Kafka (which simply holds the data and shows consumer lag). Memory is capped at 100 in-flight items and the end-to-end delay inside the process is bounded.

At Staff level the review comment is not "add maxsize" but where the excess should wait. Kafka is a durable, replayable buffer; process memory is neither. Bound the in-process queue (about rate × target latency), let lag build in Kafka, alert on lag and drain time, and decide explicitly what to shed when lag is too high. Also commit offsets only after processing, so the items sitting in memory aren't lost on a crash.

Takeaway: an unbounded queue doesn't absorb overload, it converts it into memory growth and latency; bounding it turns overload into backpressure on a source that can hold it.

Follow-ups:

1. "Case B: someone proposes raising the worker count from 4 to 40. What do you ask before approving?" What limits the downstream: if the API is slow because it's saturated, 10× concurrency makes it slower for everyone (and may trip its rate limits). Ask for the API's capacity and SLO, size concurrency with Little's law (concurrency ≈ target throughput × latency), and add a per-dependency concurrency limit or token bucket. More workers only help when the downstream has spare capacity.
2. "With a bounded queue, how do you size maxsize, and what goes wrong if it's too large or too small?" Roughly the number of items needed to keep workers busy across a poll's latency (throughput × poll round-trip), plus slack. Too small and workers idle between polls, reducing throughput. Too large and you're back to holding minutes of data in memory: a crash loses or replays more, rebalances take longer because in-flight work must drain, and latency hides in the buffer.
3. "How does this map onto Kafka consumer-group behaviour if the poller blocks on a full queue for a long time?" If the consumer stops calling poll() for longer than max.poll.interval.ms, the broker considers it dead and rebalances the partition, which can cause duplicate processing. The fix is to keep polling while pausing the partition (consumer.pause()) when the queue is full and resuming when it drains, so backpressure doesn't look like a crash.

Scoring: Strong: All three lines exact, explains that put on an unbounded queue never suspends, computes B's backlog from arrival vs service rate, and frames C as backpressure to a durable source with sizing, lag alerts, offset commits and shedding policy. Acceptable: Gets A and C, needs a hint for B's 5,000, and recommends a bounded queue without discussing where the overload should go or how to size it. Red flag: Believes every await yields to the loop, expects the workers to interleave in A, or treats an unbounded in-memory queue as an acceptable buffer for a consumer that may fall behind.

ST-HT-O01 · After the fleet upgrade, a big-integer cohort bitmap can no longer be serialised

Hard & Tricky · Output · Staff / Lead (8–12 years) · Python upgrades, integer string conversion limit, JSON, security fixes, data contracts, migration strategy · 6 min

Question: The growth team stores audience cohorts as a Python int used as a bitmap: bit i set means user i is in the cohort. A nightly job writes each cohort to a JSON API. It ran for two years on the old base image; the week after the fleet moved to the Python 3.12 image, the job started failing. What does this print on Python 3.12?

import json
import sys

# Growth team's cohort bitmap: bit i set means user i is in the cohort.
cohort = 0
for user_id in range(0, 20_000, 7):
    cohort |= 1 << user_id

print("users in cohort:", cohort.bit_count())
print("is user 700 in it?", bool(cohort >> 700 & 1))
print("hex length:", len(hex(cohort)))

for label, fn in [
    ("json.dumps", lambda: json.dumps({"cohort": cohort})),
    ("f-string", lambda: f"{cohort}"),
    ("round trip via hex", lambda: int(hex(cohort), 16) == cohort),
    ("json.loads", lambda: json.loads('{"cohort": ' + "9" * 4301 + "}")),
]:
    try:
        result = fn()
        print(f"{label}: ok, {str(result)[:40]}")
    except ValueError as e:
        print(f"{label}: ValueError: {str(e).split(';')[0]}")

print("limit:", sys.get_int_max_str_digits())

Expected output:

users in cohort: 2858
is user 700 in it? True
hex length: 5002
json.dumps: ValueError: Exceeds the limit (4300 digits) for integer string conversion
f-string: ValueError: Exceeds the limit (4300 digits) for integer string conversion
round trip via hex: ok, True
json.loads: ValueError: Exceeds the limit (4300 digits) for integer string conversion: value has 4301 digits
limit: 4300

The trap: "Python integers have arbitrary precision", so candidates expect json.dumps and the f-string to print a very long number and json.loads to parse one. That is true of the integer: arithmetic, bit operations and bit_count() all still work. What changed is conversion between int and decimal text, which now has a default length limit.

Explanation:

Converting between int and a decimal string is quadratic in the number of digits in CPython, so a few hundred kilobytes of digits in a JSON body could pin a CPU for seconds (CVE-2020-10735). The fix, released in Python 3.11 and backported to security releases 3.7.14, 3.8.14, 3.9.14 and 3.10.7, caps int(str) and str(int) at 4,300 decimal digits by default (sys.get_int_max_str_digits(), configurable with sys.set_int_max_str_digits(), -X int_max_str_digits or PYTHONINTMAXSTRDIGITS).

The bitmap has its highest bit at 19,999, so it has about 6,020 decimal digits:

• json.dumps turns ints into text with int.__repr__, and the f-string uses format, so both raise ValueError.
• Power-of-two bases are exempt because they are linear-time: hex() and int(..., 16) work at any size (5,002 characters here, including 0x).
• The parsing side is the protection: json.loads of a 4,301-digit number raises ValueError before doing the expensive work.

Why it appeared "after the upgrade": the old image ran a Python patch release from before the fix (for example 3.10.4), and the new 3.12 image has it on by default. Nothing in the job's code changed, and the cohorts only crossed ~14,300 users (4,300 digits ≈ 14,284 bits) recently, so even a test suite run on 3.12 with small fixtures stays green.

The right fix is the data contract, not the limit. A bitmap is binary data, so ship it as base64 or hex bytes (cohort.to_bytes(...)), or as a compressed bitmap format (Roaring), with a schema that says so. Raising the limit process-wide also removes the DoS protection for every untrusted JSON body the service parses.

Takeaway: an upgrade can change behaviour through security fixes, not only through deprecations; ints are unbounded, their decimal text is not.

Follow-ups:

1. "As the lead for the 3.12 migration across 300 repos, what in your rollout process should have caught this before production?" Read the 'What's New' and changelog for security-driven behaviour changes and turn each into a search (here: big ints converted to and from text, e.g. bitmaps, crypto, IDs encoded as numbers). Run real production-shaped data, not only unit fixtures, through canary jobs on the new image (shadow runs of batch jobs comparing outputs). Roll out by wave with an easy pin-back to the old image, and track failures by error signature during the bake period.
2. "A team asks to set PYTHONINTMAXSTRDIGITS=0 in the base image to 'make the error go away'. Your answer?" No for the fleet default: it disables the DoS protection for every service that parses untrusted input. If one internal batch job really needs huge decimal ints, it can raise the limit locally with sys.set_int_max_str_digits() (or a scoped context in its own code) and document why. The better fix is to stop encoding binary data as decimal numbers.
3. "Why is decimal conversion quadratic but hex conversion linear?" CPython stores ints in base 2**30 digits. Converting to a power-of-two base just regroups bits. Converting to base 10 needs repeated division (or multiplication when parsing) by a power of ten across the whole number, so the work grows with the square of the length in CPython's implementation (3.12 added a faster divide-and-conquer algorithm for very large values, but the limit stays).

Scoring: Strong: Exact output, names the 4,300-digit limit and why it exists (quadratic decimal conversion, DoS), explains the hex exemption and why the failure showed up only after the image change, and fixes the contract (bytes/base64/Roaring) rather than lifting the limit fleet-wide; adds a concrete migration safeguard. Acceptable: Predicts a failure in serialisation after recalling the limit with a prompt, and proposes encoding as hex or raising the limit locally. Red flag: Insists Python ints can always be printed, or proposes disabling the limit globally without considering untrusted input.

ST-HT-O02 · SDK exception types that break, or quietly change, when they cross a process boundary

Hard & Tricky · Output · Staff / Lead (8–12 years) · exceptions, pickle, multiprocessing, Celery, SDK design, error contracts, metrics · 7 min

Question: The payments SDK defines a small exception hierarchy. A base class counts every error it creates for the sdk_errors_total metric. Errors raised inside worker processes (a ProcessPoolExecutor, Celery, Ray) are pickled and sent back to the parent, which the loop below reproduces. What does it print?

import pickle
from collections import Counter

ERRORS_CREATED = Counter()      # exported as the sdk_errors_total metric


# --- payments_sdk/errors.py ---
class PaymentsError(Exception):
    def __init__(self, *args):
        super().__init__(*args)
        ERRORS_CREATED[type(self).__name__] += 1


class GatewayTimeout(PaymentsError):
    pass


class RateLimited(PaymentsError):
    def __init__(self, retry_after):
        super().__init__(f"retry after {retry_after}s")
        self.retry_after = retry_after


class Declined(PaymentsError):
    def __init__(self, code, reason):
        super().__init__(f"{code}: {reason}")
        self.code = code
        self.reason = reason


# --- a worker process sends each error back to its parent (multiprocessing, Celery, Ray) ---
for err in [GatewayTimeout("bank took 30s"), RateLimited(12), Declined("U16", "risk threshold exceeded")]:
    blob = pickle.dumps(err)
    try:
        clone = pickle.loads(blob)
        print(f"{type(clone).__name__}: {clone} | {vars(clone)}")
    except Exception as e:
        print(f"{type(err).__name__}: pickled fine, loads raised {type(e).__name__}: {e}")

print(dict(ERRORS_CREATED))

Expected output:

GatewayTimeout: bank took 30s | {}
RateLimited: retry after retry after 12ss | {'retry_after': 12}
Declined: pickled fine, loads raised TypeError: Declined.__init__() missing 1 required positional argument: 'reason'
{'GatewayTimeout': 2, 'RateLimited': 2, 'Declined': 1}

The trap: Most candidates expect all three to round-trip unchanged, with the counter at 1 each, because pickle normally restores an object's __dict__ without calling __init__. That is true for ordinary classes, so it's a reasonable instinct, but exceptions pickle differently.

Explanation:

BaseException.__reduce__ returns (type(self), self.args, self.__dict__). Unpickling therefore calls the class again with self.args, then restores __dict__ on top. self.args is whatever was passed to Exception.__init__, not what the subclass's __init__ received.

• GatewayTimeout: args are ('bank took 30s',), calling GatewayTimeout('bank took 30s') reproduces it. But __init__ ran a second time, so the metric counts it twice.
• RateLimited: args are ('retry after 12s',). Unpickling calls RateLimited('retry after 12s'), which formats the message again: retry after retry after 12ss. Then __dict__ is restored, so retry_after is the correct int 12. The object looks right to code that reads attributes and wrong to anyone reading logs or str(e); the counter is also doubled.
• Declined: args hold one formatted string, but __init__ needs two parameters, so pickle.loads raises TypeError, after pickle.dumps succeeded. The failure happens on the receiving side, far from the code that raised. Because the TypeError happens before super().__init__, the counter isn't incremented a second time.

In production this is worse than a wrong message. With ProcessPoolExecutor on 3.12, a result that fails to unpickle is reported as BrokenProcessPool ("A process in the process pool was terminated abruptly...") with the real TypeError buried in __cause__, and the pool is unusable, so every pending future fails. Celery can report it as an unpicklable-exception error instead of the domain error. Callers that branch on except Declined never see a Declined.

Rules for exceptions in a platform SDK:

• Pass every constructor argument to super().__init__(*args) (and build the message in __str__), or define __reduce__ to return (cls, (self.code, self.reason)).
• Keep __init__ free of side effects such as metrics, logging or network calls; count errors where they are raised or handled.
• Add a test that round-trips every public exception through pickle (and copy.copy, which uses the same protocol) and compares str(), args and attributes.

Takeaway: an exception is rebuilt by calling its class with self.args, so its constructor must accept its own args and must be safe to run twice.

Follow-ups:

1. "Write the smallest change that makes Declined round-trip correctly while keeping the message format." Either super().__init__(code, reason) plus def __str__(self): return f"{self.code}: {self.reason}", or keep the formatted message and add def __reduce__(self): return (type(self), (self.code, self.reason), self.__dict__). The first is simpler and also makes args meaningful for structured logging.
2. "Why don't ordinary classes have this problem?" For plain objects, protocol 2+ pickling uses copyreg.__newobj__: it calls cls.__new__(cls) and then restores __dict__, skipping __init__. BaseException overrides __reduce__ to re-call the class with args, because exception state historically lived in args.
3. "How would you catch this class of bug across all 40 teams' code, not only in your SDK?" A shared pytest plugin or lint rule in the platform template: discover all Exception subclasses in a package and round-trip them through pickle in CI; a ruff/flake8 custom rule flagging exception __init__ signatures that don't forward all parameters to super().__init__; and documenting the contract in the SDK guidelines. Also watch for BrokenProcessPool in error tracking, since it is often this bug in disguise.

Scoring: Strong: All four lines exact, explains BaseException.__reduce__ (re-call with args, then restore __dict__), distinguishes the three failure modes (double side effect, garbled message, failed unpickle), and states SDK rules plus a CI round-trip test; mentions how it surfaces (BrokenProcessPool). Acceptable: Spots the Declined failure and the double count after tracing, but needs prompting to see why RateLimited's message is garbled while its attribute is correct. Red flag: Says exceptions can't be pickled at all, or that pickle never calls __init__ for any class, and so predicts three clean round-trips.

ST-CR-01 · A "minor" release of the internal HTTP SDK that leaks tenant headers and breaks 40 services

Easy · Code review · Staff / Lead (8–12 years) · SDK design, platform standards, shared mutable state, multi-tenancy, backward compatibility, semantic versioning, deprecation · 15 min

Context:

PR #412 · platform-sdk/acme_http/client.py · author: Sneha (platform team) · reviewers: you (staff engineer, SDK owner)

acme_http is the internal HTTP client every Python service at a Bengaluru fintech uses to call other services: about 40 repos depend on it with acme-http>=1.7,<2 in their requirements. Release 1.8.0 is in this PR. Sneha's description:

• Adds for_tenant(tenant_id) so multi-tenant services (the lending platform serves 30 NBFC partners from one deployment) can stamp X-Tenant-ID on outgoing calls.
• Renames timeout to timeout_s to follow our "units in the name" standard.
• Minor version bump, since it is all additive.

build_request() stands in for the real send path; it returns what would go on the wire. Review it as the owner of an SDK that 40 teams will pick up automatically on their next build.

Code under review:

# --- platform-sdk/acme_http/client.py ---
SDK_VERSION = "1.8.0"


class Client:
    default_headers: dict[str, str] = {"User-Agent": f"acme-http/{SDK_VERSION}"}

    def __init__(self, base_url: str, timeout_s: float = 5.0, retries: int = 2):
        self.base_url = base_url.rstrip("/")
        self.timeout_s = timeout_s
        self.retries = retries
        self.headers = self.default_headers

    def for_tenant(self, tenant_id: str) -> "Client":
        """Return a client that sends X-Tenant-ID on every request."""
        self.headers["X-Tenant-ID"] = tenant_id
        return self

    def build_request(self, method: str, path: str) -> dict:
        return {
            "method": method,
            "url": f"{self.base_url}/{path.lstrip('/')}",
            "headers": dict(self.headers),
            "timeout": self.timeout_s,
        }

What a strong reviewer raises:

1. High: Cross-tenant leak through a shared class-level dict. self.headers = self.default_headers does not copy: every Client in the process holds a reference to the same dict object, the class attribute. for_tenant() then mutates that dict and returns self. After ledger.for_tenant("nbfc-0007") and ledger.for_tenant("nbfc-0012"), both handles, the base client, the unrelated kyc client and every client created later send X-Tenant-ID: nbfc-0012. In a multi-tenant lending service that means partner A's loan-book writes are attributed to partner B (or, if downstream services authorise by that header, one partner reads another's data). Under concurrency it is worse: two requests in flight race on one dict, so the bug is intermittent and hard to reproduce. Fix: Give each instance its own dict ({**DEFAULT_HEADERS, **headers}), make the module defaults read-only (MappingProxyType), and make for_tenant() return a new client instead of mutating and returning self. A method that returns self and changes it reads like a builder but behaves like a global. Add a test with two tenants and an unrelated client in one process, as above.
2. High: Breaking change shipped as a minor version. Renaming timeout to timeout_s makes every existing Client(url, timeout=3) raise TypeError: ... unexpected keyword argument 'timeout'. Because consumers pin >=1.7,<2, 40 services pick up 1.8.0 on their next build and fail at start-up (or, worse, in a code path that only runs at night). "Additive" is the claim; the diff removes a public parameter. Fix: Keep accepting timeout in 1.8 with a DeprecationWarning (stacklevel=2 so the warning points at the caller's line), document the removal for 2.0, and publish a codemod or a ruff/grep check the teams can run. Make new options keyword-only (*) so future additions and reorders cannot break positional callers. Gate releases on a "consumer contract" CI job that runs a sample of downstream test suites against the release candidate.
3. Medium: The SDK has no compatibility policy that tooling enforces. Both bugs would be caught mechanically: an API-diff check (for example griffe check on the public surface) flags a removed parameter, and a lint rule against mutable class attributes in the SDK catches the shared dict. Relying on reviewers to spot them across 40 consumers does not scale. Fix: Add a public-API diff to the SDK's CI that fails a minor/patch release on removals or signature changes, and ship a CHANGELOG section "Deprecated / Removed" as part of the PR template.
4. Low: default_headers as a public, mutable class attribute is an invitation. Some service will eventually do Client.default_headers["X-Debug"] = "1" in its startup code and change the behaviour of every library in the process that uses the SDK. Fix: Module-level read-only defaults, and per-client headers= for customisation.

Tests that expose the bug:

# --- tests/test_client.py ---
import warnings

# One process, two downstream services, two tenants (the lending platform's real shape).
ledger = Client("https://ledger.internal/")
kyc = Client("https://kyc.internal")

partner_a = ledger.for_tenant("nbfc-0007")
partner_b = ledger.for_tenant("nbfc-0012")

assert partner_a.build_request("POST", "/v1/entries")["headers"]["X-Tenant-ID"] == "nbfc-0007", \
    "tenant 0007's request went out stamped as another tenant"
assert partner_b.build_request("POST", "/v1/entries")["headers"]["X-Tenant-ID"] == "nbfc-0012"

# The base clients, and unrelated clients, must not inherit a tenant from someone else's call.
assert "X-Tenant-ID" not in ledger.build_request("GET", "/health")["headers"]
assert "X-Tenant-ID" not in kyc.build_request("GET", "/v1/pan/verify")["headers"]
assert "X-Tenant-ID" not in Client("https://new.internal").build_request("GET", "/")["headers"]
assert kyc.build_request("GET", "/x")["url"] == "https://kyc.internal/x"

# 40 repos call Client(url, timeout=3). A minor release must keep that working (with a warning).
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    legacy = Client("https://risk.internal", timeout=3)
assert legacy.timeout_s == 3
assert any(issubclass(w.category, DeprecationWarning) for w in caught), "callers need a migration signal"

# The new spelling works and does not warn.
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    modern = Client("https://risk.internal", timeout_s=2.5)
assert modern.timeout_s == 2.5 and not caught

Fixed version:

# --- platform-sdk/acme_http/client.py ---
import warnings
from types import MappingProxyType

SDK_VERSION = "1.8.0"

# Read-only: nobody can mutate the process-wide defaults by accident.
DEFAULT_HEADERS = MappingProxyType({"User-Agent": f"acme-http/{SDK_VERSION}"})
_UNSET = object()


class Client:
    def __init__(self, base_url: str, timeout_s: float = 5.0, retries: int = 2, *,
                 headers: dict[str, str] | None = None, timeout: object = _UNSET):
        if timeout is not _UNSET:
            # Shim for the 1.x spelling. Remove in 2.0, announced in the changelog.
            warnings.warn("Client(timeout=...) is deprecated; use timeout_s=... (removed in 2.0)",
                          DeprecationWarning, stacklevel=2)
            timeout_s = timeout
        self.base_url = base_url.rstrip("/")
        self.timeout_s = timeout_s
        self.retries = retries
        # Each client owns its own copy of the headers.
        self.headers = {**DEFAULT_HEADERS, **(headers or {})}

    def for_tenant(self, tenant_id: str) -> "Client":
        """Return a NEW client that sends X-Tenant-ID; this client is unchanged."""
        return Client(self.base_url, self.timeout_s, self.retries,
                      headers={**self.headers, "X-Tenant-ID": tenant_id})

    def build_request(self, method: str, path: str) -> dict:
        return {
            "method": method,
            "url": f"{self.base_url}/{path.lstrip('/')}",
            "headers": dict(self.headers),
            "timeout": self.timeout_s,
        }

Follow-ups:

1. "Sneha argues: "Nobody passes timeout= positionally or by name, I grepped our main repos." How do you decide whether this is breaking?" The public signature is the contract, not the usages one person can find. Grepping the main repos misses notebooks, cron scripts, forks and anything built from older branches. The cheap, safe path is a deprecation cycle: accept both for at least one minor release, measure usage (emit a metric or log from the shim tagged by service), and only remove in 2.0 when the metric is zero. At Staff level the point is that the cost of the shim is tiny and the cost of breaking 40 builds is a day of many teams' time.
2. "Why is return self from a method named for_tenant a design smell even when there is no shared dict?" The name promises a derived object; callers will write a = c.for_tenant("x"); b = c.for_tenant("y") and expect two independent clients. Mutating self makes a is b is c, so the last call wins for everyone holding a reference, including code in other threads. Either return a new immutable object (preferred for anything shared across requests) or name it set_tenant() and document that the client is not shareable.
3. "How would you roll out the 2.0 removal across 40 repos without a flag day?" Publish the deprecation in 1.8 and track shim usage per service. Provide an automated codemod PR (a bot that opens a PR per repo) and a lint rule. Turn DeprecationWarning into an error in each repo's test config (filterwarnings = error::DeprecationWarning:acme_http) once it is clean, so it cannot regress. Release 2.0 when usage is zero, and keep the <2 pins, so repos upgrade on their own schedule while 1.x gets only security fixes for a published period.
4. "Would you put tenant identity in a header that the SDK sets at all?" It's acceptable for routing and attribution, but downstream services must not authorise on a header the caller sets freely. Tenant identity for authorisation should come from a signed token (a JWT claim or mTLS identity plus a scoped token) that the server verifies. That way, a client-side bug like this one produces wrong attribution that is caught, not a cross-tenant data read.

Scoring: Strong: Spots that self.headers aliases the class dict and that for_tenant mutates and returns self, explains the cross-tenant consequence (including concurrency), and fixes both. Calls the rename a breaking change in a minor release, given <2 pins, and proposes a deprecation shim with stacklevel=2, a removal plan and mechanical enforcement (API-diff in CI, consumer test runs). Talks about policy, not just this diff. Acceptable: Finds the shared-headers leak and fixes it with a copy; recognises the rename is breaking once asked "what happens to callers using timeout=?", and proposes a shim. Red flag: Treats the rename as fine because it matches the naming standard, or suggests consumers "just pin 1.7". Fixes the leak with a lock around the shared dict. Comments only on type hints or docstrings. For an SDK owner at this level, missing the blast radius of a minor release is a serious gap.

ST-CR-02 · Percentage rollouts in the feature-flag SDK that pass every unit test

Easy & Tricky · Code review · Staff / Lead (8–12 years) · feature flags, progressive delivery, hashing, PYTHONHASHSEED, experimentation, platform SDK, determinism · 15 min

Context:

PR #96 · flags-sdk/flags/rollout.py · author: Arjun · reviewers: you (staff engineer, developer platform)

The company is replacing a paid feature-flag vendor with an in-house SDK. Flag definitions are pulled from a config service; evaluation happens in-process in about 60 services (FastAPI pods, Celery workers, the notification batch jobs), so a check costs microseconds and survives a config-service outage.

This PR adds percentage rollouts. Arjun's description:

is_enabled(flag, user_id) puts each user in a bucket 0–99 and enables the flag for buckets below rollout_percent. Allow-listed users (internal testers) always get it. Unit tests pass and the distribution is uniform: 10% gives 9.8–10.3% on 100k user IDs. Next quarter the growth team will run 6–8 concurrent experiments on this.

The CI suite runs in one process with a fixed PYTHONHASHSEED. Review it as the owner of the flags platform.

Code under review:

# --- flags-sdk/flags/rollout.py ---
from dataclasses import dataclass


@dataclass(frozen=True)
class Flag:
    key: str
    rollout_percent: float              # 0..100
    allow_list: frozenset[str] = frozenset()


def _bucket(user_id: str) -> int:
    """Stable bucket in 0..99 for a user."""
    return hash(user_id) % 100


def is_enabled(flag: Flag, user_id: str) -> bool:
    if user_id in flag.allow_list:
        return True
    return _bucket(user_id) < flag.rollout_percent

What a strong reviewer raises:

1. High: hash() of a str is randomised per process. Since Python 3.3, string hashing is salted with a random seed chosen at interpreter start (unless PYTHONHASHSEED is fixed). So hash("user-00042") % 100 is a different bucket in every pod, every Celery worker and after every restart. The same user sees the new checkout on one request and the old one on the next; the web tier and the worker that sends the confirmation email disagree; experiment exposure logs are meaningless. The unit tests pass because they run in one process with a fixed seed, which is precisely the condition production never has. Fix: Use a stable, documented hash of the input bytes: hashlib.sha256 (or murmur3/xxhash if profiling says so) of "{salt}:{user_id}", take 8 bytes as an integer, mod the bucket count. Add a test that evaluates in a separate process with a different seed, and a golden test that pins a few (flag, user) -> bucket values so a future refactor cannot silently reshuffle every user.
2. High: The bucket ignores the flag. Every flag at 10% selects the same 10% of users. With 6–8 concurrent experiments, one slice of users receives every treatment and the rest receive none: effects are confounded (you cannot tell whether conversion moved because of the checkout or the search change), and the same customers absorb the risk of every launch. Fix: Include a per-flag salt in the hash input, defaulting to the flag key. An explicit salt field lets the experimentation team deliberately share a salt for mutually exclusive layers, or change it to re-randomise after a bad launch.
3. Medium: Integer buckets 0–99 give 1% granularity. rollout_percent=0.5 compares bucket < 0.5, which means bucket 0 only, about 1% of users, twice what was asked. For a risky payments change on a 50M-user base, the smallest real step is 500,000 people. Fix: Use 10,000 buckets (basis points) and compare against rollout_percent * 100. Validate the range in __post_init__ so rollout_percent=1000 (someone thinking in basis points) fails at load time, not in production.
4. Low: The description's evidence does not test the claim. A uniform distribution in one process says nothing about stability across processes, independence across flags or monotonic ramps, which are the three properties a rollout system needs. Fix: Make those three properties the test suite (as above). For an SDK evaluated in 60 services, also add a cross-language golden file if a JavaScript or Go SDK must agree on the same buckets.

Tests that expose the bug:

# --- tests/test_rollout.py ---
import os
import subprocess
import sys

USERS = [f"user-{i:05d}" for i in range(4000)]
CHECKOUT = Flag("new-checkout", 10)


def enabled(flag):
    return {u for u in USERS if is_enabled(flag, u)}


# Child mode: another "pod" evaluates the same flag and prints who got it.
if os.environ.get("FLAGS_CHILD"):
    print(",".join(sorted(enabled(CHECKOUT))))
    sys.exit(0)


def evaluate_in_new_process(seed):
    env = {**os.environ, "FLAGS_CHILD": "1", "PYTHONHASHSEED": seed}
    out = subprocess.run([sys.executable, __file__], env=env, capture_output=True, text=True, check=True)
    return set(out.stdout.strip().split(","))


here = enabled(CHECKOUT)
assert 300 <= len(here) <= 500, len(here)                  # roughly 10%
assert is_enabled(Flag("new-checkout", 0, frozenset({"qa-meera"})), "qa-meera")

# Every pod, worker and restart must give a user the same answer.
assert evaluate_in_new_process("101") == here, "a user's flag flips between pods / after a restart"
assert evaluate_in_new_process("202") == here

# Ramping 10% -> 25% must only ADD users; nobody who had the feature loses it.
assert here <= enabled(Flag("new-checkout", 25))

# Two independent 10% experiments must not land on the same users.
search = enabled(Flag("new-search-ranking", 10))
overlap = len(here & search) / len(here)
assert overlap < 0.3, f"{overlap:.0%} of checkout users are also in the search experiment"

# Sub-percent rollouts are needed for risky launches (0.5% of 50M users is 250k people).
assert 5 <= len(enabled(Flag("payments-v2", 0.5))) <= 40

Fixed version:

# --- flags-sdk/flags/rollout.py ---
import hashlib
from dataclasses import dataclass

BUCKETS = 10_000                        # 0.01% granularity


@dataclass(frozen=True)
class Flag:
    key: str
    rollout_percent: float              # 0..100
    allow_list: frozenset[str] = frozenset()
    salt: str = ""                      # defaults to the key; set explicitly to re-randomise or to link flags

    def __post_init__(self):
        if not 0 <= self.rollout_percent <= 100:
            raise ValueError(f"{self.key}: rollout_percent must be within 0..100, got {self.rollout_percent}")


def _bucket(salt: str, user_id: str) -> int:
    """Bucket in 0..BUCKETS-1, identical in every process, host and Python version."""
    digest = hashlib.sha256(f"{salt}:{user_id}".encode()).digest()
    return int.from_bytes(digest[:8], "big") % BUCKETS


def is_enabled(flag: Flag, user_id: str) -> bool:
    if user_id in flag.allow_list:
        return True
    return _bucket(flag.salt or flag.key, user_id) < flag.rollout_percent * BUCKETS / 100

Follow-ups:

1. "Why did the tests pass, and what does that tell you about how platform code should be tested?" The suite runs in one interpreter, and CI fixes PYTHONHASHSEED, so hash() is consistent within the run. The property that matters, agreement between processes, was never exercised. Platform code needs tests for its contract, not for its implementation: here, cross-process stability (spawn a subprocess with a different seed), cross-flag independence and monotonic ramps. Pin golden values for a few users so the hashing scheme is part of the versioned contract.
2. "A teammate suggests setting PYTHONHASHSEED=0 in every Dockerfile instead. Why not?" It couples correctness of a business feature to an environment variable that 60 deployment manifests, local shells, notebooks and batch hosts must all set identically, forever. It also disables hash randomisation, which exists to protect dict-heavy services against hash-flooding denial of service from attacker-chosen keys. And hash() is not guaranteed stable across Python versions, so the 3.13 upgrade could reshuffle everyone anyway. Use an explicit hash that is part of the SDK's contract.
3. "The flag is ramped 10% to 50% to 10% again after an incident. Which users lose the feature, and is that what you want?" With a threshold on a stable bucket, the same users who were added between 10% and 50% are the ones removed: buckets 10–49 drop out, buckets 0–9 keep it. That is usually right (the original cohort keeps a consistent experience). If the incident affected specific users, you want a targeted deny list, not a ramp-down, and you want the evaluation event logged so you can find who was exposed.
4. "Should bucketing use user_id, device ID or account ID?" Whatever unit the experiment is analysed on and the treatment must be consistent for. A checkout change should be per account (a family sharing one account should see one checkout); a logged-out landing page can only use a device or anonymous ID. Make the unit an explicit field of the flag so analysts and the SDK agree, and handle the logged-out to logged-in transition deliberately.

Scoring: Strong: Identifies salted hash() immediately, explains why it hides in tests (single process, fixed seed), and describes the user-visible failure across pods and workers. Spots that the bucket ignores the flag key and explains confounded experiments. Raises granularity and validation, insists on cross-process and golden tests, and rejects PYTHONHASHSEED=0 as a fix with reasons. Acceptable: Finds the hash() problem (possibly after a hint about "multiple pods") and fixes it with hashlib; sees the shared-bucket problem when asked how two experiments interact. Red flag: Approves because the distribution is uniform and tests are green. Suggests random.random() < p per request, or seeding random with the user ID through hash(). Proposes fixing the env var across all deployments as the solution.

ST-CR-03 · Monthly chargeback of the shared data-platform bill to product teams

Medium · Code review · Staff / Lead (8–12 years) · cost attribution, FinOps, chargeback, money as integers, largest remainder, tagging, incentives · 15 min

Context:

PR #233 · finops/chargeback.py · author: Divya · reviewers: you (staff engineer, data platform)

The shared Spark and Trino clusters cost about ₹1.2 crore a month. Leadership has asked for chargeback: from April, each product team's budget is debited for its share, and finance books one journal entry per team. The bill arrives from the cloud provider's invoice in paise. Usage comes from the job scheduler: one row per job run with the team tag from the job's config (older jobs often have no tag) and vCPU-hours.

Divya's description:

Splits the bill pro-rata by vCPU-hours. Money stays in integer paise. Untagged jobs are skipped because we can't attribute them. Output feeds the finance journal export (dict[team, paise]).

Review it as the engineer who owns the platform and will have to defend these numbers to finance and to 15 engineering managers.

Code under review:

# --- finops/chargeback.py ---
UNATTRIBUTED = "UNATTRIBUTED"


def usage_by_team(job_runs: list[dict]) -> dict[str, float]:
    usage: dict[str, float] = {}
    for run in job_runs:
        team = run.get("team")
        if not team:
            continue                     # can't attribute untagged jobs
        usage[team] = usage.get(team, 0.0) + run["vcpu_hours"]
    return usage


def chargeback(bill_paise: int, job_runs: list[dict]) -> dict[str, int]:
    usage = usage_by_team(job_runs)
    total = sum(usage.values())
    return {team: round(bill_paise * hours / total) for team, hours in usage.items()}

What a strong reviewer raises:

1. High: Untagged usage is silently billed to the teams that tagged their jobs. usage_by_team drops rows without a team, but the whole bill is still divided by the tagged total. If 30% of vCPU-hours are untagged, every tagged team pays about 43% more than it used. The incentive is exactly backwards: teams that tag carefully subsidise those that don't, and nobody can see the 30% that needs fixing. Chargeback numbers are disputed line by line by engineering managers; this is the line they will find. Fix: Attribute untagged usage to an explicit UNATTRIBUTED line owned by the platform team and show it on every report. Track it as a metric with a target (for example, under 5% by Q2), and make the team tag mandatory at job submission for new jobs, so the bucket shrinks instead of hiding.
2. High: The allocations don't sum to the invoice. "Integer paise" is only true of the output type: round(bill * hours / total) rounds each share independently, so three equal teams on ₹1,20,00,000.01 get ₹40,00,000.00 each and one paisa disappears; other splits create a paisa. The journal entry doesn't balance, and finance rejects the export (or someone books a manual adjustment every month). The intermediate value is also a float, so for very large bills the shares lose precision before rounding. Fix: Floor every share and distribute the remainder with the largest-remainder method, breaking ties by a stable key (team name), so the total is exact by construction and the result is reproducible. For very large values, compute the floors with integer arithmetic (bill * num // den on integer units) instead of a float division.
3. Medium: A month with no usage crashes. total == 0 raises ZeroDivisionError. Committed-use discounts and reserved nodes are billed whether or not jobs run, so a quiet month (or a scheduler export failure that returns no rows) aborts the whole finance run at month-end close. Fix: Handle total == 0 explicitly (bill to UNATTRIBUTED). Separately, alert if usage rows are missing for a month that has a bill: an empty export is more likely a pipeline failure than a quiet month.
4. Medium: Pro-rata by vCPU-hours is a policy decision hidden in a function. Idle reserved capacity, storage, and the platform team's own overhead are all being spread by compute share. That may be right, but it is a choice with budget consequences for 15 teams, and it should be written down and agreed with finance, not discovered in code review. Fix: Document the allocation policy (which cost pools, which drivers: vCPU-hours for compute, TB-months for storage, a fixed platform fee) in a short design note signed off by finance and the EMs; make the driver per cost pool explicit in the code; and run a month of showback before turning on chargeback, so teams can contest numbers before money moves.

Tests that expose the bug:

# --- tests/test_chargeback.py ---
def runs(*pairs):
    return [{"team": t, "vcpu_hours": h} for t, h in pairs]


# 1. The journal must balance to the paisa: finance rejects an export that doesn't sum to the invoice.
bill = 1_20_00_000_01                    # ₹1,20,00,000.01 in paise
out = chargeback(bill, runs(("payments", 1), ("risk", 1), ("growth", 1)))
assert sum(out.values()) == bill, f"journal is off by {bill - sum(out.values())} paise"
assert max(out.values()) - min(out.values()) <= 1, out

odd = chargeback(1_000, runs(("payments", 2.5), ("risk", 2.5), ("search", 2.5), ("ads", 2.5)))
assert sum(odd.values()) == 1_000, odd

# 2. Untagged usage must stay visible, not be silently billed to the teams that DID tag their jobs.
out = chargeback(10_000, runs(("payments", 70), (None, 20), ("", 10)))
assert out == {"payments": 7_000, UNATTRIBUTED: 3_000}, out

# 3. A month with reserved capacity but no jobs (the platform freeze in Diwali week happened once).
assert chargeback(5_000, []) == {UNATTRIBUTED: 5_000}

# 4. Same input, same output, every time it is re-run for an audit.
inp = runs(("risk", 1), ("payments", 1), ("growth", 1))
assert chargeback(100, inp) == chargeback(100, list(reversed(inp)))

Fixed version:

# --- finops/chargeback.py ---
UNATTRIBUTED = "UNATTRIBUTED"            # owned by the platform team; shown on every report


def usage_by_team(job_runs: list[dict]) -> dict[str, float]:
    usage: dict[str, float] = {}
    for run in job_runs:
        team = run.get("team") or UNATTRIBUTED
        usage[team] = usage.get(team, 0.0) + run["vcpu_hours"]
    return usage


def chargeback(bill_paise: int, job_runs: list[dict]) -> dict[str, int]:
    """Split an integer bill pro-rata; the result always sums to exactly `bill_paise`."""
    usage = {team: hours for team, hours in usage_by_team(job_runs).items() if hours > 0}
    total = sum(usage.values())
    if total == 0:
        return {UNATTRIBUTED: bill_paise}

    # Largest-remainder (Hamilton) method: floor every share, then hand the leftover paise
    # one by one to the largest fractional remainders. Ties break by team name, so re-runs match.
    exact = {team: bill_paise * hours / total for team, hours in usage.items()}
    alloc = {team: int(share) for team, share in exact.items()}
    leftover = bill_paise - sum(alloc.values())
    by_remainder = sorted(exact, key=lambda t: (-(exact[t] - alloc[t]), t))
    for team in by_remainder[:leftover]:
        alloc[team] += 1
    return dict(sorted(alloc.items()))

Follow-ups:

1. "An engineering manager says: "Our share went up 40% this month but our jobs didn't change." How should the system help you answer that?" Every allocation must be explainable from stored inputs: persist the invoice line items, the raw usage rows with job IDs, the policy version and the computed allocation for each month, so the run can be reproduced. Common causes are someone else's usage dropping (pro-rata shares shift even if yours didn't change), untagged jobs being newly tagged to that team, or a price change. A per-team drill-down by job and a month-over-month diff report usually ends the conversation.
2. "Why is the largest-remainder method a better default than rounding and "putting the difference on the biggest team"?" Both balance the journal, but plugging the residue onto one team biases that team every month, and the residue can be many paise when there are hundreds of teams. Largest remainder minimises each team's rounding error (no team is off by more than one paisa from its exact share) and, with a stable tie-break, is reproducible. It is the same method used for seat allocation, so it is easy to explain.
3. "Should this be chargeback or showback, and how do you decide?" Showback (report costs without moving budget) changes behaviour cheaply and lets the data quality be fixed first. Chargeback is worth it when teams have real control over their usage and the attribution is trusted, otherwise it creates disputes and gaming (for example, moving jobs to untagged accounts). I'd run showback until unattributed usage is low and the numbers have survived a couple of months of review, then switch with finance's agreement.
4. "The ML team says spot instances make their jobs cheaper, so pro-rata by vCPU-hours overcharges them. Fair?" Yes, if they actually run on cheaper capacity. Allocate per cost pool: on-demand, reserved and spot nodes are separate pools with their own prices, and each team's usage is priced in the pool it ran in. That rewards the behaviour you want (using spot for tolerant workloads) instead of averaging it away.

Scoring: Strong: Finds that untagged usage inflates every tagged team's bill and frames it as an incentive problem, not just a bug. Shows the journal not balancing and fixes it with largest-remainder and a stable tie-break. Catches the zero-usage crash and links it to month-end close. Pushes the allocation policy into a written, finance-approved design and suggests showback before chargeback. Acceptable: Finds the rounding imbalance and the divide by zero; recognises the untagged problem when asked "where did the untagged jobs' cost go?". Red flag: Fixes the imbalance by switching to Decimal and rounding each share (which still doesn't sum), or by adding the residue to an arbitrary team. Thinks skipping untagged usage is fine "because we can't attribute it". Treats cost allocation as purely a coding question.

ST-CR-04 · The platform's "exactly-once" consumer loop that loses wallet credits on a pod restart

Hard · Code review · Staff / Lead (8–12 years) · event-driven architecture, delivery semantics, idempotency, offsets, transactions, dead-letter queues, sqlite3, reliability · 25 min

Context:

PR #57 · streamkit/consumer.py · author: Rohit · reviewers: you (staff engineer, payments platform)

streamkit is the shared library that 12 teams use to consume events from the message bus. Each consumer stores its committed offset (the next offset to read) and its side effects in its own Postgres database; the bus redelivers everything from the committed offset after a restart or rebalance. The first adopter is the wallet service, which credits cashback when a cashback.granted event arrives.

Rohit's description:

run_batch() gives teams exactly-once processing out of the box: - we commit the offset up front, so one poison message can never block the partition; - every event ID is recorded in processed, so redeliveries are skipped; - handler errors are logged and the batch continues.

Teams write handler(conn, msg) and never think about offsets again.

The tests use sqlite3 in place of Postgres; transaction semantics are the same for this code. Handlers raise TransientError for "try again later" (a downstream timeout) and any other exception for "this message is bad". Pods are killed by Kubernetes during deploys and node drains, which the tests simulate with an exception the library does not catch. Review it as the owner of the payments platform's reliability.

Code under review:

# --- streamkit/consumer.py ---
import logging
import sqlite3

logger = logging.getLogger("streamkit")
logger.addHandler(logging.NullHandler())


class TransientError(Exception):
    """Raised by handlers when a retry later may succeed (timeouts, 503s)."""


def setup(conn: sqlite3.Connection, topic: str) -> None:
    conn.execute("CREATE TABLE IF NOT EXISTS offsets (topic TEXT PRIMARY KEY, next_offset INTEGER)")
    conn.execute("CREATE TABLE IF NOT EXISTS processed (event_id TEXT PRIMARY KEY)")
    conn.execute("CREATE TABLE IF NOT EXISTS dead_letters (event_id TEXT PRIMARY KEY, error TEXT)")
    conn.execute("INSERT OR IGNORE INTO offsets VALUES (?, 0)", (topic,))
    conn.commit()


def committed_offset(conn: sqlite3.Connection, topic: str) -> int:
    return conn.execute("SELECT next_offset FROM offsets WHERE topic = ?", (topic,)).fetchone()[0]


def run_batch(conn: sqlite3.Connection, topic: str, messages: list[dict], handler) -> None:
    if not messages:
        return
    # Commit first so a poison message can never block the partition.
    conn.execute("UPDATE offsets SET next_offset = ? WHERE topic = ?", (messages[-1]["offset"] + 1, topic))
    conn.commit()
    for msg in messages:
        seen = conn.execute("SELECT 1 FROM processed WHERE event_id = ?", (msg["event_id"],)).fetchone()
        if seen:
            continue
        conn.execute("INSERT INTO processed VALUES (?)", (msg["event_id"],))
        conn.commit()
        try:
            handler(conn, msg)
            conn.commit()
        except Exception:
            logger.exception("handler failed for %s", msg["event_id"])
            conn.rollback()

What a strong reviewer raises:

1. High: Offsets are committed before the work is done: this is at-most-once, not exactly-once. The batch's final offset is committed first. If the pod is killed during message 2 (a deploy, a node drain, an OOM), the bus resumes after message 3: cb-1002 and cb-1003 are never processed. In a wallet that is cashback customers were promised and never received, with nothing in the logs, because the process that would have logged it was killed. Fix: Commit the offset after the effect, in the same transaction as the effect, one message at a time (or per small batch). With the offset, the dedupe marker and the handler's writes in one database transaction, any crash rolls back all three and the message is redelivered.
2. High: The dedupe marker is committed before the handler runs. INSERT INTO processed is committed on its own; if the handler then fails (or the pod dies), the event is marked as processed but has no effect. Every redelivery skips it. The mechanism meant to give "exactly once" turns any failure into "never". Fix: Insert the marker inside the same transaction as the handler's writes (with conn:), after the handler succeeds. This only works when the effect lives in the same database; for external side effects (an SMS, a call to the bank), pass an idempotency key derived from event_id to the downstream API or use an outbox table.
3. High: Handler failures are logged and dropped. A TransientError (the ledger timed out for two seconds) is treated the same as a malformed event: rolled back, logged, and, because the marker and the offset were already committed, gone forever. Logs are not a recovery mechanism: nobody replays from Kibana. Fix: Distinguish the two. Transient: stop and re-raise without advancing, so the message is retried after backoff (keeping order within the partition). Permanent: write it to a durable dead-letter table (or DLQ topic) with the error, advance, and alert on DLQ growth. Provide a replay tool for the DLQ. That is how "a poison message never blocks the partition" is achieved without losing data.
4. Medium: The PR promises "exactly-once" and "never think about offsets", which sets the wrong expectations for 12 teams. The bus delivers at least once; the library can only make effects in the same database idempotent. Teams calling payment gateways or sending notifications from handlers will still double-send on replay unless they use idempotency keys. Fix: Document the contract precisely ("at-least-once delivery; effects in the provided transaction are applied once; external calls must be idempotent") and give the handler API a shape that makes the safe path the easy one, for example an outbox helper for external effects.
5. Low: processed grows forever, one row per event, and is checked on every message. Fix: Store markers with a timestamp and expire them after a window comfortably longer than the bus's maximum redelivery horizon (retention plus the longest plausible rewind), or key dedupe on (partition, offset) when events are only ever replayed from the same log.

Tests that expose the bug:

# --- tests/test_consumer.py ---
TOPIC = "cashback.granted"
LOG = [
    {"offset": 0, "event_id": "cb-1001", "user": "ananya", "paise": 5_000},
    {"offset": 1, "event_id": "cb-1002", "user": "vikram", "paise": 2_500},
    {"offset": 2, "event_id": "cb-1003", "user": "ananya", "paise": 1_000},
]


class PodKilled(BaseException):
    """SIGKILL during a deploy: nothing after this line runs."""


def fresh_db():
    conn = sqlite3.connect(":memory:")
    setup(conn, TOPIC)
    conn.execute("CREATE TABLE wallet (event_id TEXT, user TEXT, paise INTEGER)")
    conn.commit()
    return conn


def credit(conn, msg):
    conn.execute("INSERT INTO wallet VALUES (?, ?, ?)", (msg["event_id"], msg["user"], msg["paise"]))


def deliver(conn, handler):
    """What the bus does: hand over everything from the committed offset."""
    run_batch(conn, TOPIC, LOG[committed_offset(conn, TOPIC):], handler)


def wallet(conn):
    return sorted(conn.execute("SELECT event_id, paise FROM wallet"))


ALL_CREDITS = [("cb-1001", 5_000), ("cb-1002", 2_500), ("cb-1003", 1_000)]

# 1. The pod is killed while crediting cb-1002. After restart, every credit lands exactly once.
conn = fresh_db()


def dies_on_second(conn, msg):
    if msg["event_id"] == "cb-1002":
        raise PodKilled()
    credit(conn, msg)


try:
    deliver(conn, dies_on_second)
except PodKilled:
    pass
deliver(conn, credit)                    # the new pod
assert wallet(conn) == ALL_CREDITS, f"after restart the wallet holds {wallet(conn)}"

# 2. Redelivery after a rebalance (offset rewound) must not double-credit.
conn.execute("UPDATE offsets SET next_offset = 0")
conn.commit()
deliver(conn, credit)
assert wallet(conn) == ALL_CREDITS

# 3. A downstream timeout is retried, not dropped.
conn = fresh_db()
attempts = {"cb-1002": 0}


def flaky(conn, msg):
    if msg["event_id"] == "cb-1002" and attempts["cb-1002"] == 0:
        attempts["cb-1002"] += 1
        raise TransientError("ledger timed out")
    credit(conn, msg)


try:
    deliver(conn, flaky)
except TransientError:
    pass                                 # the library may surface it so the runner backs off
deliver(conn, flaky)
assert wallet(conn) == ALL_CREDITS, wallet(conn)

# 4. A malformed event goes to the dead-letter table with its error and doesn't block the rest.
conn = fresh_db()


def strict(conn, msg):
    if msg["event_id"] == "cb-1002":
        raise ValueError("paise must be a multiple of 100 for this campaign")
    credit(conn, msg)


deliver(conn, strict)
assert wallet(conn) == [("cb-1001", 5_000), ("cb-1003", 1_000)]
assert conn.execute("SELECT event_id, error FROM dead_letters").fetchall() == [
    ("cb-1002", "ValueError: paise must be a multiple of 100 for this campaign")]
assert committed_offset(conn, TOPIC) == 3

Fixed version:

# --- streamkit/consumer.py ---
import logging
import sqlite3

logger = logging.getLogger("streamkit")
logger.addHandler(logging.NullHandler())


class TransientError(Exception):
    """Raised by handlers when a retry later may succeed (timeouts, 503s)."""


def setup(conn: sqlite3.Connection, topic: str) -> None:
    conn.execute("CREATE TABLE IF NOT EXISTS offsets (topic TEXT PRIMARY KEY, next_offset INTEGER)")
    conn.execute("CREATE TABLE IF NOT EXISTS processed (event_id TEXT PRIMARY KEY)")
    conn.execute("CREATE TABLE IF NOT EXISTS dead_letters (event_id TEXT PRIMARY KEY, error TEXT)")
    conn.execute("INSERT OR IGNORE INTO offsets VALUES (?, 0)", (topic,))
    conn.commit()


def committed_offset(conn: sqlite3.Connection, topic: str) -> int:
    return conn.execute("SELECT next_offset FROM offsets WHERE topic = ?", (topic,)).fetchone()[0]


def _advance(conn: sqlite3.Connection, topic: str, msg: dict) -> None:
    conn.execute("UPDATE offsets SET next_offset = ? WHERE topic = ?", (msg["offset"] + 1, topic))


def run_batch(conn: sqlite3.Connection, topic: str, messages: list[dict], handler) -> None:
    """At-least-once delivery + idempotent effects = effectively-once.

    For each message, ONE transaction holds: the handler's writes, the dedupe marker and the offset.
    They commit together or not at all, so a crash at any point is safe to replay.
    """
    for msg in messages:
        try:
            with conn:                   # commits on success, rolls back on any exception
                seen = conn.execute("SELECT 1 FROM processed WHERE event_id = ?", (msg["event_id"],)).fetchone()
                if not seen:
                    handler(conn, msg)
                    conn.execute("INSERT INTO processed VALUES (?)", (msg["event_id"],))
                _advance(conn, topic, msg)
        except TransientError:
            # Stop here: the offset still points at this message, so it is redelivered after backoff.
            logger.warning("transient failure on %s; will retry", msg["event_id"])
            raise
        except Exception as exc:
            # Permanent failure: park it durably with the reason, then move on.
            logger.exception("dead-lettering %s", msg["event_id"])
            with conn:
                conn.execute("INSERT OR REPLACE INTO dead_letters VALUES (?, ?)",
                             (msg["event_id"], f"{type(exc).__name__}: {exc}"))
                _advance(conn, topic, msg)

Follow-ups:

1. "Walk through exactly what is in the database after test 1 runs against the original code, and why the second delivery doesn't repair it." Before any handler runs, next_offset is committed as 3. cb-1001 gets its marker committed, then its credit committed. cb-1002 gets its marker committed, then PodKilled propagates (it isn't an Exception, so the except doesn't run) and its credit never happens. The restarted pod asks for everything from offset 3, which is nothing. Even if someone rewinds the offset, cb-1002 is in processed, so it is skipped. Only cb-1001 is credited.
2. "With the fix, a single slow downstream makes TransientError fire for ten minutes. What happens to the partition, and what would you do about it?" The partition stalls on that message, by design, to preserve ordering, and consumer lag grows. That's correct for a wallet (credits for one user shouldn't reorder), but it needs limits: exponential backoff with a cap, a maximum retry count or age after which the message is dead-lettered with a "retry exhausted" reason, lag alerts per partition, and a circuit breaker on the downstream so thousands of consumers aren't hammering it. If ordering only matters per user, partitioning by user keeps one bad user from stalling everyone else's credits for long.
3. "Rohit replies that per-message transactions will be too slow at 5,000 events/s. How do you evaluate that?" Measure before redesigning: a Postgres commit with synchronous WAL flush is typically around a millisecond, so one consumer doing per-message commits tops out at a few hundred to a thousand per second; more partitions and consumers scale it horizontally. If that isn't enough, batch correctly: process N messages in one transaction with savepoints per message, so a permanent failure rolls back only that message's savepoint and is dead-lettered inside the same transaction, then commit the effects, markers and the batch's final offset together. The invariant to keep is "offset never moves ahead of committed effects", not "one commit per message".
4. "When would you recommend a team NOT use this library's in-database dedupe at all?" When the effect isn't in their database: calling a bank API, sending a push notification, writing to S3. The library can't make those atomic with its offset table. Those teams need the downstream to accept an idempotency key (for example, the event ID), or an outbox pattern: write the intent and the offset in one transaction, and have a separate relay deliver intents with retries and idempotency keys.

Scoring: Strong: Traces the kill-during-batch scenario and shows that both the early offset commit and the early marker commit lose events; states that this is at-most-once. Puts effect, marker and offset in one transaction. Distinguishes transient from permanent failures, with a durable DLQ and retry semantics, and corrects the "exactly-once" claim into a precise contract for 12 teams, including external side effects. Discusses throughput (batching with savepoints) without giving up the invariant. Acceptable: Finds the early offset commit and moves it after processing; finds the marker problem when asked "what if the handler fails after the insert?". Proposes a DLQ but doesn't separate transient from permanent errors unprompted. Red flag: Accepts "exactly-once" at face value. Suggests wrapping the loop in a bigger try/except or adding retries inside the handler as the fix. Doesn't recognise that a killed process skips except blocks. Treats logging an exception as handling it.

ST-CR-05 · A shared circuit breaker that turns one bank's outage into a payments-wide outage

Hard & Tricky · Code review · Staff / Lead (8–12 years) · circuit breakers, resilience patterns, failure isolation, bulkheads, half-open probing, thread safety, platform libraries, incident leadership · 30 min

Context:

PR #318 · resilience/breaker.py, checkout/payments.py · author: Faizal · reviewers: you (staff engineer, checkout platform)

Last month a card network's authorisation API degraded for 40 minutes. Checkout threads piled up waiting on timeouts, and UPI payments, which were healthy, failed too because the worker pool was exhausted. The post-incident review (you led it) had one action item: "fail fast on an unhealthy dependency".

Faizal's PR adds a circuit breaker to the shared resilience package (Python services across checkout, refunds and payouts will adopt it) and wires it into checkout. His description:

• After 5 failures the breaker opens and calls fail fast with CircuitOpen for 30 s.
• After 30 s it goes half-open, tries the dependency, and closes on success.
• Checkout wraps both payment rails in one breaker, so the config lives in one place.
• The clock is injectable for tests.

Checkout runs gunicorn with threaded workers (many requests in one process at once). UpstreamError is what the payment clients raise for timeouts and 5xx responses; DeclinedError is a normal business outcome (insufficient balance, wrong UPI PIN, card blocked). Review the design and the code as the person accountable for the action item actually preventing a repeat.

Code under review:

# --- resilience/breaker.py ---
import time


class CircuitOpen(Exception):
    pass


class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, reset_timeout_s: float = 30.0, clock=time.monotonic):
        self.failure_threshold = failure_threshold
        self.reset_timeout_s = reset_timeout_s
        self.clock = clock
        self.state = "closed"
        self.failures = 0
        self.opened_at = 0.0

    def call(self, fn, *args, **kwargs):
        if self.state == "open":
            if self.clock() - self.opened_at >= self.reset_timeout_s:
                self.state = "half_open"
            else:
                raise CircuitOpen()
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failures += 1
            if self.failures >= self.failure_threshold:
                self.state = "open"
                self.opened_at = self.clock()
            raise
        self.state = "closed"
        return result


# --- checkout/payments.py ---
class UpstreamError(Exception):
    """Timeout or 5xx from a payment rail."""


class DeclinedError(Exception):
    """The bank said no: insufficient funds, wrong PIN, blocked card. Not a fault."""


_breaker = CircuitBreaker(failure_threshold=5, reset_timeout_s=30.0)


def pay_by_upi(gateway, order_id: str, paise: int):
    return _breaker.call(gateway.upi_collect, order_id, paise)


def pay_by_card(gateway, order_id: str, paise: int):
    return _breaker.call(gateway.card_authorise, order_id, paise)

What a strong reviewer raises:

1. High: One breaker for both rails re-creates the incident it is meant to prevent. _breaker is a module singleton wrapping card_authorise and upi_collect. When the card network degrades, five card failures open the breaker and every UPI payment fails fast with CircuitOpen, even though UPI is healthy. Last month UPI failed because of exhausted threads; with this PR it fails by design. A circuit breaker is a bulkhead per failure domain; sharing it couples the domains. Fix: One breaker per dependency (per rail, and per issuer or endpoint where they fail independently), for example via a small registry keyed by name, with config shared through defaults rather than through a shared instance.
2. High: Business declines count as failures. except Exception counts DeclinedError: five customers in a row with insufficient balance or a mistyped UPI PIN (common at 11 pm on a sale day) open the breaker and take payments down for everyone for 30 s. Worse, the breaker opens hardest at peak traffic. Fix: Count only dependency failures: an explicit failure_types (timeouts, connection errors, 5xx, as UpstreamError) or an is_failure(exc) predicate. Business outcomes and 4xx caused by the request pass through without touching the state. Make the safe default part of the library's API, not each caller's discipline.
3. High: Failures are cumulative and never reset, so the breaker measures "five failures since the process started", not "the dependency is down". A pod that has served 10,000 good payments and 5 scattered timeouts opens its breaker on the fifth. Across 60 pods, breakers open at random through the day. Fix: Reset on success and trip on consecutive failures, or better for high traffic, on a failure rate over a rolling window with a minimum request count (for example, at least 20 calls and 50% failures in the last 10 s).
4. High: Half-open lets every caller through. Once the cool-down has passed, state = "half_open" admits every concurrent request until one completes; on a threaded worker that is the full burst of queued checkouts hitting a dependency that is barely recovering, which can knock it straight back over (and each of them waits for the full timeout if it is still down). The state is also mutated from many threads without a lock, so concurrent transitions are racy. Fix: Allow a single probe (or a small fixed number) in half-open, tracked with a probe_in_flight flag under a threading.Lock; everyone else gets CircuitOpen until the probe resolves. A failed probe reopens immediately for a full cool-down. Release the probe slot if the probe ends with a non-failure exception, or the breaker can get stuck half-open forever.
5. Medium: The action item is only partly addressed by a breaker. The incident was thread-pool exhaustion from slow calls; a breaker only helps after enough calls have failed, and each of those still waits for the timeout. Without tight client timeouts and a per-dependency concurrency limit, the first minute of the next outage looks like the last one. Fix: Pair the breaker with explicit per-call timeouts sized from the dependency's p99, a per-dependency bulkhead (a semaphore capping in-flight calls so card calls can never take more than, say, 30% of worker threads), and metrics and alerts on breaker state changes. Put breaker state on the checkout dashboard so the incident commander can see it, and test the whole thing with a fault-injection drill.
6. Low: State is per process. Every pod learns about the outage independently, so the dependency still receives pods × threshold failing calls and, after the cool-down, pods probes. Fix: Usually acceptable and simpler than shared state; say so explicitly in the design. If it matters, add jitter to the cool-down so 60 pods don't probe in the same second.

Tests that expose the bug:

# --- tests/test_resilience.py ---
import threading


class FakeClock:
    def __init__(self):
        self.now = 1_000.0

    def __call__(self):
        return self.now


def ok():
    return "ok"


def down():
    raise UpstreamError("504 from issuer")


def declined():
    raise DeclinedError("insufficient balance")


def attempt(breaker, fn):
    try:
        return breaker.call(fn)
    except (UpstreamError, DeclinedError, CircuitOpen) as exc:
        return type(exc).__name__


# 1. Five failures spread over a busy day are noise, not an outage: count CONSECUTIVE failures.
b = CircuitBreaker(failure_threshold=5, reset_timeout_s=30, clock=FakeClock())
for _ in range(4):
    attempt(b, down)
for _ in range(10_000):
    attempt(b, ok)
attempt(b, down)
assert attempt(b, ok) == "ok", "breaker opened after 5 failures in 10,005 calls"

# 2. Customers typing the wrong UPI PIN must not take payments down for everyone.
b = CircuitBreaker(failure_threshold=5, reset_timeout_s=30, clock=FakeClock())
for _ in range(50):
    assert attempt(b, declined) == "DeclinedError", "declines opened the breaker"
assert attempt(b, ok) == "ok", "business declines tripped the breaker"

# 3. A card network outage must not block UPI (the incident we are fixing).
class Gateway:
    def card_authorise(self, order_id, paise):
        raise UpstreamError("card network 503")

    def upi_collect(self, order_id, paise):
        return f"upi-ok:{order_id}"

gw = Gateway()
for i in range(20):
    try:
        pay_by_card(gw, f"ord-{i}", 49_900)
    except (UpstreamError, CircuitOpen):
        pass
assert pay_by_upi(gw, "ord-99", 49_900) == "upi-ok:ord-99", "UPI blocked by the card breaker"

# 4. Half-open lets exactly ONE probe through; concurrent callers keep failing fast.
clock = FakeClock()
b = CircuitBreaker(failure_threshold=5, reset_timeout_s=30, clock=clock)
for _ in range(5):
    attempt(b, down)
assert attempt(b, ok) == "CircuitOpen"
clock.now += 31

probe_started, release_probe = threading.Event(), threading.Event()
calls_reaching_upstream = []


def slow_recovering_upstream():
    calls_reaching_upstream.append(threading.current_thread().name)
    probe_started.set()
    release_probe.wait(1)
    return "ok"


probe = threading.Thread(target=attempt, args=(b, slow_recovering_upstream), name="probe")
probe.start()
assert probe_started.wait(5)
others = [attempt(b, slow_recovering_upstream) for _ in range(3)]   # 3 more requests arrive meanwhile
release_probe.set()
probe.join(5)
assert calls_reaching_upstream == ["probe"], f"{len(calls_reaching_upstream)} calls hit a recovering upstream"
assert others == ["CircuitOpen"] * 3
assert attempt(b, ok) == "ok"            # the probe succeeded, so the breaker closed

# 5. A failed probe reopens for a full cool-down.
clock = FakeClock()
b = CircuitBreaker(failure_threshold=5, reset_timeout_s=30, clock=clock)
for _ in range(5):
    attempt(b, down)
clock.now += 31
assert attempt(b, down) == "UpstreamError"
clock.now += 10
assert attempt(b, ok) == "CircuitOpen"

Fixed version:

# --- resilience/breaker.py ---
import threading
import time


class CircuitOpen(Exception):
    pass


class DependencyError(Exception):
    """Base for 'the dependency is unhealthy' errors (timeouts, 5xx). Clients subclass it."""


# Safe default: only infrastructure-shaped failures trip a breaker. Business outcomes never do.
DEFAULT_FAILURE_TYPES: tuple[type[BaseException], ...] = (DependencyError, TimeoutError, ConnectionError)


class CircuitBreaker:
    """Closed -> (N consecutive failures) -> open -> (cool-down) -> half-open, ONE probe -> closed/open.

    Only exceptions in `failure_types` count as dependency failures; everything else (business
    outcomes, caller bugs) passes through without affecting the state.
    """

    def __init__(self, failure_threshold: int = 5, reset_timeout_s: float = 30.0, clock=time.monotonic,
                 failure_types: tuple[type[BaseException], ...] = DEFAULT_FAILURE_TYPES):
        self.failure_threshold = failure_threshold
        self.reset_timeout_s = reset_timeout_s
        self.clock = clock
        self.failure_types = failure_types
        self._lock = threading.Lock()
        self._state = "closed"
        self._consecutive_failures = 0
        self._opened_at = 0.0
        self._probe_in_flight = False

    def _before_call(self) -> bool:
        """Decide under the lock whether this call may proceed; returns True if it is the probe."""
        with self._lock:
            if self._state == "closed":
                return False
            if self._state == "open" and self.clock() - self._opened_at >= self.reset_timeout_s:
                self._state = "half_open"
            if self._state == "half_open" and not self._probe_in_flight:
                self._probe_in_flight = True
                return True
            raise CircuitOpen()

    def _on_success(self) -> None:
        with self._lock:
            self._state = "closed"
            self._consecutive_failures = 0
            self._probe_in_flight = False

    def _on_failure(self, is_probe: bool) -> None:
        with self._lock:
            self._consecutive_failures += 1
            if is_probe or self._consecutive_failures >= self.failure_threshold:
                self._state = "open"
                self._opened_at = self.clock()
            self._probe_in_flight = False

    def call(self, fn, *args, **kwargs):
        is_probe = self._before_call()
        try:
            result = fn(*args, **kwargs)
        except self.failure_types:
            self._on_failure(is_probe)
            raise
        except BaseException:
            # Not a dependency failure (a decline, a caller bug, cancellation): release the probe slot
            # without changing the verdict, so the next caller can probe.
            if is_probe:
                with self._lock:
                    self._probe_in_flight = False
            raise
        self._on_success()
        return result


class BreakerRegistry:
    """One breaker per dependency (a bulkhead per failure domain), created on first use."""

    def __init__(self, **defaults):
        self._defaults = defaults
        self._breakers: dict[str, CircuitBreaker] = {}
        self._lock = threading.Lock()

    def get(self, name: str) -> CircuitBreaker:
        with self._lock:
            if name not in self._breakers:
                self._breakers[name] = CircuitBreaker(**self._defaults)
            return self._breakers[name]


# --- checkout/payments.py ---
class UpstreamError(DependencyError):
    """Timeout or 5xx from a payment rail."""


class DeclinedError(Exception):
    """The bank said no: insufficient funds, wrong PIN, blocked card. Not a fault."""


_breakers = BreakerRegistry(failure_threshold=5, reset_timeout_s=30.0)


def pay_by_upi(gateway, order_id: str, paise: int):
    return _breakers.get("upi").call(gateway.upi_collect, order_id, paise)


def pay_by_card(gateway, order_id: str, paise: int):
    return _breakers.get("card").call(gateway.card_authorise, order_id, paise)

Follow-ups:

1. "Faizal asks: "Why not just keep one breaker and raise the threshold to 50?" How do you respond?" A higher threshold makes the breaker slower to protect the card path and still couples UPI to card health: 50 card failures still open the breaker for UPI. The problem is the failure domain, not the number. Separate breakers per dependency, triggered by a failure rate over a window with a minimum volume, address both sensitivity and isolation.
2. "Test 4 uses threads and events. Why is a concurrency test for half-open worth its complexity in a shared library?" The half-open behaviour only matters under concurrency: in a single-threaded test, "only one probe" and "everyone probes" look identical. The events make the interleaving deterministic (the probe is provably in flight while three more calls arrive), so it is not a flaky timing test. For a library that many services will adopt, the concurrency contract is the part most worth pinning down, because each adopting team will assume it holds.
3. "Where should the breaker live in a world with a service mesh (Envoy, Istio) that already offers outlier detection?" Mesh-level outlier detection ejects unhealthy hosts of a service and works across languages, which is good for infrastructure failures. It doesn't understand business semantics: it can't distinguish a decline from a fault inside a 200 response, and it can't fail over from card to UPI or show the user "card payments are temporarily unavailable". I'd use both: mesh for host-level ejection and connection limits, and an application breaker where the fallback is a product decision. Write that division down so teams don't build a third layer.
4. "As the incident lead, how would you verify that this change actually prevents a repeat before calling the action item done?" Run a game day: in staging (then production at low traffic), inject latency and 503s into the card dependency and observe: UPI success rate stays flat, card fails fast within seconds, worker utilisation stays bounded, the breaker-state alert fires and the dashboard shows it, and recovery happens with one probe and no thundering herd. Close the action item with the results linked, and add the fault injection to a recurring resilience test so later changes can't silently undo it.

Scoring: Strong: Immediately sees that the shared breaker couples UPI to card, which reproduces the incident. Finds all three state-machine bugs: declines counted as failures, cumulative counting, half-open herd (plus missing locking). Proposes per-dependency breakers, a failure predicate, rate-based tripping and a single probe, and argues that timeouts and bulkheads are needed alongside the breaker. Talks about verifying the action item with fault injection rather than declaring it done on merge. Acceptable: Finds the shared-breaker coupling and the declines problem; fixes counting to consecutive failures. Needs a prompt ("what happens when 200 threads arrive just after 30 s?") to see the half-open herd. Red flag: Reviews the breaker as a standalone class and approves the wiring. Thinks declines "are failures too". Proposes a global breaker with a higher threshold, or suggests retries inside the breaker as the fix. Doesn't connect the PR back to what failed in the incident.