• Mixedbread Search is an API service that builds searchable multimodal indexes from files such as PDFs, images, audio, and video.
• I was curious about how the product might address two problems.
• Based on several assumptions, I wrote down how I would approach them through a few small experiments (customer-specific retrieval signals and web retrieval dependency).
• Mixedbread may already have solved these problems. Still, thinking through them was fun!

1. Introduction

Recently, I have been interested in agentic search and exploring the topic from different angles. Along the way, I came across a YouTube talk by Mixedbread AI’s Hanna Lichtenberg and Aamir Shakir titled How We Taught Agents to Use Good Retrieval.¹

I had recently been thinking about similar problems myself, so the insights from the talk immediately caught my attention. I was already familiar with the Mixedbread name from having used one of its embedding models, and the talk made me curious to learn more about the company. I started by looking through its products and blog posts.

Two questions came to mind while I was exploring the product:

1. How can retrieval quality be measured and improved for each customer?
2. How should a dependency on web retrieval be managed?

The rest of this post discusses both questions in more detail. Mixedbread may already have solutions to these problems, but my goal is simply to organize how I would approach them.

2. Understanding Mixedbread Search

2.1 Expected Product Objective

Mixedbread Search is an API-based service that creates a searchable Store from uploaded PDFs, images, documents, code, audio, and video.² It provides natural-language Search, Agentic Search, and Question Answering. Customers do not need to build their own parsing, chunking, embedding, indexing, and retrieval pipelines.

In other words, the product ultimately aims to make retrieval infrastructure something customers do not have to think about.

Based on this understanding, I expect the product objectives to be:

• Accurately retrieve the evidence a customer needs from its heterogeneous private documents.
• Manage latency, storage, and inference costs while maintaining high retrieval quality.

2.2 Mixedbread’s Technical Strengths

Retrieval Models. Mixedbread has developed Wholembed v3, a unified omnimodal, multilingual late-interaction retrieval model.³ It handles text, images, audio, and video with a single retrieval model.

Reranking. Mixedbread also provides its own listwise reranker.⁴ It ranks a candidate set as a whole and supports instruction-based ranking criteria such as relevance, freshness, and source preference.

Serving and Retrieval Infrastructure. Silo is Mixedbread’s retrieval engine for serving multimodal late-interaction retrieval at billion scale.⁵ maxsim-cpu optimizes MaxSim operations for CPU serving,⁶ while asymmetric quantization reduces storage and retrieval costs.⁷

Agentic Search. Mixedbread’s Agentic Search supports query decomposition, parallel sub-query search, wide search, and multi-round retrieval.⁸ Mixedbread also uses supervised fine-tuning and reinforcement learning to teach its search agent to use retrieval tools more effectively.¹

Benchmarks. According to Mixedbread’s published evaluations, its system ranks first on BrowseComp-Plus. It also performs strongly on multimodal and enterprise agentic retrieval benchmarks such as MADQA and OfficeQA-Pro.⁹

2.3 Inferred High-Level Architecture

The following architecture is my inference based on Mixedbread’s public materials.

Indexing Flow

Customer files
    ↓
File-type-specific parsing
(OCR, layout analysis, transcription, and media processing)
    ↓
Chunking and metadata generation
    ↓
Multimodal late-interaction encoding
(Wholembed v3)
    ↓
Customer-specific Store or index
(S3-based)

After a file is uploaded, asynchronous workers likely perform parsing, chunking, embedding, and indexing. I also expect the system to build an approximate nearest neighbor (ANN) index.

According to Mixedbread’s post on billion-scale multimodal late-interaction retrieval, S3 serves as the source of truth for index data. At query time, the required data is loaded onto local NVMe SSDs and cached in memory.⁵

Logical isolation between Stores clearly exists, but the physical sharding and tenant-isolation strategies are not publicly documented.

Search Flow

Search request
    ├─ Standard Search
    │      └─ Customer-specific index
    │         + external web API when requested
    ├─ Agentic Search
    │      └─ Agentic loop using Standard Search
    │         + query decomposition & reranking by the agent
    └─ Question Answering
           └─ Standard Search
              + answer & citation generation

Standard Search retrieves candidates from an internal index, with optional query rewriting and reranking.

Agentic Search performs multiple rounds of query planning and retrieval. Question Answering uses retrieved results as context to generate an answer with citations.

How the Web Store¹⁰ obtains its search candidates is not publicly documented. It could rely on its own selectively maintained index, external search providers, or a combination of both. One possibility is that it uses an external search provider with a strict timeout to stay within its latency budget. In the later proposal, I consider the case where an external provider is involved.

3. Two Questions

Mixedbread’s public benchmarks demonstrate that its retrieval stack performs well across general, multimodal, and enterprise workloads. However, operating the product in production raises two additional questions.

3.1 How Can Retrieval Quality Be Measured for Each Customer?

Each customer has different documents, vocabulary, query distribution, and definition of relevance. Strong performance on public benchmarks therefore does not necessarily guarantee that every customer-specific Store continues to work well on its actual workload.

This creates a measurement problem. Mixedbread may observe the search query and returned documents, but it may not directly observe whether the final user was satisfied or whether the retrieved evidence led to a correct answer.

The first question is therefore how to generate useful relevance signals (or reliable proxies) from customer-specific traffic and documents.

Such signals could support:

• Customer-specific evaluation sets
• Retrieval regression monitoring
• Failure discovery and analysis
• Selection of indexing and retrieval configurations
• Training or calibration of query rewriting and reranking components

The broader goal would be to create a feedback loop in which each Store can be evaluated and gradually improved as its documents and query distribution evolve.

3.2 How Should a Dependency on Web Retrieval Be Managed?

Mixedbread also provides a Web Store that can be searched independently or together with private Stores. Its candidate acquisition mechanism is not public, but one possible implementation is to rely partly on an external web search provider.

If such a dependency exists, several challenges arise:

• Changes to the external provider may alter which documents are available to Mixedbread’s downstream ranking system, causing retrieval quality to change even when Mixedbread’s own components remain unchanged.
• Queries that work well for private documents may not be appropriate for the web, where public terminology, freshness constraints, entity disambiguation, and source preferences may matter more.
• Repeated calls to an external provider may increase latency and API costs, particularly during multi-round agentic search.

These lead to three practical questions:

• How can Mixedbread detect whether a quality regression originates from the external provider or its own downstream ranking?
• Should private and web retrieval use different query-generation policies?
• How can repeated web retrieval be cached without returning stale evidence?

The following sections explore each of these questions through a concrete evaluation or optimization experiment.

4. Building Customer-Specific Retrieval Signals

The core challenge is obtaining a useful relevance signal for each customer’s actual workload.

The ideal signal would tell us whether the retrieved documents satisfied the user’s information need. However, Mixedbread may only observe the search query and returned documents, while the final answer and user behavior remain inside the customer’s application.

There are several possible ways to address this.

4.1 Explicit Feedback

The most reliable option is to let customers send retrieval feedback directly through an API.

For example, customers could report that:

• A result was relevant or irrelevant.
• An expected document was missing.
• A search succeeded or failed.
• A particular result was used in the final answer.

One possible way to encourage customers to provide this feedback would be to offer lower API costs, evaluation reports, or customer-specific search optimization.

This is primarily a product and incentive-design problem rather than a research problem, so I will not focus on it further here.

4.2 Implicit Signals from Query Behavior

When explicit feedback is unavailable, query sequences may provide weak signals of dissatisfaction.

Examples include:

• Repeating the same query while changing top_k or another search parameter.
• Rephrasing a query shortly after the initial search.
• Broadening or narrowing the query.
• Switching from Standard Search to Agentic Search after an initial attempt.

None of these behaviors proves that the first result was poor. However, they can help identify query-result pairs that are more likely to contain retrieval failures and are therefore worth further evaluation.

4.3 Estimating Relevance

Once potentially problematic queries have been identified, relevance labels can be created in several ways.

Human Labeling. For large customers, it may be worthwhile to label a sample of important queries manually. Human judgments provide the most reliable signal, especially in specialized domains. However, they are expensive, and external annotators may not understand customer-specific terminology or relevance criteria.

LLM-as-a-Judge. An LLM judge can evaluate whether a retrieved document is relevant to a query. This makes it possible to label a large number of query-document pairs at relatively low cost. The judge itself could be designed and calibrated using the process I described in LLM-as-a-Judge in Practice.

However, evaluating only the documents returned by the current system creates selection bias. The judge may determine whether retrieved documents are relevant, but it cannot identify relevant documents that the system never retrieved.

A Stronger Offline Search Agent. To discover documents that the production system may have missed, a more expensive search process could run offline.

Unlike the serving system, the offline process would not need to meet the same latency or cost constraints. It could use:

• More query expansions.
• A larger candidate pool.
• Multiple retrieval configurations.
• Exact keyword and semantic search.
• Full-document inspection.
• More search rounds.

The union of these results could be judged by a stronger model or verified by humans to construct a more complete set of relevant documents.

This would not create perfect ground truth, but it could provide a stronger reference than the production system alone.

4.4 From Signals to a Feedback Loop

The resulting explicit or proxy signals could be used to build a customer-specific evaluation set.

Customer queries
    ↓
Explicit feedback, behavioral signals,
LLM judgments, or stronger offline search
    ↓
Customer-specific evaluation set
    ↓
Failure discovery and regression monitoring
    ↓
Index and retrieval improvements
    ↓
Evaluation on newly collected queries

The evaluation set should be updated periodically as the customer’s documents and query distribution change.

It could then support:

• Detecting regressions after model or index updates.
• Comparing chunking and metadata strategies.
• Selecting retrieval depth and reranking configurations.
• Improving query rewriting.
• Mining hard examples for retriever or reranker training.

The important point is not only to measure customer-specific retrieval quality, but to use the measurement to keep each Store from becoming stale as its workload evolves.

4.5 A Minimal Experiment

A small initial experiment could focus on a few customers with sufficiently large query volumes.

1. Sample real search queries from each customer.
2. Identify likely failure cases using query-sequence signals.
3. Generate relevance labels using an LLM judge and a stronger offline search process.
4. Verify a small subset with human labels.
5. Use the resulting evaluation set to identify one recurring failure pattern.
6. Improve one component, such as query rewriting, metadata contextualization, or reranking.
7. Evaluate the change on held-out customer queries.

The primary metric could be customer-specific Recall@K or nDCG@K. Latency and cost should be monitored as constraints to ensure that the improvement remains practical.

5. Managing a Dependency on Web Retrieval

If Mixedbread’s Web Store feature relies on an external search API, three potential problems arise.

5.1 Robustness to Upstream Search Changes

If the Web Store relies on an external search provider, Mixedbread does not fully control which documents enter its ranking pipeline. A provider update could change the candidate set even when Mixedbread’s own system remains unchanged.

When final retrieval quality drops, there are two possible causes:

• The provider no longer retrieves the relevant documents.
• The relevant documents are still retrieved, but the downstream ranking system ranks them poorly.

Building a Web Retrieval Evaluation Set

A pooled relevance set could be built from a representative sample of real Web Store queries.

For each query, candidates could be collected from the current provider, historical snapshots, an alternative provider if available, and several query rewrites. The union would then be deduplicated and labeled for relevance using an LLM judge, with a subset manually verified.

Pooling candidates from multiple sources is important because evaluating only the current provider’s results cannot reveal relevant documents that the provider failed to retrieve entirely.

Complementary Metrics

Two metrics can then separate upstream retrieval quality from downstream ranking quality:

• Candidate Recall@K: whether the external provider includes the relevant documents in its candidate set.
• nDCG@K: whether the downstream ranking system places those relevant documents near the top of the final results.

They are complementary because nDCG alone cannot tell whether a relevant document was ranked poorly or never entered the candidate set.

Their combination also makes regressions easier to diagnose:

Candidate Recall ↓
    → Upstream provider is losing relevant documents.

Candidate Recall stable + nDCG ↓
    → Relevant documents are still available,
      but downstream ranking has degraded.

Candidate Recall stable + nDCG stable
    → The provider change has not materially affected retrieval quality.

This distinction also suggests different responses: upstream regressions may require query rewrites, deeper retrieval, or a fallback provider, while downstream regressions may call for reranker recalibration or retraining.

A Minimal Experiment

A small initial experiment could use a few hundred representative Web Store queries.

1. Sample queries from real Web Store traffic.
2. Retrieve results from the current provider.
3. Add candidates from an alternative source and several query rewrites.
4. Merge, deduplicate, and label the resulting candidate pool.
5. Save the current provider output as the baseline snapshot.
6. Re-run the same queries periodically or after a provider update.
7. Compare Candidate Recall@K and final nDCG@K against the baseline.

This would reveal not only whether web retrieval quality changed, but also whether the regression originated upstream or in Mixedbread’s downstream ranking.

5.2 Different Query-Generation Policies for Private and Web Retrieval

Queries that work well for private documents may not work equally well on the web because the two sources contain different kinds of information.

Consider the question:

Does our parental leave policy comply with current California law?

For the private Store, useful queries may focus on the company’s own terminology, such as parental leave policy or employee leave handbook.

For the Web Store, the goal is different: retrieving the current external regulation. A query such as California CFRA parental leave 2026, possibly with a preference for official government sources, may be more appropriate.

This suggests that Agentic Search may benefit from generating source-specific queries rather than applying the same rewritten query to both sources.

A Minimal Experiment

A simple experiment could compare two policies on hybrid private-and-web queries:

• Shared query policy: use the same generated query for both private and web retrieval.
• Source-specific policy: generate separate queries for the private Store and the Web Store.

The comparison could measure:

• Evidence Recall@K
• Number of search rounds
• Web search calls
• End-to-end latency

If source-specific generation retrieves the required evidence more reliably or with fewer search rounds and web calls, it would suggest that private and web retrieval should use different query-generation policies.

5.3 Freshness-Aware Caching for Web Retrieval

Repeated calls to an external web provider can add latency and API cost, especially during multi-round Agentic Search. Caching can reduce this overhead, but web results have different freshness requirements.

For example, a query about a stable historical fact may remain reusable for a long time, while a query about current news or prices may become stale within minutes.

A practical approach is to make cache lifetime depend on the expected freshness of the query:

Web query
    ↓
Freshness estimation
    ↓
Static          → long TTL
Slow-changing   → medium TTL
Time-sensitive  → short TTL
Real-time       → bypass cache

Caching could be applied at multiple levels, such as search results for repeated queries and fetched or parsed documents for repeated URLs.

A Minimal Experiment

Using historical Web Store traffic, compare:

• No cache
• Exact-query cache
• Semantic-query cache
• Freshness-aware semantic cache

Measure:

• External API call reduction
• p95 latency
• Retrieval quality
• Stale-result rate

The goal is to reduce external calls and latency without materially degrading the freshness or relevance of the retrieved evidence.

6. Conclusion

Mixedbread has strong models of its own, the infrastructure to serve them efficiently, and a well-built product that brings those capabilities together.

I thought about what signals this product would need to keep improving over time and explored ways to obtain those signals, even if only through imperfect proxies.

Of course, Mixedbread may already have solved all of these problems, and my assumptions about its system may be completely wrong.

If you have read this far, something in this post probably caught your interest. If you would like to know more about me, feel free to connect with me on LinkedIn.

References

• This post designs how the ML system from Part 1 could be implemented in production.
• It focuses on the key production architecture decisions specific to a Fast Answer system.

This is a four-part series:
Part 0. Product Teardown and Related Work
Part 1. Business Context and ML System Design
Part 3. Operations and Continuous Improvement

1. Serving Requirements

The requirements established in Part 1 can now be translated into system constraints:

• A Fast Answer must be served much faster than an LLM-generated response.
• A Fast Answer miss must not significantly increase the latency of the normal LLM response path.
• The system must support high QPS.
• A Fast Answer Service failure must not disrupt the existing ChatGPT response path.
• The QA catalog and Decision model must be updateable without disrupting serving.

2. Scale and Capacity Estimation

2.1 Online Serving Scale

As discussed in Part 1, ChatGPT processed more than 2.5 billion messages per day by mid-2025. Averaged over a full day, this corresponds to approximately 29K messages per second:

However, Fast Answers in this design are evaluated only for the first turn of a new conversation, and OpenAI does not publicly report the average number of user messages per conversation. For capacity estimation, I assume an average of three user queries per conversation. The average Fast Answer request rate is therefore approximately:

This is only the daily average. Assuming peak traffic is approximately three times the average, the Fast Answer Service must handle:

At a coarse level, this is a 20K+ QPS system. To operate safely under the estimated peak, however, the initial capacity target should be approximately 30K QPS and validated through load testing.

2.2 Latency SLO

How should the SLO for the Fast Answer Service be defined? In Part 0, I observed that Fast Answers appeared with an end-to-end TTFT of roughly 600 ms.

The service must satisfy two requirements. First, a cache hit must leave enough latency budget for the network, request routing, and client rendering to keep end-to-end TTFT near the observed value. Second, and more importantly, a cache miss should not significantly delay a request that eventually falls back to normal LLM generation.

The Fast Answer TTFT can be decomposed as:

An initial latency budget might look like this:

To keep end-to-end TTFT around the observed 600 ms, the Fast Answer Service should initially target a p99 latency of 200 ms or less. Adding more than 200 ms of p99 latency to the fallback path is therefore assumed to be undesirable, making 200 ms p99 the initial latency budget for the Fast Answer Service. The exact number is a design assumption rather than a claim about OpenAI’s actual infrastructure.

For a cache miss, the resulting latency is:

As observed in Part 0, a normally generated response had a TTFT of roughly two seconds. To bound the additional delay on this path, the Chat Service should apply a hard timeout when calling the Fast Answer Service. If no answer arrives within 200 ms, it should immediately begin LLM generation. This keeps a Fast Answer miss from adding more than approximately 200 ms to the normal generation path.

2.3 Offline Data Processing Scale

Online serving is only one side of the scale problem. Building the QA Catalog also requires an offline data-processing pipeline that can operate over historical traffic.

The Catalog Construction Pipeline designed in Part 1 extracts first-turn queries from historical chat logs, calculates query frequencies, and promotes frequent queries that are likely to be suitable for Fast Answers into QA candidates.

Under the earlier assumption of three user queries per conversation, the number of first-turn queries among 2.5 billion daily messages can be estimated as:

Suppose the initial Catalog is built from the most recent 30 days of traffic. The initial backfill job must then process approximately:

This does not mean that every one of the 25 billion queries receives LLM generation or human review. Most records are reduced through inexpensive operations near the beginning of the pipeline:

Historical Chat Logs
        ↓
First-Turn Filtering
        ↓
Query Normalization
        ↓
Distributed Aggregation
        ↓
Frequent Unique Queries
        ↓
Eligibility Filtering
        ↓
QA Candidates
        ↓
Human Review
        ↓
Answer Generation
        ↓
Human Review
        ↓
Approved QA Catalog

The early stages perform relatively simple operations such as first-turn filtering, normalization, exact-query aggregation, and frequency counting. Repeated forms of the same normalized query can be reduced to one query and its frequency count:

"What is quickselect?"
"what is quickselect"
"what is quickselect?"

        ↓

"what is quickselect" → 1,234,567 occurrences

A frequency threshold and eligibility heuristics can then reduce the candidate set before human review. Raw-log processing may operate over billions of records, but expensive model inference and human review apply only to the much smaller candidate set that remains after filtering.

The initial backfill should also be separated from recurring updates. The backfill processes billions of historical records once. Later Catalog updates do not need to rescan the complete history. An incremental pipeline can process only newly created first-turn queries and merge their counts into the existing query-frequency aggregates.

The daily incremental input is approximately:

If the system maintains a rolling window, query counts can be stored in daily partitions. Each update adds a new partition and removes an expired one, avoiding a complete rescan of the 30-day window.

The main capacity requirements for the offline pipeline are therefore:

• Initial backfill: approximately 25B first-turn query records
• Daily incremental processing: approximately 833M new first-turn queries
• Distributed aggregation over normalized queries
• Incremental updates rather than repeated full-history scans

These numbers are not estimates of OpenAI’s actual infrastructure. They are design assumptions used to determine the scale that the Catalog Construction Pipeline should support.

2.4 Human Review Capacity

Even if distributed computation can process billions of queries, the final stages of Catalog Construction face a different bottleneck: human review.

Human review cannot be scaled indefinitely by simply adding machines. The upstream candidate-generation pipeline should therefore optimize for selecting the most valuable candidates within a limited review budget, rather than producing as many candidates as possible.

Human-review throughput can be approximated as:

For example, suppose 20 reviewers each spend six hours per day actively reviewing queries, and an eligibility decision takes an average of 60 seconds:

Even when hundreds of millions of new first-turn queries arrive each day, human-review capacity may remain on the order of only thousands of items.

This means that query frequency alone may not be sufficient for candidate generation. The pipeline should therefore adjust how aggressively it filters candidates to match the available human-review capacity. The exact filtering threshold is an operational parameter that should be chosen for the current situation.

If human review becomes a bottleneck that keeps the Catalog too small to achieve a meaningful cache hit rate, the review stage could be optimized with a lightweight LLM Judge. The LLM Judge could handle straightforward cases automatically and route only difficult cases to human reviewers. I will not include this extension in the current design.

2.5 Design Targets

The assumptions above can now be summarized as the initial design targets for the rest of this post:

The online scale determines how the system must serve requests. The offline scale determines how the Catalog must be constructed. The human-review scale determines how aggressively candidate queries must be filtered and prioritized.

The rest of this post will design the serving and infrastructure architecture against these constraints.

3. Online Serving Architecture

The online serving path is designed around the constraints established in Section 2. The key design principle is to keep the Fast Answer path as simple as possible. Every additional network dependency or remote model call increases tail latency and creates another potential failure point. Since the current system uses exact-match retrieval and a lightweight Decision model, most of the serving logic can remain inside the Fast Answer Service itself.

3.1 Fast Answer Service

Figure 1. Online serving architecture of the Fast Answer system.

The Fast Answer Service itself should be stateless and deployed as a horizontally scalable service on Kubernetes. The Chat Service communicates with it through gRPC.

A stateless design allows the service to scale horizontally as traffic changes. Requests can be routed to any healthy replica without requiring session affinity, which simplifies load balancing and failure recovery.

Kubernetes is not the only deployment option. A serverless platform could provide automatic scaling with less operational overhead, while a fixed pool of virtual machines would be simpler to operate at small scale. However, the workload assumed here has both high sustained traffic and a strict p99 latency requirement. Maintaining a pool of warm replicas provides more predictable tail latency than relying on serverless instances that may experience cold starts.

I would therefore accept the additional operational complexity of Kubernetes in exchange for predictable warm capacity, horizontal scaling, rolling deployments, and controlled traffic rollout.

For communication between the Chat Service and Fast Answer Service, I would use gRPC rather than a JSON-based REST API. This is an internal, high-QPS service-to-service call where compact serialization and strongly typed interfaces are useful. The trade-off is additional schema management through Protocol Buffers and somewhat less convenient manual debugging compared with HTTP/JSON.

3.2 Retrieval Store: An Immutable In-Memory Catalog

The first version of Retrieval uses exact query matching. This makes the serving requirement substantially simpler than that of a general-purpose search system:

A conventional design might place the Catalog behind Redis or another distributed key-value store. Redis would provide low-latency lookups, replication, and independent updates to the serving data. However, every lookup would still introduce an additional network hop and another dependency on the critical path.

Instead, I would first investigate whether the entire serving Catalog can be loaded directly into the memory of each Fast Answer Service replica.

Each Catalog version would be published as an immutable artifact. When a Fast Answer Service replica starts, it downloads the selected Catalog version and loads a mapping from normalized query to QA entry into memory. Retrieval then becomes a local hash-table lookup.

Normalized Query
        ↓
In-Memory Hash Lookup
        ↓
QA Pair + Catalog Features

This design has several advantages for the current workload:

• No network call is required for retrieval.
• Exact-match lookup is extremely cheap.
• Retrieval availability is coupled only to the health of the serving process itself.
• Every request is served against a known Catalog version.

The main question is whether the Catalog is small enough to replicate in memory across all serving pods. The assumptions from Section 2 allow a rough upper-bound estimate.

The assumed human-review capacity is approximately 7,200 query candidates per day. Even if the initial Catalog construction runs for 30 days and every reviewed query is approved, the Catalog would contain at most:

entries.

The actual number would likely be smaller because some queries would fail eligibility review and some generated answers would fail answer review. For capacity planning, a Catalog of approximately 200K entries is therefore a conservative initial estimate.

Suppose each entry contains approximately:

• 100 bytes for the normalized query
• 2 KB for the prepared Fast Answer
• 500 bytes for Catalog metadata and precomputed Decision features

The serialized data is therefore roughly 2.6 KB per entry. Allowing approximately 2× overhead for strings, hash-table structures, and the in-memory representation gives a conservative estimate of roughly 5 KB per entry.

The resulting memory footprint is:

A roughly 1 GB Catalog can comfortably fit in the memory of each Fast Answer Service replica. Even allowing additional memory for the service process and Decision model, an instance with several gigabytes of memory should be sufficient.

Even if the human-review bottleneck were reduced and Catalog construction throughput improved by 10×, the same estimate would produce a Catalog of roughly 2 million entries, or about 10 GB per replica. That is still practical for a memory-optimized serving instance, so process-local replication would remain a reasonable choice at that scale.

3.3 Feature Serving

The Decision model uses two broad categories of features: Catalog-entry features and user features.

Catalog-entry features, such as historical uplift, answer age, feedback volume, and query frequency, are naturally associated with the retrieved QA entry. These features should therefore be packaged directly with the Catalog artifact.

A successful retrieval would return both the answer and its precomputed features:

Catalog Entry
  ├── Query
  ├── Answer
  ├── Historical Uplift
  ├── Query Frequency
  ├── Catalog Age
  └── Other Precomputed Features

This avoids a separate feature-store lookup for data that changes relatively slowly and can be recomputed during Catalog publication.

User-level features are different. Features such as historical Fast Answer affinity or recent regeneration behavior may need to be fetched from an online feature store.

Adding this remote call, however, directly consumes part of the 200 ms p99 latency budget and introduces another dependency on the serving path.

I would therefore make the inclusion of user features conditional on their demonstrated predictive value.

The initial Decision model would first be evaluated using only features already available in the request and Catalog entry. I would add a remote user-feature lookup only if offline and online experiments show that the additional features produce a meaningful increase in Fast Answer serve rate under the same satisfaction constraint.

3.4 Decision Model Serving

The first learned Decision model described in Part 1 does not require a large neural network. A logistic regression model, GBDT, or similarly lightweight model should be sufficient as an initial implementation.

For a model of this size, I would not deploy a separate model-serving system such as Triton or a dedicated inference microservice. Instead, the model would be loaded directly into the Fast Answer Service process and evaluated in-process.

Retrieved QA + Features
        ↓
In-Process Decision Model
        ↓
Estimated Satisfaction Impact
        ↓
Decision Threshold
        ↓
Serve / Fall Back

This removes another network hop from the critical path and allows Decision inference to remain extremely cheap.

The trade-off is tighter coupling between the Decision model and the Fast Answer Service. Updating the model requires updating the artifact loaded by the service, and the model cannot scale independently from the rest of the Fast Answer Service.

For the current system, I would accept that trade-off. The model is small, and the Catalog itself already requires periodic version updates. Catalog and Decision model versions can therefore be packaged as part of the same serving release.

A serving bundle might look like:

Fast Answer Release v42
  ├── Catalog v42
  ├── Catalog Features v42
  ├── Decision Model v17
  └── Decision Threshold v8

New pods can start with the new bundle while existing pods continue serving the previous version.

This also allows gradual rollout. Kubernetes can run both old and new replica sets simultaneously and shift traffic incrementally. If metrics regress, traffic can be shifted back to the previous version without rebuilding the Catalog or model.

3.5 Additional Considerations

Fast Answer Service replicas should be deployed in the same regions as the Chat Service, with the same immutable serving bundle replicated across regions. This avoids adding a cross-region network hop to the latency-critical path.

Special care is also required to keep query-normalization logic consistent between the online and offline paths. A Catalog key produced by the offline pipeline must be identical to the key produced from the same query during online serving; otherwise, valid entries will become false misses.

The system also requires proper observability across the Chat Service, Fast Answer Service, Catalog, Decision model, and fallback path. I will discuss the monitoring and operational details in Part 3.

4. Offline Architecture

The offline architecture has two responsibilities: constructing the QA Catalog and training the Decision model. The Catalog pipeline consumes historical chat logs to discover new QA opportunities and production feedback to identify existing entries that should be updated or deleted. Both offline pipelines process large-scale logs, but their final outputs are small enough to be packaged directly into the immutable serving bundle described in Section 3.

Figure 2. Offline architecture of the Fast Answer system.

4.1 Log Storage and Incremental Aggregation

Historical chat logs and production feedback are stored in object storage in a columnar format such as Parquet. Production feedback includes the served Catalog entry, entry-level traffic, thumbs-down feedback, and regeneration or retry behavior. Given the estimated scale of 25 billion queries for the initial 30-day backfill, Spark is a reasonable choice for distributed filtering, normalization, and aggregation.

The initial backfill should process these logs as daily partitions. This makes incremental updates to a 30-day sliding window straightforward: remove the oldest daily partition, add the newest one, and recompute the aggregate over the remaining 30 days. Each partition is processed independently, so the window can be refreshed without rescanning the complete raw history.

The pipeline produces two derived datasets. Normalized query counts feed the path that discovers new Catalog entries, while entry-level feedback aggregates feed the path that proposes updates and deletions. This avoids rescanning approximately 25 billion raw records whenever the Catalog is rebuilt. The expensive raw-log processing happens only once per daily partition, while subsequent Catalog builds operate on much smaller aggregated representations.

I prefer this approach over maintaining a single continuously updated global frequency table. Daily aggregates can remain immutable, which makes backfills and recovery simpler. If processing for one day is incorrect, that partition can be recomputed independently and the 30-day aggregate rebuilt from the corrected result.

This composition works naturally for additive statistics such as query counts and feedback counts. More complex statistics may require additional mergeable state. Query frequency is the primary signal for discovering new Catalog entries, while production feedback provides the primary signals for updating or deleting existing entries.

4.2 Candidate Generation and Human Review

Both offline paths produce candidates for human review. As estimated in Section 2, the assumed review capacity is approximately 7,200 candidates per day, so each path must reduce its aggregated data to a small set of high-value candidates.

Add Candidates

The 30-day query aggregation still produces far more queries than humans can review. The initial system can keep candidate generation simple: queries already covered by the Catalog are removed, basic eligibility rules filter obvious non-candidates, and the remaining queries are ranked primarily by traffic frequency.

30-Day Query Frequency
        ↓
Remove Existing Catalog Entries
        ↓
Eligibility Rules
        ↓
Rank by Expected Coverage Gain
        ↓
Top Candidates within Review Budget

Update and Delete Candidates

The second path aggregates production feedback for existing Catalog entries. Entries with unusually negative feedback, increasing regeneration or retry behavior, or freshness concerns become update candidates. Entries with persistently low traffic or severe quality problems can become delete candidates.

These rules do not mutate the Catalog directly. They produce a prioritized set of update and delete candidates that joins the add candidates at the human-review stage.

At this point, the problem is no longer a distributed-compute bottleneck. Spark may process hundreds of millions of records upstream, but the combined output of both paths should be only thousands of review candidates per day.

The selected candidates can then be written to PostgreSQL, which serves as the backend for the internal review workflow. A relational database is appropriate here because the workload is relatively small and the important requirements are transactional state transitions and auditability rather than large-scale analytical processing.

The review workflow depends on the proposed action:

PENDING_REVIEW
  ├── ADD_APPROVED → ANSWER_GENERATING → ANSWER_REVIEW
  ├── UPDATE_APPROVED → ANSWER_GENERATING → ANSWER_REVIEW
  └── DELETE_APPROVED → DELETED

PostgreSQL also becomes the source of truth for the Catalog management workflow. The online serving system does not query this database directly.

4.3 Catalog Mutation and Answer Generation

Approved add candidates proceed to answer generation. Update candidates may also require a newly generated answer, while approved delete candidates can be removed without generation. Because the upstream filtering stage reduces hundreds of millions of daily records to at most thousands of reviewed candidates, answer generation does not require a large dedicated serving system.

A durable task queue and a small pool of stateless generation workers should be sufficient. Each job contains the approved query or existing Catalog entry together with its generation configuration, and the result is written back to PostgreSQL for final human review.

The queue provides retry and failure isolation without coupling answer generation to the review application. Generation throughput can also be scaled independently if review capacity increases.

Once an answer for an add or update candidate passes review, the approved QA pair is written to the Catalog source of truth. An approved deletion marks the entry for exclusion from the next Catalog snapshot. The Catalog table would contain the information required to construct the serving artifact, including:

Catalog Entry
  ├── Normalized Query
  ├── Fast Answer
  ├── Query Frequency
  ├── Historical Feedback
  ├── Catalog Age
  ├── Human Review Metadata
  └── Precomputed Decision Features

The Catalog itself remains mutable in PostgreSQL as entries are added, updated, or removed. Online serving, however, consumes only immutable snapshots built from an approved Catalog state.

4.4 Decision Model Training Pipeline

The Decision model is trained from the randomized experiment data described in Part 1. The raw experiment logs are stored in the same object-storage-based data platform used by the Catalog pipeline.

The scale requirements of data preparation and model training are different. Experiment logs may contain hundreds of millions of requests, so feature generation and training-dataset construction should use Spark. The actual Decision model, however, is expected to be a logistic regression, GBDT, or similarly lightweight model.

Distributed computation should handle data preparation, while the model itself can be trained on a single CPU machine using a library such as scikit-learn or XGBoost. There is little benefit in introducing distributed model-training infrastructure for a model this small.

After training, candidate thresholds are evaluated on randomized holdout data using the off-policy evaluation procedure described in Part 1:

The output of the training pipeline is therefore not only a model artifact but also the selected Decision threshold.

Decision Model v17
Decision Threshold v8

Both are versioned independently so that a threshold can be changed without necessarily retraining the model.

4.5 Building and Publishing the Serving Bundle

The Catalog and Decision pipelines eventually converge into a single release artifact.

Approved Catalog
      ↓
Catalog Snapshot
      │
      ├──────────────┐
      │              │
Decision Model       │
Decision Threshold   │
      │              │
      └──────┬───────┘
             ↓
      Release Build
             ↓
Fast Answer Release v42
  ├── Catalog v42
  ├── Catalog Features v42
  ├── Decision Model v17
  ├── Decision Threshold v8
  └── Normalization Version v7
             ↓
Versioned Object Storage
             ↓
Kubernetes Rollout

The normalization version is included because offline Catalog construction and online Retrieval must produce exactly the same Catalog key for the same query. A normalization change must therefore be deployed together with a compatible Catalog snapshot.

The release artifact is immutable and stored in versioned object storage. When a new Fast Answer Service pod starts, it downloads the configured release, validates the artifact, loads the Catalog and Decision model into memory, and becomes ready to receive traffic only after initialization succeeds.

This allows old and new releases to coexist during a rollout. If the new release causes a regression, traffic can be moved back to pods running the previous immutable bundle.

This design deliberately moves complexity away from the online critical path. Catalog management, large-scale aggregation, model training, validation, and artifact construction all happen offline. The resulting online system only needs to load a verified bundle and execute local retrieval and lightweight Decision inference.

5. Failure Handling and Capacity Validation

The Fast Answer path must remain optional from the Chat Service’s perspective. Each request receives a 200 ms hard timeout. A Fast Answer Service miss, error, or timeout causes the Chat Service to fall back to normal LLM generation, so a Fast Answer failure cannot disrupt the existing response path.

A circuit breaker provides protection against persistent failures. If the Fast Answer Service remains unhealthy, the Chat Service should temporarily stop calling it and route requests directly to LLM generation. On the serving side, a new pod should pass readiness checks only after it has downloaded, loaded, and validated its Catalog and Decision model artifacts.

Capacity should be determined through load testing rather than by guessing a replica count. The sustainable QPS of one fully initialized pod should be measured under a realistic request mix and verified against the 200 ms p99 latency budget. The required replica count can then be calculated as:

The final load test should verify the approximately 30K QPS peak target from Section 2 with enough headroom for traffic spikes, pod failures, and rolling deployments.

6. Wrap-up

This post was intended as an architectural overview rather than a complete production design document. My goal was to identify the major infrastructure decisions that are specific to this system and reason about them from the scale and latency assumptions established earlier.

As a result, I intentionally left out general-purpose infrastructure concerns such as authentication, network configuration, access control, and other platform-level details that would be required in a real production system but are not particularly specific to Fast Answers.

There are also important questions around observability, deployment, experimentation, and ongoing operations that I have only briefly touched on here. These become especially important once the system is running in production and the Catalog, Decision model, and user traffic continuously change.

I will cover those topics in Part 3, focusing on how I would monitor, operate, and continuously improve the system after launch.

If you have read this far, something in this post probably caught your interest. If you would like to know more about me, feel free to connect with me on LinkedIn.

• Part 1 turns the product observations from Part 0 into a business objective and an ML system design.
• This post is less about sophisticated model architectures and more about how to translate a business objective into concrete ML problems.

This is a four-part series:
Part 0. Product Teardown and Related Work
Part 2. Production Architecture
Part 3. Operations and Continuous Improvement

1. Business Context & Objective

1.1 Business Objective

Part 0 examined Fast answers from the outside: observable product behavior, related products, and relevant research. From this point on, I will imagine the reasoning that might have justified this project when OpenAI first began considering it internally. This may differ substantially from OpenAI’s actual thought process, but I wrote it for the fun of designing one product end to end.

A reasonable business objective for introducing Fast answers would be:

Reduce inference cost and latency without degrading user satisfaction.

This section briefly justifies that objective, then defines the domain assumptions and metrics used throughout the rest of the design.

1.2 Traffic Scale and Cacheable Opportunity

OpenAI’s internal data analysts and scientists were probably already studying the wide range of user queries. They would be able to estimate how often recurring information-seeking queries appear each day, what share of them are duplicates, and how many could be answered with a cached response. A question such as Why is the sky blue? has a stable answer over time, so a cached response may be sufficient.

ChatGPT operates at enormous scale. OpenAI’s report How People Use ChatGPT states that the product received 2.5 billion messages per day in 2025, with 24% classified as Seeking Information.¹ This design applies Fast Answers only to the first query in a new chat, so the exact addressable traffic is unknown, but the scale is still informative.

The next question is how much of that traffic is repetitive and cacheable. Query frequencies in web search are often highly skewed, suggesting that a relatively small set of frequent queries may account for a meaningful share of traffic. ChatGPT may follow a different distribution, so applying a specific rule such as 80/20 would be questionable. Still, even 1% of the 600 million daily information-seeking messages would correspond to 6 million requests per day.

This motivates the working assumption that a relatively small prepared-answer catalog could cover a non-trivial share of eligible first-turn queries. Each successful cache hit replaces an otherwise necessary LLM generation, creating meaningful opportunities to reduce both inference cost and latency.

1.3 Estimating Potential Savings

A simple general expression for the resulting savings is:

• $N_{\text{new chats}}$ is the unknown number of new conversations per day.
• $P_{\text{info}}$ is the unknown probability that the first query is information-seeking.
• $P_{\text{eligible}\mid\text{info}}$ is the unknown probability that an information-seeking first query is eligible for a Fast answer.
• $\text{HitRate}$ is determined by the system discussed in this series.
• $C_{\text{generation}}$ is the unknown internal generation cost.
• $C_{\text{fast}}$ is the much smaller lookup and serving cost.

This expression does not include the increase in user satisfaction from lower latency or the additional value created by freeing GPU capacity.

1.4 Domain Assumptions

• A meaningful portion of traffic is information-seeking and cacheable.
• Fast answers are applied only to the first query in a new chat, since a subsequent query implicitly demands context from the prior conversation.
• Returning a cached answer is always cheaper than generating a new answer.
• Some users may not be satisfied with a cached answer.
• The initial system supports English queries only.

The decisions in the rest of this design will be based on these assumptions.

1.5 Metrics

Primary metrics

• Inference cost reduction
• Latency reduction

Guardrail metrics

• User satisfaction, measured through negative feedback such as thumbs-down
• Regeneration or retry rate²

2. Problem Formulation & ML Objective

2.1 From the Business Objective to Fast Answer Serve Volume

The business objective can be written as:

This objective means that the Fast Answer system $\pi$ should maximize cost and latency savings while ensuring that user satisfaction is not lower than when users receive LLM-generated answers. The $\epsilon$ term accounts for noise in the observed satisfaction metric.

As assumed above:

Based on the domain assumptions above, serving an answer through the Fast Answer path always saves generation cost and latency. The cost and latency savings from each Fast Answer are therefore always positive. Let the combined saving per served Fast Answer be approximated as a constant $K > 0$. Then:

Total saving is therefore approximately proportional to the number of Fast Answers served. The business objective can be approximated as:

In plain language:

Serve as many requests as possible with Fast Answers without degrading user satisfaction.

If cached answers could satisfy users unconditionally, always serving a Fast Answer would preserve satisfaction and maximize savings. In reality, Fast Answers cannot be served for every request:

• Catalog failure: no satisfactory, fresh, reusable answer exists.
• Retrieval failure: such an answer exists, but the system does not retrieve it.
• Decision fallback: the right answer is retrieved, but the policy falls back to LLM generation.

The objective therefore remains maximizing the number of Fast Answers served subject to the satisfaction constraint. The serving volume is determined by how many opportunities the Catalog creates, how many Retrieval finds, and how many the Decision Policy converts into Fast Answers. Satisfaction remains a separate aggregate constraint on the overall serving policy.

2.2 Decomposing Fast Answer Serve Volume

The next step is to decompose how the Fast Answer serve volume is produced. Ideally, the objective would be optimized end to end. However, dividing the problem into subproblems based on the funnel and failure cases implied by the domain knowledge above makes it much easier to solve. Although not optimizing the objective directly may introduce some loss, each individual problem can be solved more easily and accurately, potentially producing better overall performance. This type of staged approach is common in search and recommendation systems.

At least three events must occur in sequence:

Under this sequential decomposition, the expected Fast Answer serve volume can be expressed as:

From this decomposition, each of the three modules (Catalog, Retrieval, and Decision Policy) should have its own objective and metrics.

2.3 Module-Level Objectives and Metrics

Module 1 — Catalog Construction

Question

Do we have a satisfactory reusable answer for this query?

Input → Output

Historical query traffic → Prepared QA catalog

Objective

Maximize catalog coverage: the fraction of eligible traffic for which a satisfactory reusable answer exists.

Metrics

• Primary: Traffic-weighted Coverage
• Guardrails: Answer Quality, Freshness, Safety

Module 2 — Retrieval

Question

If a reusable answer exists, can we find the right one?

Input → Output

User query + QA catalog → Best candidate QA pair

Objective

Maximize the retrieval success rate, given that an appropriate answer exists.

Metrics

• Primary: Top-1 Compatibility or Precision@1
• Secondary: Recall@K, False-match Rate

Module 3 — Decision Policy

Question

If we found the right answer, should we actually serve it?

Input → Output

User query + retrieved QA + available context → Serve Fast Answer / Fall back to LLM

Objective

The Decision Policy should maximize the Fast Answer serve rate while maintaining the satisfaction guardrail:

Here, $\Delta S$ represents the change in user satisfaction under the Fast Answer policy relative to always using LLM generation.

This may look similar to the global business objective, but it operates only after the upstream Catalog and Retrieval stages have succeeded. Under the earlier assumption that each Fast Answer produces approximately the same cost and latency saving, the remaining Decision problem reduces to identifying the requests for which that saving can be realized with an acceptable impact on user satisfaction.

This objective can be further reduced to a prediction problem. Let $x$ represent the information available to the Decision Policy, such as the user query, retrieved answer, and available user or contextual features. Let

denote the policy, where $\pi(x)=1$ means serving the Fast Answer and $\pi(x)=0$ means falling back to LLM generation.

For each request, define the expected satisfaction impact of serving the Fast Answer instead of the LLM as

The main modeling challenge is that $\tau(x)$ is not directly observable for an individual request, since the same request cannot simultaneously receive both a Fast Answer and an LLM response. Section 6 discusses how randomized experimental data can be used to estimate it.

The Fast Answer serve rate is then

Because requests with $\pi(X)=0$ still receive the baseline LLM response, the aggregate satisfaction change introduced by the policy is

Therefore, the Decision Policy objective becomes

Using Lagrangian relaxation, this can be written as

Rearranging,

For a fixed $\lambda$, the optimal policy is therefore

The original constrained optimization problem can therefore be implemented by estimating $\tau(x)$, the expected satisfaction impact of serving a Fast Answer instead of an LLM response, and choosing a threshold that maximizes the serve rate while satisfying the aggregate satisfaction constraint.

The Lagrange multiplier $\lambda$ controls the trade-off between serving more Fast Answers and preserving satisfaction. In practice, rather than explicitly optimizing $\lambda$, I would sweep the decision threshold on held-out experimental data and choose the lowest threshold whose estimated satisfaction delta still satisfies the guardrail. This yields the highest Fast Answer serve rate among policies that meet the satisfaction constraint.

Metrics

• Primary: Fast Answer Serve Rate
• Guardrail: Satisfaction Delta

2.4 Connecting Module Objectives to Business Value

The complete relationship can be summarized with one equation:

and:

The logic is therefore:

This explains why optimizing each module’s local objective should improve the global business objective.

The reason to define this framework before designing the individual modules is that it provides debuggability and prioritization. If the complete system underperforms, module-level objectives and metrics make it possible to isolate the root cause and decide where further investment will have the greatest impact.

Additionally, each module does not need to be a fancy deep-learning model. For each module, define what goes in, what comes out, what should be optimized, and how success is measured.

3. High-Level Architecture

3.1 High-Level Architecture

Figure 1. High-level architecture of the Fast Answer system.

The high-level architecture is divided into two paths. The first is the online path that handles user requests. When a request reaches the Chat Service, it calls the Fast Answer Service first for the initial query in a chat. If the Fast Answer Service returns an answer, the Chat Service immediately sends it back to the user. Otherwise, it invokes LLM generation.

Within the Fast Answer Service, the Retrieval Layer first searches a prebuilt index for a candidate QA pair. The Decision Layer then determines whether that candidate should be shown. It combines the candidate QA pair with context from the online feature store to make the final decision. Each stage must also log its outputs asynchronously so that the data can later support analysis and model training.

The offline path uses accumulated logs to build the QA catalog. An automated pipeline extracts candidate QA pairs, which pass through a review process before entering the final catalog. This process may add new QA pairs or update and remove existing ones. A periodic job then builds the retrieval index from the catalog. The Decision Layer should also be continuously retrained on newly collected data and updated when a new model is ready.

A more complete system would require additional discussion of retrieval-model training, feature-store pipelines, deployment, A/B testing, and other production concerns. The diagram shows only the components most central to the ML design. The technical implementation and operational details will be covered in later posts.

3.2 Heuristic-Based V0

Suppose the first Fast Answer system is implemented and launched from this architecture. Because the Catalog, Retrieval, and Decision layers are modularized, the work can be divided across the available team. If I had to build it alone, however, the first version would exclude complex ML components and focus on getting the complete system running. Section 2 defines each module’s objective as an ML problem, but the launch version does not necessarily need an ML model.

For example, the Catalog module could use exact matching to find high-frequency queries, generate LLM answers for them, and send those answers through manual review. Retrieval could use an index built for exact matching. After upstream eligibility checks pass, an experiment assignment layer could randomly assign a small fraction of traffic to either a Fast Answer or LLM generation. None of these approaches is sophisticated, but together they provide a set of heuristics that should behave reasonably well for an initial launch.

3.3 What Should We Improve Next?

Suppose the system is now running with heuristic-based modules. The next step is to experiment on a small fraction of traffic, verify that the system and business metrics are healthy, and confirm that the required logs are being collected correctly. Once those checks pass, each module can be improved against the objectives defined earlier.

If resources allow only one module to be improved first, I would choose the Decision Layer. The Catalog should already work reasonably well because every entry passes through manual review, even though that review is expensive. Exact-match Retrieval is deliberately high precision. The largest uncertainty immediately after launch is therefore: Should this retrieved answer actually be shown? This decision sits closest to the satisfaction guardrail. Because the V0 experiment assignment is randomized at the chat-session level, it also provides controlled comparison data for learning a policy.

Using this randomized data to train a learned Decision Layer would make it possible to incorporate user context and make better serving decisions. For example, the system could show more Fast Answers to users who are usually satisfied with them.

The Catalog would be my second priority. Automating part of the manual-review process is likely to have high ROI. It would reduce review costs, and the work required for automation would force the team to define the intended behavior of Fast Answers more precisely. The knowledge and data accumulated during V0 operation would make that definition easier.

Retrieval would come last. Once the Catalog work has clarified the product objective, the system could begin moving from exact matching toward semantic matching. This ordering follows from the example in Part 0, where a general-purpose text embedder was not aligned with answer equivalence.

The remainder of this post focuses on the modeling design for the three components, with particular emphasis on the Decision Policy as the first learned optimization target. Even when the Catalog or Retrieval modules remain heuristic-based, the discussion will cover what data should be used to evaluate and launch them. It will also briefly introduce possible approaches for more advanced modeling.

4. Catalog

The Catalog system is essentially a system that extracts information satisfying a particular set of conditions from chat logs offline and stores only a selected subset. Those conditions may require a query to occur frequently, ask for information, support a stable answer, and satisfy other product requirements.

This is a well-known industry pattern. Structurally, this resembles offline indexing and knowledge-base construction systems: extract information useful for a specific serving objective, filter it, and publish it in a retrievable form. Examples include search indexing and ChatGPT’s recently introduced improved Memory feature.³

4.1 Catalog Lifecycle

As discussed above, I would begin with a heuristic-based system. The full lifecycle would look like this:

Figure 2. Build the catalog, evaluate it on recent data, publish it, collect online feedback, and repeat.

The sequence is straightforward. Build the catalog from a slightly older data window, evaluate it offline on recent data, publish it, collect feedback from production, and rebuild it with new data. The iteration cadence depends on how frequently the query distribution changes. It would be better to iterate more frequently at the beginning. Once the catalog has converged to some degree, a longer iteration cycle should be acceptable.

4.2 Heuristic-Based Catalog Build

Catalog construction has three paths: adding new QA pairs, deleting existing QA pairs, and updating existing answers.

Figure 3. The three paths in the heuristic-based catalog construction system.

Adding New QA Pairs

The first path adds a new QA pair through three stages:

1. Query extraction and manual review
2. Answer generation and manual review
3. Catalog publishing

Query extraction begins by pulling a list of first-turn queries from chat logs. The system counts them using exact matching and selects the top $N$ most frequent queries. A human then filters this list for information-seeking queries.

Next, a separate LLM generates an answer for each approved query. Reusing an answer from existing chat logs would be risky for two reasons. First, the answer may contain personalized information because the model may have used memory. Second, a Fast Answer may require its own target length and tone.

Generating a fresh answer is therefore a reasonable choice. The generated answer also passes through manual review. An approved QA pair is then published to the catalog.

Deleting QA Pairs

The second path removes QA pairs. Many heuristic rules are possible, but two signals are especially important:

• Poor user feedback
• Low hit rate

An entry with consistently poor user feedback can be sent through manual review and deleted. An entry with sufficiently low traffic can be removed automatically when catalog maintenance or review cost matters.

Updating QA Pairs

The third path updates QA pairs. The clearest reason to update an entry is that its answer has become outdated, but automatically detecting this is difficult. A simple solution is to assign a TTL to each answer and review it when the TTL expires. User feedback can also trigger manual review and an answer update.

Updating a question does not need a separate path. In practice, changing a question is equivalent to deleting the old entry and adding a new one.

The design above assumes that every manual review is performed by a person. An LLM Judge could make this process faster, but simply prompting an LLM would not be sufficient. The process described in LLM-as-a-Judge in Practice provides a better way to build one, so I will not repeat that discussion here.

4.3 Offline Evaluation

Even a heuristic-based catalog requires evaluation before deployment. As discussed in Section 2, the Catalog’s role is to maximize coverage of QA pairs eligible for Fast Answers. The evaluation split should therefore be time-based: build the catalog from historical data, then measure traffic-weighted catalog coverage on more recent data.

The metric can be written as:

Here, $n_q$ is the number of times query $q$ appears in the recent evaluation window.

Answer quality, freshness, and safety are the guardrail metrics. The catalog should be independently re-audited on the evaluation date rather than evaluated using the same labels that were used to approve entries. The resulting audit can measure a pass rate for each dimension:

The audit labels could use a Likert scale, but I would recommend binary labels. The reason is discussed in LLM-as-a-Judge in Practice.

This produces a heuristic-based catalog system that can still be evaluated with data and deployed with a reasonable degree of confidence.

4.4 Online Evaluation

The online evaluation of the Catalog asks one question:

Is the published catalog creating the opportunities we expected under actual live traffic?

The first metric is Online Catalog Coverage:

This checks whether the coverage measured offline on a historical window is maintained under actual production traffic.

The second metric is the coverage gain from a newly published catalog:

This measures how much additional traffic the new catalog covers relative to the previous version. It also shows whether newly added QA pairs are actually receiving traffic. If many entries are rarely used, the Catalog Construction process may be inefficient.

Finally, downstream feedback should be aggregated at the catalog-entry level. For every catalog answer that is actually served, useful signals include:

• Thumbs-down feedback
• Regeneration or retry

These signals can trigger the review, update, or deletion paths described above.

4.5 Advanced Directions

A more advanced catalog system should be able to answer the following questions:

• Can query extraction become more robust than simple frequency ranking?
• How can answer generation use user feedback to update answers automatically and improve user satisfaction?
• How far can manual review be reduced? Can the LLM Judge become reliable enough that humans review only hard samples?
• How frequently should the catalog be updated to return stable answers?

5. Retrieval

Retrieval deliberately uses exact matching to keep the system complexity low. When a query arrives, the system looks it up in the catalog. If the exact same query exists, it retrieves the corresponding answer.

From the perspective of query matching, this provides effectively perfect precision. That does not mean the retrieved answer will always satisfy the user. Answer satisfaction still depends on the quality of the QA curation performed by the Catalog system.

Because exact-match Retrieval introduces no additional modeling uncertainty, I would not build a separate ML evaluation pipeline for Retrieval in V0. Basic correctness and index-consistency checks would still be required.

If the system is later extended to retrieve answers for semantically similar queries, see the Retrieval discussion in Part 0.

6. Decision Layer

6.1 Collecting Training Data

The first problem is that the expected satisfaction impact defined in Section 2 cannot be observed directly for an individual request:

The same request cannot simultaneously receive both a Fast Answer and an LLM response. V0 should therefore randomize traffic only after it passes the upstream eligibility checks:

Retrieved Valid QA
        ↓
Eligibility Gate
        ↓
Experiment Assignment
   ├── Fast Answer
   └── LLM

For each assigned request, the system should collect:

• $x$: the query, retrieved QA pair, catalog metadata, and available user or contextual features
• Treatment assignment: Fast Answer or LLM
• Satisfaction outcomes, including thumbs-down and regeneration or retry
• A composite satisfaction label, if needed

Randomization makes the two groups comparable and provides the counterfactual evidence needed to estimate the satisfaction impact of Fast Answers.

Even after the Decision Policy is upgraded from V0 to V1, a small fraction of eligible traffic should remain randomized. This continuously provides unbiased data that can be used for future model training and evaluation.

6.2 Estimating Satisfaction Impact

The simplest approach is to fit two outcome models (T-learner). Let $T$ denote the randomized treatment assignment:

The estimated satisfaction impact is then:

This is essentially an uplift estimation problem. There is no need to begin with a complex model. Logistic regression, a GBDT, or a small MLP would all be reasonable candidates.

The training pipeline may be sophisticated, but online inference must remain cheap. The Decision Layer sits on the latency-critical path:

If $L_{\text{Decision}}$ becomes large, the Decision Layer undermines the latency advantage that gives Fast Answers their value.

6.3 Features

The first model would not use query or answer text directly. Supporting text inputs would introduce additional complexity, including offline and online embedding pipelines. Simple aggregation-based features are a better starting point.

Whenever possible, features should be available without an additional remote call. The first version should rely primarily on query, answer, catalog, and user information that is already available or retrieved as part of an existing call.

Catalog entry features

• Historical uplift for each query
• Answer length
• Query frequency
• Catalog entry age
• Time since the last human review
• Historical feedback volume
• Historical satisfaction variance
• Answer update count

User features

• Historical Fast Answer negative-feedback rate
• Historical LLM negative-feedback rate
• Smoothed user-level uplift
• Randomized-history sample count
• Average number of queries per conversation
• Language or locale
• Account tenure
• Recent Fast Answer feedback
• Recent regeneration behavior

Fast Answer affinity should be measured relative to the user’s LLM baseline rather than from Fast Answer feedback alone:

The two historical outcome rates, their smoothed difference, and the randomized-history sample count provide the model with both the estimated effect and the amount of evidence behind it.

Average queries per conversation. This may indicate how the user typically uses ChatGPT. A low value may describe someone who usually asks a short information-seeking question and leaves, making that user more likely to be receptive to Fast Answers.

User features may be unavailable for new or cold-start users. If they do not provide a sufficiently strong predictive signal, it may be better to exclude them and keep the first model simpler.

6.4 Offline Evaluation and Threshold Selection

Offline evaluation should operationalize the constrained objective from Section 2. For each decision threshold $t$, evaluate:

Because the holdout data comes from a randomized experiment, I would evaluate each candidate policy using an unbiased off-policy estimator, such as IPW or a doubly robust estimator, rather than relying only on its predicted uplift.

The final threshold is selected by sweeping $t$ on held-out randomized experimental data:

This constrained policy metric is more important than a model metric such as AUROC. Model A with an AUROC of 0.80 may be preferable to Model B with an AUROC of 0.82 if Model A achieves a higher serve rate under the same satisfaction guardrail.

Offline evaluation should also benchmark system constraints, particularly model-inference latency and feature-fetch latency.

6.5 Online Evaluation

The final evaluation is an A/B test on eligible traffic:

• Control: eligible traffic receives LLM generation.
• Treatment: eligible traffic is handled by the learned Decision Policy.

Primary metrics

• Fast Answer Serve Rate
• Inference cost reduction
• Latency reduction

Guardrail metrics

• Satisfaction Delta
• Thumbs-down rate
• Regeneration or retry rate

The learned policy should increase Fast Answer serve volume and reduce cost and latency while remaining within the satisfaction guardrail.

7. Wrap-up

This post does not include fancy machine-learning models. Instead, it focuses on understanding the business context, translating it into an optimization problem, decomposing that problem under a few reasonable assumptions, and connecting each component to the ML problem it needs to solve. Once the problem has been formalized this way, improving each module with more sophisticated modeling becomes a separate, later problem.

Implementation, serving, and operations have also influenced parts of this design. Although I intentionally kept this post focused on ML system design, a real ML system cannot be designed without considering how it will be implemented and operated. Part 2 will discuss how to serve the ML models designed here, and Part 3 will cover operations and continuous improvement in more detail.

If you have read this far, something in this post probably caught your interest. If you would like to know more about me, feel free to connect with me on LinkedIn.

References

• ChatGPT has a feature called Fast answers.¹ It appears to serve ready-made answers to frequently asked, information-seeking questions.
• What if I were the person responsible for launching it? For fun, I worked backward from the product and wrote a hypothetical system design.

This is a four-part series. Part 0 covers my understanding of the product and a survey of the related literature. The rest of the system design continues in the following posts:
Part 1. Business Context and ML System Design
Part 2. Production Architecture
Part 3. Operations and Continuous Improvement

1. ChatGPT Does Not Always Generate a New Response

One day, I was solving a LeetCode problem and wanted to refresh my memory on quickselect. ChatGPT has become my go-to knowledge hub, so I naturally typed quickselect into it.

This time, though, the answer came back surprisingly fast and was labeled Fast answer.

Figure 1. My first encounter with a Fast answer in ChatGPT.

It was the first time I had seen the feature, and I was curious enough to try the same query again. I opened another chat, typed quickselect, and got the same answer.

That was unusual. An LLM will generally produce somewhat different responses even when it receives the same prompt twice. This time, the wording, structure, and examples were identical. My immediate guess was that ChatGPT was returning a stored query-answer pair instead of generating a fresh response. Query-answer caching would be a reasonable way to reduce both latency and inference cost for common questions with stable answers.

That led me to a more interesting question: if I were building and launching this feature, how would I design it?

What follows is a hypothetical design based on public information, a few product experiments, and assumptions I made myself. I have no inside knowledge of how Fast answers is actually implemented.

2. Product Observation

A quick search did not reveal much, but the feature does appear in the ChatGPT release notes.¹

Figure 2. OpenAI’s release note introducing Fast answers.

The note says Fast answers are intended for common information-seeking questions, such as “Show me the Seven Wonders of the World” or “Which football team has the most Super Bowl titles?”² ChatGPT may respond faster when two conditions hold:

1. The question does not require a personalized response.
2. ChatGPT has a high-confidence answer ready.

It also says that Fast answers do not reference the user’s past chats or memory.

The release note does not explain what “has a high-confidence answer ready” means technically. My working hypothesis is that ChatGPT maintains a collection of pre-generated answers for common, straightforward questions and returns one when an incoming query matches with sufficient confidence. That is the assumption I use throughout this series.

A few small experiments helped clarify the product surface before considering the system behind it. These were informal observations rather than a controlled evaluation, and the behavior may change as the feature evolves.

2.1 The Same Query Returns the Same Answer

Several frequently searched topics triggered Fast answers, including:

• quickselect
• why is the sky blue
• what is ROAS

For these experiments, Fast answers were enabled under Settings → Personalization, with Intelligence set to Instant. With this setup, each query produced a Fast answer. Repeating a query in a new chat returned the same response.

Figure 3. Fast answers for why is the sky blue and what is ROAS.

Below a Fast answer, ChatGPT shows a lightning-bolt icon. Clicking it lets the user regenerate the answer immediately or describe in natural language how it should be regenerated.

This button is probably one of the main channels for collecting user feedback. The system design later in this series will discuss how feedback from this surface could be used.

Figure 4. The Fast answer menu offers regeneration and a free-form instruction field.

2.2 The Same Query Does Not Always Trigger It

Even the exact same query did not always trigger a Fast answer. In the screenshot below, ChatGPT generated a response to quickselect and began answering in Korean. I normally use ChatGPT in Korean, so my memory or conversation history was probably included in the context for this response.

Figure 5. The same quickselect query did not trigger a Fast answer this time.

There are several possible reasons why the feature triggers inconsistently. OpenAI may still be running conversation-level A/B tests. There may be a heuristic that treats repeated submissions of the same query as dissatisfaction with the Fast answer and switches back to generation. Or some hidden logic may decide whether to show a Fast answer for each user based on personalized signals. This last possibility will be discussed in more detail in the system design.

Regardless of the reason, randomized triggering may actually be the better launch strategy. Rather than always showing the cached result for the same query, it may be better to randomize whether the user receives the cached answer. The resulting data—including user reactions and inference-cost savings—could inform the next product decision. Randomized data would also reduce bias in later analysis and make counterfactual analysis more credible. The system design will cover this in more detail.

2.3 Semantically Similar Queries Return Different Answers

At first, I thought the feature might use semantic search to retrieve answers for similar queries. For example, quickselect and what is quickselect can both be satisfied by a single answer explaining the concept. It would be reasonable to prepare one answer and return it for semantically similar user queries.

Figure 6. quickselect and what is quickselect trigger different Fast answers.

The actual behavior was different. As the screenshots show, even semantically equivalent queries returned different answers. This suggests that the current Fast answer system may store one answer per query and determine cache hits through exact query matching.

There are two tradeoffs to consider:

1. Maintain one polished answer, which makes answer-quality management easier, but requires solving semantic cache hit and miss decisions accurately.
2. Use exact matching for queries, which keeps the lookup logic simple but increases the number of query-answer pairs to manage.

From the perspective of launching and operating an initial system, the second approach may have been simpler. It makes sense if the product decision is that user disappointment from a false cache hit and a low-quality answer is more costly than the LLM generation cost of a false cache miss. The system design will examine this tradeoff in more detail.

2.4 It Appears to Support Multiple Languages

The feature also triggered outside English, suggesting that eligibility is not limited to one language. As in Section 2.3, two questions with similar meanings received different answers. The screenshots below ask for the definition of ROAS using two different Korean expressions, and the responses are different as well. This supports the exact-query-matching hypothesis regardless of language.

Figure 7. Two Korean queries asking for the definition of ROAS return different Fast answers.

This was interesting because limiting an initial launch to English would normally be safer. Cached answers need to be reviewed for quality before they are served. Multilingual review adds outsourcing, coordination, and operational costs, so supporting a language with relatively few global speakers, such as Korean, would not be easy.³

This suggests two possibilities. First, OpenAI may believe that the cost savings from multilingual Fast answers are large enough to justify the additional review cost. Second, it may have enough confidence in an automated quality-review system to operate the feature across languages.

2.5 Fast Answers Have a Faster TTFT

Browser developer tools showed a clear difference in how quickly the two response types began arriving. Using time to first response byte as a proxy for TTFT, Fast answers took roughly 400–600 ms, while normally generated answers took roughly 1.8–2.2 seconds.

Figure 8. The Fast answer began arriving after about 559 ms.

Figure 9. The normally generated answer began arriving after about 1.81 seconds.

Interestingly, Fast answers were still delivered as a stream, even though the answers appeared to have been prepared in advance. These screenshots were taken in July 2026. When the feature first launched two months earlier, I remember the entire answer arriving more quickly without visible streaming. From a UX perspective, the response appeared to fill in immediately, which made the feature feel extremely fast.

Something may have changed in the meantime. There are at least three possible explanations:

1. The pre-generated-answer hypothesis is wrong, and Fast answers are actually generated by a very lightweight model.
2. The answers are pre-generated, but OpenAI switched to streaming to keep the UX consistent with normal responses and may be A/B testing that presentation.
3. The behavior changed for some other reason that cannot be observed from the outside.

The goal of this series is not to reverse-engineer the feature exactly. It is to develop my own system design from the product behavior I can observe. I will therefore continue with the assumption that Fast answers return pre-generated responses.

2.6 It May Trigger in the Middle of a Conversation

I expected this feature to apply only to the first query in a conversation. According to OpenAI’s study of how people use ChatGPT, 24% of messages were classified as Seeking Information as of July 2025—in other words, use that closely resembles search.⁴ Information-seeking traffic should contain frequently repeated queries, so I thought this feature was introduced to reduce LLM inference cost by caching answers to those queries.

However, a comment in a Reddit discussion about Fast answers complains that a keyword triggered a Fast answer in the middle of a multi-turn conversation. In that case, ChatGPT ignored the context accumulated so far and returned an answer that had already been prepared.

That behavior was unexpected. I do not know whether it is intended or a bug. In my design, I would probably allow Fast answers only for the first query. Once a conversation becomes multi-turn, the user is more likely to care about the context built up so far, which makes it much harder for a context-free Fast answer to satisfy them.

3. Related Products and Literature

3.1 Direct Answers Before LLMs

Fast answers may look like a new LLM product feature, but search engines have long faced a similar problem. Google Featured Snippets extract an answer-like passage from a webpage and place it above the ordinary search results.

Figure 10. A Google Featured Snippet answering the query “Why is the sky blue.”

The difference between Google and ChatGPT is that Google has to select good answers from external content in advance, while ChatGPT has to select good answers from responses generated internally.

Google’s blog post suggests three insights.

First, Google must have invested substantial human-rater effort in evaluating search quality. The post discusses cases in which quality became a problem and shares a 182-page document called the Search Quality Rater Guidelines. I expect that ChatGPT made a similar effort to ensure that Fast answers are trustworthy.

Second, retrieval and the serving decision are separate problems. Storing and retrieving a good snippet is one problem; deciding whether to show it to the user is another. Google did not show a snippet when its authority, quality, or compatibility with the query was insufficient. Similarly, I will describe Fast answers as a three-stage system: curation, retrieval, and policy.

Third, Google worked on cases in which queries were lexically similar but had different semantic intent. As mentioned above, its approach may have been to keep the number of snippets small to reduce management costs while solving the semantic query-matching problem. ChatGPT, however, provides internally generated content, which should make that content easier to generate and manage than Google’s external content. With that in mind, Fast answers may not have needed semantic query matching in its initial version.

3.2 Managing QA Catalog

While I could not find prior work that exactly matches the Fast answers setting, there is a substantial body of work on automatically building and maintaining FAQ-style knowledge bases from historical user interactions.⁵

The common idea is to mine recurring information needs from historical user queries or conversations, group semantically similar questions, and turn them into reusable question-answer pairs. These pairs can then form a prepared-answer catalog that a separate retrieval system may consult when a similar request arrives in the future.

One particularly relevant example is AI Knowledge Assist. This paper follows the common approach described above and provides a concrete example of how such a system can be built.

Figure 11. Overview of AI Knowledge Assist. Source: Figure 2 in Laskar et al. (2025), licensed under CC BY 4.0.

The paper assumes that many companies want to build conversational AI chatbots or RAG systems but do not have company-specific knowledge bases. However, if a company has customer service chat data, QA data is already embedded in the agents’ answers to customer questions. The paper therefore proposes a way to extract and clean this data and keep it updated over time. This is similar to ChatGPT’s situation: user chat logs already exist, and the task is to extract a QA set from them. However, the same question may have different answers in ChatGPT depending on each user’s context, so those answers cannot be used directly. The data could at least be used to identify recurring questions.

The paper uses LLMs in most stages. It first uses an LLM to extract reusable QA pairs from raw conversation data. Rather than extracting arbitrary pairs, it applies the following conditions: information-seeking, non-personalized, no PII, not time-sensitive, universal, and useful. It then clusters the QA pairs by question similarity and uses an LLM to select a representative QA pair from each cluster. A pair is either added automatically or sent for human review. The paper also mentions an automatic update mechanism that uses question similarity and answer similarity for future catalog updates.

In my view, there are three major points to consider when building a catalog system.

First, QA pair filtering has to be done well. It effectively determines the final quality of the product. A clear policy is needed to define which kinds of questions align with the product’s goals. This is not something that can be solved simply by having a product owner write down a constitution-like set of rules. The initial policy will most likely contain many vague areas, creating conflicts no matter how closely a human rater or an LLM rater tries to follow it. The policy therefore has to improve through multiple iterations. A previous post from my time at Hyperconnect discusses this process in more detail.⁶

Second, the decision to add a QA pair to the catalog will also likely be automated with an ML model, but designing that decision policy is not easy. For example, when setting a confidence cutoff, several variables have to be considered: whether the confidence comes from a calibrated model, how the cutoff affects human-rater costs, and how much total traffic the system receives. The problem is to find a Pareto-optimal point among these variables.

Third, the system has to support continuous updates. Even seemingly stable catalogs can become stale over time, and the distribution of what counts as a good QA pair is likely to shift. The system should therefore be designed from the beginning to be robust to this shift. For example, as user-feedback data accumulates, the system should automatically adapt to distribution shifts so that its performance is maintained or improves over time.

3.3 Retrieval

Once a good QA catalog has been built, the system has to retrieve the right entry for each user query. Although this system design assumes exact query matching to reduce system complexity, it is still worth identifying and comparing the other available options.

Information retrieval in modern ML systems is already a well-studied problem. Several papers have also explored it specifically in the FAQ retrieval domain.⁷ Each presents a novel idea, but the core modeling decisions come down to two questions:

1. How should relevance between a user query and a QA pair be defined?
2. Should the retrieval target consider only the question, or the answer as well?

As with any ML modeling problem, defining the objective is the most important step. For Fast answers, the retrieval objective could be aligned directly with the product objective: maximizing user satisfaction with the answer. However, user satisfaction can be sparse, noisy, and difficult to measure. In that case, an LLM-based prediction of user satisfaction could be used instead. An answerability score from a domain expert could also serve as a training proxy. Whatever signal is chosen, it should be quantifiable and aligned with the product objective, and the model should be trained to optimize it.

Table 1. Cosine similarities measured with BAAI/bge-small-en-v1.5.

The important point is that an off-the-shelf pretrained sentence embedding is unlikely to be aligned with the Fast answers objective. A general sentence embedding model considers the first query pair more similar than the second. However, the first pair requires different answers, while the second pair shares the same answer. Query similarity alone is therefore not necessarily aligned with answer equivalence. Fine-tuning for the actual objective is likely to be essential.

3.4 Final Decision Layer

So far, the discussion has focused on which QA pairs should be included in the candidate set and how to select the best one among them. A production-level system has to consider one more step: the decision policy that determines whether the selected QA pair should actually be shown to the user. In other words, the system has to answer this question automatically: For this user, in this situation, is showing a Fast answer better than normal generation? The need for this layer will be covered in the next system design post. For now, it is useful to examine how the QA domain has approached this problem.

vCache: Verified Semantic Prompt Caching⁸ is the paper most directly related to Fast answers. When a user query and a cached query are similar in embedding space, the cached response is reused. Rather than using one fixed similarity threshold, vCache assigns a separate threshold to each cached entry and learns those thresholds online to maximize cache hits while satisfying a user-specified global error-rate constraint. When the system is uncertain, it explores by invoking the underlying model, collects additional correctness observations, and uses them for online threshold learning. This is similar to the approach I would take for Fast answers. In my case, the objective would be to minimize LLM cost while maintaining user satisfaction rather than maximizing cache hits under a global error-rate constraint, and the technical details of learning the policy would also differ.

Selective Question Answering under Domain Shift⁹ uses gating to answer as many questions as possible while maintaining accuracy above a specified level. Its objective is therefore to maximize coverage subject to a target-accuracy constraint. Because naively using confidence can fail due to calibration problems, the paper trains a calibrator, a type of meta-model, and argues that it remains robust under domain shift. The objective is different, but the approach is similar in that it performs constrained optimization.

Generate-then-Retrieve: Intent-Aware FAQ Retrieval in Product Search¹⁰ takes the unusual approach of applying gating before retrieval. This is effective when a large amount of pruning can be done at the user-query level. Although the mechanism is different, exact matching in my design plays a similarly aggressive filtering role: only a narrow subset of queries is allowed to enter the Fast-answer path.

There are many possible decision policies. However, this layer can be made arbitrarily sophisticated, so the design here will keep only the essential pieces and focus on making the system extensible later. Whether a particular method matters should be tested and decided based on the product context. The more important point is to recognize the need for a decision layer and introduce the concept early in the product launch.

4. From Observation to System Design

The rest of this series uses three working assumptions based on the product observations and related work discussed so far.

First, Fast answers are served from a catalog of pre-generated question-answer pairs rather than generated from scratch for every request. Second, to keep the initial system simple and minimize incorrect matches, I will assume that retrieval is based primarily on exact query matching. Third, retrieving a valid QA pair does not necessarily mean that it should be shown. A separate decision layer determines whether serving the Fast answer is preferable to normal LLM generation for a given user and context.

These assumptions are not claims about how ChatGPT is actually implemented. They are design choices I would make if I were responsible for launching the product. With this foundation in place, the next post moves from observation to system design.

If you have read this far, something in this post probably caught your interest. If you would like to know more about me, feel free to connect with me on LinkedIn.

References

What I want to discuss here is not how to prompt an LLM Judge. It is how to treat an LLM Judge as an ML system component and build an evaluation loop that lets us continuously optimize a system in the direction we actually want.

You Can’t Improve What You Can’t Measure

It is a well-known saying. If we cannot measure which version of a system is better, we cannot know what to improve. On the other hand, if we can measure the direction we want with a reasonably quantitative signal, we can use that signal to train models, compare experiments, analyze failures, and continuously improve the system.

A similar idea comes up frequently in AI today: if a problem is verifiable, it can eventually be optimized through reinforcement learning or a related method.³ The reward does not have to be perfect. If we can produce a signal that is sufficiently related to the direction we want, we can begin an optimization loop.

For generative features or search results whose quality is difficult to evaluate programmatically, it is natural to consider LLM-as-a-Judge. For example, a chatbot team may want to evaluate the helpfulness, conciseness, or correctness of a response. A search team may want to measure query-document relevance. In the past, humans labeled these subjective qualities directly. Now, part of that work can be delegated to an LLM Judge.

But introducing an LLM Judge immediately requires caution. Just because an LLM Judge produces a number does not mean that increasing that number will produce the system we actually want. If we optimize the Judge score aggressively but the real user experience or business outcome does not improve at all, we have not optimized the system well. We may simply have optimized the wrong objective very efficiently.

Before building an LLM Judge, it is therefore worth taking one step back.

Do You Really Need an LLM Judge?

First, ask whether there are any programmatic or directly observable signals that can evaluate the quality you care about.

Consider approaches such as RLVR, or Reinforcement Learning with Verifiable Rewards. In coding tasks, we can programmatically verify whether the final answer is correct, whether the code runs, whether it passes tests, and whether it satisfies the required format.

These signals do not perfectly express every kind of code quality we care about. Passing tests does not necessarily mean that code is readable or maintainable. In other words, these signals are incomplete with respect to the quality we ultimately want. But they are still highly reliable for an important part of that quality.

To simplify the idea, let $U$ be the user utility we ultimately want to optimize and $M$ be a measurable proxy signal. We do not need:

In practice, finding such a perfect metric is difficult.

What matters instead is whether the following relationship generally holds:

The first question is whether an imperfect signal is still useful enough to move the system in the right direction.

For a consumer product, when evaluation is difficult, a useful starting point is the business objective: what does the product ultimately need to accomplish? From there, work backward through the user journey or funnel to identify measurable signals.

Suppose we are building an AI chatbot for customer support. Its ultimate business objective might be:

In other words, the goal is to reduce support costs while keeping customer satisfaction above an acceptable level.

What does reducing support costs mean in practice?

One proxy is whether the AI chatbot resolves the user’s problem before escalating it to a human agent. We can measure the escalation rate. But a lack of escalation does not necessarily mean the problem was resolved. The user may simply have given up.

We can also ask whether the user reported that the problem was resolved after the conversation. We might examine whether the user contacts support again about the same issue within a certain period, or whether the ticket is reopened.

The problem can be decomposed in the following direction:

Each signal is clearly noisy and suboptimal. But taken together, several signals may cover a substantial portion of the original objective.

Before building an LLM Judge, ask:

Can programmatic or directly observable signals already capture enough of the objective we want to achieve?

If they can, those signals are the better starting point.

“It’s Hard to Eval” Is a Product Smell

If, after all this, we still cannot find any way to evaluate whether the product is working, it is worth asking whether something else is wrong. To borrow Hamel Husain’s phrase, “It’s Hard to Eval” Is a Product Smell.⁴

If the goal of a product is to satisfy customers but there is no way at all to determine whether customers are satisfied, we may not have defined clearly enough what we are trying to satisfy.

Of course, some final business outcomes appear only after several months. Causal attribution can be difficult, and signals can be extremely sparse. This is not an argument that every product has a perfect and immediate metric.

But a product ultimately solves some problem for a user. There must be a process through which the user moves from having that problem to having it resolved. If we understand the product and the user’s problem well enough, we should be able to break that process into smaller steps or define intermediate outcomes that provide some way to check whether the system is moving in the right direction.

If even that is difficult, evaluation may not be the only problem. We may need to revisit whether the product objective is clear enough, or whether the product itself is designed so that the user’s problem-solving process can be observed and verified.

So, When Should You Use an LLM Judge?

After thinking through the questions above, we may find several quantitative signals but still feel that they are too noisy or fail to cover an important part of the product objective. That is when an LLM Judge becomes worth considering. An LLM Judge can also be useful before a product has launched, when real user signals are not yet available.

A problem is a good candidate for an LLM Judge when it satisfies roughly the following conditions.

• Measuring this quality would help improve the system.
• The quality is subjective and difficult to calculate directly.
• A sufficiently trained human can still judge it relatively consistently.
• Repeating that judgment at scale would make it useful in a real optimization loop.

Consider query-document relevance in search. Relevance is difficult to calculate through simple string overlap. But in most cases, a domain expert who reads both the query and the document can judge whether the document is relevant to the query. This is a good candidate for an LLM Judge.

An LLM Judge is not a tool for solving problems where even humans do not know what good looks like. It is more appropriate for:

Problems that humans can judge, but where human judgment is difficult to scale.

What Makes a Good Judge?

What conditions should an LLM Judge satisfy once it has been built? There are at least three.

• It must produce a quantifiable signal that can be used to compare systems and optimize them.
• It must agree sufficiently with the judgment of the human experts best qualified to assess that quality.
• Improving the resulting offline signal should move the real user signal or business outcome in the same direction.

This reveals two separate alignment problems. The first is the alignment of the Judge itself:

Here, $J$ is the judgment produced by the LLM Judge and $H$ is the judgment of a human expert.

But this alone is not enough. Ultimately, we also need:

Here, $U$ is actual user utility or the product objective.

Even if an LLM Judge achieves 95% agreement with human experts, we may still be measuring the wrong objective very precisely if the human experts’ criteria are not aligned with real user outcomes.

It is therefore important to separate the following two relationships:

and:

The first is a Judge-calibration problem. The second asks whether the evaluation objective itself is correct.

Align Humans Before Aligning the Judge

People are often less aligned on subjective problems than we expect. Before building an LLM Judge, it is therefore useful to align the humans first.

Collect enough real system outputs, then ask the relevant domain experts, PMs, or even MLEs to label them independently. It is important to record not only a simple Pass or Fail, but also the reasoning or critique behind each judgment.

When the results are compared, there will probably be more disagreement than expected at first. These disagreements are the most valuable samples. They identify the current gray decision boundary of the product.

Comparing critiques and discussing why the judgments differ reveals implicit assumptions that the existing rubric did not specify.

The team can then evaluate the outputs again with the updated rubric. As this process repeats, disagreement gradually decreases and human judgments begin to converge.

Importantly, this process updates more than the rubric document. It also updates the team’s implicit understanding of the product. The decision boundary becomes clearer:

What counts as a good output for this product?

What is still acceptable, and where does it become unacceptable?

As a result, the team develops domain experts who can judge the relevant product quality with reasonable consistency.

There is also a useful side effect. Repeating this process naturally creates a human-labeled dataset.

Iterate and Update the Rubric

A rubric cannot be made perfect from the beginning. It is more natural for the rubric to emerge from real outputs and failures.

Evaluate outputs with the initial rubric, find disagreements, compare critiques, and update the rubric. Then evaluate new samples again.

After enough iterations, major changes to the rubric become less frequent and human agreement stabilizes. Only then can we say that “what our team considers a good output” has become reasonably explicit.

The dataset accumulated by this point can later serve as a calibration dataset for evaluating and improving the LLM Judge.

Make Evaluation as Simple as Possible

To make human alignment easier, the evaluation itself should be kept as simple as possible.

When possible, binary evaluation is a useful starting point. Using a Likert scale from 1 to 5 requires answers to questions such as:

What exactly is the difference between a 3 and a 4?

Is a 4 good enough to ship to production?

If one person gives the same output a 3 and another gives it a 4, is that truly a disagreement?

Defining the meaning of every tick becomes another difficult problem.

Binary evaluation, in contrast, focuses on a single decision boundary:

Does this output satisfy the standard we want?

Or, in more product-oriented language:

Would you ship this output?

Even if each sample is evaluated in binary terms, aggregating those decisions produces a quantitative metric:

This metric can compare model A with model B or detect regressions.

Of course, binary evaluation compresses a great deal of information. The tradeoff can be framed this way:

• Binary for optimization
• Critiques for diagnosis

Pass or Fail supports quantitative comparison and optimization. Detailed critiques explain why the system failed and help identify the next improvement.

Treat the Judge as a Model, Not an Oracle

Once the rubric has stabilized and a human-labeled dataset exists, we can build the LLM Judge.

A reasonable starting point is to place the rubric in the prompt and provide examples and critiques written by human experts as few-shot examples.

But the important point is not to trust the LLM Judge as an oracle. The Judge is still a model. We can represent the human-labeled dataset as:

Here, $x_i$ is the output being evaluated, $y_i$ is the human Pass or Fail label, and $c_i$ is the human critique.

For the same input, the LLM Judge produces:

We can begin by comparing:

This measures agreement between the Judge and the human.

But aggregate agreement is not enough. We need to inspect the samples that humans labeled Fail but the Judge labeled Pass, as well as those that humans labeled Pass but the Judge labeled Fail. Then we compare the Judge’s reasoning with the human critique. Analyzing these disagreements reveals where the Judge misunderstood the rubric, missed a condition, or lacked sufficient examples.

The interesting part is that this process has almost the same structure as the one above. First, we analyzed disagreements between:

to improve the rubric.

Now, we analyze disagreements between:

to improve the prompt and the Judge.

In short:

Human-human disagreement $\rightarrow$ rubric iteration

Human-Judge disagreement $\rightarrow$ Judge iteration

By repeating this process, the Judge can gradually align with the decision boundary the team has established.

Optimize the System

Only now are we ready to use the offline evaluation signal produced by the LLM Judge to improve the actual system. We might change the model, improve the retrieval system, or modify the training data.

Each time, we use the same evaluation set and Judge to compare the previous system with the new one.

If the Judge score improves, we move to the next iteration.

In this way, the LLM Judge is not merely a tool that evaluates whether a model is good. It becomes a measurement component inside the system-optimization loop.

Close the Loop with Online Evaluation

But the process cannot end there. As discussed earlier, alignment between the LLM Judge and human experts does not guarantee that the signal is aligned with actual user utility.

Once the system is deployed and user signals become available, we need to examine the relationship between offline evaluation and online outcomes.

Then we ask the most important question:

• Did the model we judged to be better offline actually improve user satisfaction?
• When the search-relevance Judge improved, did engagement or task success improve as well?
• When the chatbot-resolution Judge improved, did the real escalation rate or repeat-contact rate improve?

If not, improving the Judge itself may not be the priority. We need to return to the beginning and revisit whether the business objective, proxy signals, and human rubric were defined correctly. Then, if necessary, we start the iteration again.

Iterate Fast

Finally, all of these iterations should happen quickly and frequently. Do not try to create a perfect objective, a perfect rubric, or a perfect Judge from the start. This is such a familiar principle that I will not spend more time explaining it here.

Closing the Loop

Ultimately, when I use LLM-as-a-Judge, the Judge itself is not the most important part.

First, understand the business and the product, and define what we are ultimately trying to optimize.

Next, find measurable signals that serve as useful proxies for that objective. Prefer programmatic or directly observable signals when they are available.

If those signals cannot capture an important quality, and the remaining quality is subjective but trained humans can judge it relatively consistently, then consider an LLM Judge.

Before building the Judge, align the humans. Use human disagreements to discover the decision boundary and improve the rubric. Use the resulting human-labeled dataset to align the Judge with the humans.

Then use the Judge’s offline evaluation signal to optimize the system.

Finally, once the system meets real users, verify that the offline signal is aligned with the online product outcome.

To me, LLM-as-a-Judge is less a tool that scores outputs instead of a human and more an ML system component that makes part of an otherwise subjective and difficult-to-optimize product objective measurable.

Only when we continue iterating while checking that this measurement remains aligned with the direction we actually want can an LLM Judge play a meaningful role in continuously improving a system.

If you have read this far, something in this post probably caught your interest. If you would like to know more about me, feel free to connect with me on LinkedIn.

References

• I built a data preprocessing agent with my teammates Max, Sean, and Shreya.
• We built a benchmark suite called KaggleBench. The novel idea is to evaluate agents on high-quality, human-expert data preprocessing workflows selected from Kaggle, rather than synthetic data.
• We built an agent with a workflow and tools tailored specifically to this task. We also built the agentic loop from scratch to understand the internals of AI agents.
• It was fun!

You can take a look at the details of the benchmark and the agent!

How it started

In CS 498 AI Agents in the Wild, we worked on a project where we designed both an AI agent and a benchmark from scratch.

We spent the most time choosing the topic. Since we were going to spend a semester on it, we wanted the result to be meaningful. Even though this was a student project funded out of pocket, we wanted the outcome to be high quality enough to be proud of.

So we made decisions using three rules.

1. The topic should only be solvable with an agentic system. If a rule-based system or traditional machine learning could solve the problem well enough, we should avoid it.
2. The topic should be measurable and quantifiable. We needed a dataset and a metric that could be justified reasonably. This matters because we wanted to show that our system works better than a simple baseline or prior work, and more importantly, that we can evaluate the system in a way that makes sense. If we cannot measure progress, we cannot improve the system in a disciplined way.¹
3. It should be possible to frame the topic as a problem with real industrial interest. The topic itself did not have to be industry-specific, but the core idea had to matter in industry. A research assistant agent maps naturally to information retrieval and summarization from user queries. A multi-turn customer service agent maps to chatting under policy constraints while retrieving information. These patterns are useful to many companies.

After comparing several ideas by pros, cons, feasibility, and viability, we decided to study whether an agent can solve data preprocessing tasks. The topic fit our rules. First, it is essentially a coding problem, so it is hard to solve without an agent. Second, by using Kaggle, we thought we could turn it into a quasi-quantifiable evaluation problem. Third, data science agents are already a visible industrial use case. OpenAI has written about its internal data agent, and Databricks also documents a data science agent workflow in notebooks.²³

That is how we started working on the problem.

KaggleBench

For any system, defining quantitative evaluation first is important. Many LLM-based features or agent projects end with qualitative demos. The generated text looks impressive, and the project stops at subjective impressions. That is fun, but it does not create a path for improvement. Without a measurement setup, it is hard to know whether the next version is actually better.

At first glance, data preprocessing sounds like a small subset of a data science agent. But if you think about it carefully, it is not that different from a coding or math-solving problem. A data science project starts from ambiguous context, an imperfectly specified objective, and an open-ended search space. There is often no single perfect answer. Given how much money is being invested into coding agents, it would be unrealistic for a student team to solve the fully open-ended version in one semester.

So we had to scope the problem down. We reframed it as a multiple-choice decision problem. Given a Kaggle competition, its dataset, a set of candidate preprocessing actions, and a set of actions already taken, the agent has to decide which action should be added or removed.

This framing makes evaluation much easier because the problem is no longer fully open-ended. The hard part becomes generating good candidate actions. A good candidate action has to be clearly right or clearly wrong. It should not depend too much on subjective judgment.

For that, we used human-expert Kaggle notebooks as the source of truth. The details of how we built the dataset and evaluation suite are in the KaggleBench post.

Data preprocessing agent from scratch

One constraint from the class was that we could not use a well-known agent SDK such as LangGraph. That was a useful constraint. At a high level, an agent can sound simple. You call an LLM in a loop, parse tool calls, execute tools, and feed the result back into the next step.

The real difficulty is not the loop itself. The important parts are the harness around it and the durable execution behavior. The agent needs the right tools, the right prompt structure, the right state representation, and a way to keep making progress without losing track of the task.

There are already many emerging practices around agent harness design. We selected a few that fit our setting and built the architecture around them. We also implemented the agentic loop ourselves so that we could understand what is happening inside the system instead of treating an SDK as a black box.

We evaluated the agent against several baselines, including rule-based heuristics and a naive agentic loop. The baselines were designed so that our final method could be understood as a combination of smaller design choices. That made the comparison work like an ablation study. We also compared against Claude Code to see how our task-specific agent performed against a commercial general-purpose coding agent.

The detailed implementation and results are in the agent architecture post.

What I Learned

I got two main things from this project.

First, I learned a lot about agent architecture. Building an agent from scratch gave me a clearer big-picture understanding and more confidence. Of course, what we built was still an application-layer system that combined existing components. Each layer hides a lot of depth. Frontier labs know how to train LLMs with reinforcement learning for agentic use cases. LLM API providers do serious engineering around caching and request orchestration to use GPU resources efficiently. Agent harnesses for long-running tasks need many layers of reliability engineering. This project gave me a practical taste of what has to be considered at the application layer to build an end product.

Second, I got much better at using agents. Codex played a huge role in the development process. Because of it, I could spend more time on planning and decisions, which raised the quality of the final project. But it was not enough to simply hand off requests. I had to check persistently whether the thing I wanted had actually happened. That check could not only be conversational. I needed mechanical checks, tests, or direct inspection of outputs.

That process helped me feel the limitations of coding agents in a way that metrics alone cannot show. It was also interesting to experience model improvements during the project, from Codex 5.3 to 5.5.

Reference

Coming soon… You can take a look at the details of the benchmark and the agent!

• I built a data pipeline that stores public events from Hacker News.
• The first goal was data analysis, but I also wanted a reusable event backbone for future HN projects, including recommendation systems.
• Tech Stack: EC2, SQS, Lambda, DynamoDB, Kinesis Data Firehose, S3, Athena, CloudWatch, Terraform, and Docker.

The data collected by this pipeline was used in When Does a Post Go Viral on Hacker News?.

Why I Built This

Hacker News already has a public API.¹ It exposes a useful view of the current HN ecosystem, including item details, user profiles, recently changed items and profiles, and the current Top Stories list.

That is enough for many applications. If I want to show the current title, score, and comments for a story, the API is fine. For the analysis I had in mind, though, the current state was not enough. I was interested in the dynamics of the system. I wanted to know what path a post took before it reached the Top 10, how its upvotes accumulated, when comments arrived, when it moved between pages, and whether author-level signals were useful later.

Those questions need snapshots over time. A final item record does not tell me how the item got there.

So I decided to build my own pipeline. Part of the motivation was practical. I needed the data for analysis. I also wanted to practice building something closer to a real data pipeline than a local scraper.

For Hacker News scale, the simplest system might have been one EC2 instance that polls the API and writes files directly to disk or S3. That probably would have been cheaper and easier. I intentionally chose a more overbuilt design because I wanted to practice the shape of a larger production data system. That meant queues, serverless workers, latest-state storage, append-only archives, CDC, monitoring, and infrastructure-as-code.

That made the project more complicated than it strictly needed to be. The extra complexity gave me a reusable base for later HN work. Once I have item snapshots, ranking events, and user snapshots in a consistent format, I can attach other pipelines on top for causal analysis, recommender experiments, author features, topic modeling, or personalized ranking.

Hacker News API Overview

The HN API is small, but it has the pieces needed to observe the public surface of the site.

The first entity is an item. In HN, stories, comments, jobs, polls, and poll options are all items. They live under this endpoint.

/v0/item/<id>.json

An item can have fields like id, type, by, time, title, url, text, score, kids, descendants, dead, and deleted. This is the main entity for the item pipeline.

The second entity is a user.

/v0/user/<id>.json

A user record contains fields like id, created, karma, about, and submitted. I used this for author context and future recommender features.

The API also exposes live-ish change surfaces.

/v0/updates.json
/v0/topstories.json

updates.json returns recently changed item IDs and profile IDs. I used it as the source for the item and user pipelines. topstories.json returns the current Top Stories ranking list, up to 500 items. I used it to build ranking snapshots and ranking events.

Item Pipeline

Figure 1. Item pipeline from HN updates to raw snapshots and derived change events.

The item pipeline is the main pipeline.

It starts with an EC2 poller. The poller reads updates.json, looks at the items array, and detects item IDs that were not in the previous poll. It sends those IDs to SQS.

The next step is a Lambda fetcher. The fetcher reads item IDs from SQS, calls /v0/item/<id>.json, computes a deterministic hash of the HN payload, and writes the latest item state to DynamoDB only if the payload changed. When it writes a changed item, it also sends the raw item snapshot to Firehose, which writes compressed JSONL files into S3.

DynamoDB Streams are enabled on the latest-items table. A second Lambda receives old and new images from the stream, compares the item field, and emits field-level change events to another Firehose stream. Those events also land in S3.

So the item pipeline has two S3 outputs.

raw item snapshots
item change events

This split was deliberate. The raw snapshot is the safer source of truth. The event stream is a derived convenience layer. If I later decide that an event definition was wrong, I can recompute it from raw snapshots.

The advantage of this design is that each piece has a simple job. The poller detects changes. SQS absorbs bursts. Lambda fetches item details. DynamoDB holds the latest state. S3 holds history. The CDC Lambda turns state changes into analysis-friendly events.

The downside is obvious. This is more moving parts than HN strictly requires. It also has a small blind spot around poller restarts because the poller keeps its previous baseline in memory. If the poller restarts, it takes a fresh baseline and only catches later diffs. I accepted this because the goal was not perfect archival of every possible API state. I wanted a reliable enough public-event history for analysis.

Top Stories Pipeline

Figure 2. Top Stories pipeline for rank snapshots and rank movement events.

The Top Stories pipeline tracks visibility.

The HN item API tells me an item’s score and metadata, but it does not tell me when the item was on page 1, when it crossed from rank 31 to rank 30, or how long it stayed near the top. For that, I needed ranking snapshots.

The top stories poller reads topstories.json on a fixed interval. For each snapshot, it writes the ranked list to Firehose and S3. It also compares the current ranking with the previous ranking and emits rank events.

APPEAR
DISAPPEAR
RANK_UP
RANK_DOWN

The advantage is that ranking dynamics become explicit. I can ask when an item first appeared in Top Stories, whether it reached the Top 30, how long it stayed visible, and how rank movement related to later upvotes.

The tradeoff is storage volume. A full ranking snapshot writes many records even when the ranking only changes a little. In this pipeline, that tradeoff was acceptable because the records are simple, compressed S3 storage is cheap, and the analysis becomes much easier. I would rather store a slightly redundant ranking history than reconstruct visibility from partial state later.

User Pipeline

Figure 3. User pipeline for profile snapshots and historical author backfill.

The user pipeline came later and had the lowest priority.

Conceptually, it is similar to the item pipeline. A user poller reads the profiles field from updates.json and sends changed user IDs to SQS. A Lambda fetcher calls /v0/user/<id>.json, stores the latest user profile in DynamoDB, and archives raw snapshots to S3 through Firehose.

The main difference is that I did not build a CDC event pipeline for users.

User records have much less dynamic structure than items. For my use case, I mostly needed author context such as karma, account age, profile text, and submitted history. Field-level user events were not important enough to justify another stream.

The biggest operational decision was user backfill. Since the user pipeline was added after the item pipeline, I needed a backfill to recover author profiles for historical items. I used a residual backfill instead of a blind one to avoid redundant HN API calls for users already collected by the live pipeline. I started with small canaries, then 100 users, then 1,000, then 5,000, and eventually continued in 5,000-user chunks. The rate ceiling stayed at 5 users per second. The pace was conservative. It kept the live queue healthy and avoided unnecessary load on HN.

Observability and Operation

The operational goal was not to build a perfect monitoring system. I wanted enough observability to know whether the pipeline was alive, whether queues were backing up, and whether data was still reaching S3.

The pollers publish CloudWatch metrics such as heartbeat, poll latency, and number of enqueued IDs. SQS queue age is the main downstream health signal. If the poller heartbeat is present while the queue age is rising, the fetcher side is not keeping up.

I also added a small monitoring layer with CloudWatch alarms, SNS email alerts, and a weekly summary Lambda. The summary checks monitored surfaces, throughput, alarm history, queue health, and coarse S3 freshness. It can tell me that the monitored surfaces look healthy, but it cannot prove every individual record was processed exactly as intended. That was the right level for a small always-on research pipeline.

Infrastructure was deployed with Terraform where I wanted a repeatable control plane. The pollers ran as Docker containers under systemd on EC2. Lambda deployment used scripts and packaged zip artifacts. S3 data was queried with Athena when I needed quick checks or lightweight analysis. I also added small cost controls later, including batched CloudWatch metric publication, lower heartbeat frequency, and log retention.

Results and Takeaway

The archive contains item data from November 17, 2025 UTC through May 11, 2026 UTC. The Top Stories archive starts on November 27, 2025 UTC. The user archive starts on March 18, 2026 UTC.

Across that period, CloudWatch and S3 show roughly the following volume.

• 9.4M raw item snapshot records sent to Firehose
• 10.6M item change event records sent to Firehose
• 218.2M Top Stories rank snapshot records sent to Firehose
• 29.0M Top Stories rank event records sent to Firehose
• 2.7M raw user snapshot records sent to Firehose
• 26.0M item SQS messages received
• 2.8M user SQS messages received

Operationally, I would not claim literal zero errors. CloudWatch shows a small number of transient Lambda errors across millions of invocations. I did not see a sustained pipeline incident in the metrics I checked. Firehose failed-put metrics were zero, and max queue age stayed bounded during the archived period.

I think this was partly because the system was not actually very complex. HN is also a stable community site rather than a high-variance traffic source. The traffic is large enough to be interesting and small enough that a conservative AWS pipeline can handle it comfortably.

References