Tags

Building a Data Preprocessing Agent

TL;DR

• I built a data preprocessing agent with my teammates Max, Sean, and Shreya.
• We built a benchmark suite called KaggleBench. The novel idea is to evaluate agents on high-quality, human-expert data preprocessing workflows selected from Kaggle, rather than synthetic data.
• We built an agent with a workflow and tools tailored specifically to this...

When Does a Post Go Viral on Hacker News?

TL;DR

• Hacker News (HN) is a popular tech news aggregator, and I was curious about its ranking dynamics and how virality works on HN.
• I built an event-driven data pipeline on AWS to collect every event that happens on Hacker News.
• I ran causal inference using a Bayesian approach and found that 1) there is an upvote-rate...

Causal Analysis of Virality on Hacker News

TL;DR

• HN virality is not just “good post gets upvotes.” Visibility, ranking, timing, and moderation all interact.
• I treated virality as reaching the Top 10 within 72 hours, then looked at the process through trajectory patterns, page exposure, and submission timing.
• The results suggest that page exposure and timing matter, but the causal interpretation only makes sense under...

Hacker News Data Pipeline

TL;DR

• I built a data pipeline that stores public events from Hacker News.
• The first goal was data analysis, but I also wanted a reusable event backbone for future HN projects, including recommendation systems.
• Tech Stack: EC2, SQS, Lambda, DynamoDB, Kinesis Data Firehose, S3, Athena, CloudWatch, Terraform, and Docker.

The data collected by this pipeline was used...

Data Agent Architecture

TL;DR

LLM-as-a-Judge in Practice

This post summarizes the principles I keep in mind whenever I use LLM-as-a-Judge to evaluate and improve a system. Most of them come from my experience building LLM Judges while working at Hyperconnect¹ and from Hamel Husain’s writing.²

What I want to discuss here is not how to prompt an...

KaggleBench

TL;DR

Coming soon… You can take a look at the details of the benchmark and the agent!

ChatGPT Fast Answers Part 3: Operations and Continuous Improvement

Coming soon.

ChatGPT Fast Answers Part 2: Production Architecture

TL;DR

• This post designs how the ML system from Part 1 could be implemented in production.
• It focuses on the key production architecture decisions specific to a Fast Answer system.

This is a four-part series:
Part 0. Product Teardown and Related Work
Part 1. Business Context and ML System Design
<a...

ChatGPT Fast Answers Part 1: Business Context and ML System Design

TL;DR

• Part 1 turns the product observations from Part 0 into a business objective and an ML system design.
• This post is less about sophisticated model architectures and more about how to translate a business objective into concrete ML problems.

This is a four-part series:
Part 0. Product Teardown and Related Work
<a...

What If I Had Launched ChatGPT Fast Answers?

TL;DR

• ChatGPT has a feature called Fast answers.¹ It appears to serve ready-made answers to frequently asked, information-seeking questions.
• What if I were the person responsible for launching it? For fun, I worked backward from the product and wrote a hypothetical system design.

This is a four-part series. Part 0 covers my...

Two Questions I Had While Exploring Mixedbread Search

TL;DR

• Mixedbread Search is an API service that builds searchable multimodal indexes from files such as PDFs, images, audio, and video.
• I was curious about how the product might address two problems.
• Based on several assumptions, I wrote down how I would approach them through a few small experiments (customer-specific retrieval signals and web retrieval...

ChatGPT Fast Answers Part 3: Operations and Continuous Improvement

Coming soon.

ChatGPT Fast Answers Part 2: Production Architecture

TL;DR

• This post designs how the ML system from Part 1 could be implemented in production.
• It focuses on the key production architecture decisions specific to a Fast Answer system.

This is a four-part series:
Part 0. Product Teardown and Related Work
Part 1. Business Context and ML System Design
<a...

ChatGPT Fast Answers Part 1: Business Context and ML System Design

TL;DR

• Part 1 turns the product observations from Part 0 into a business objective and an ML system design.
• This post is less about sophisticated model architectures and more about how to translate a business objective into concrete ML problems.

This is a four-part series:
Part 0. Product Teardown and Related Work
<a...

What If I Had Launched ChatGPT Fast Answers?

TL;DR

• ChatGPT has a feature called Fast answers.¹ It appears to serve ready-made answers to frequently asked, information-seeking questions.
• What if I were the person responsible for launching it? For fun, I worked backward from the product and wrote a hypothetical system design.

This is a four-part series. Part 0 covers my...

What If I Had Launched ChatGPT Fast Answers?

TL;DR

• ChatGPT has a feature called Fast answers.¹ It appears to serve ready-made answers to frequently asked, information-seeking questions.
• What if I were the person responsible for launching it? For fun, I worked backward from the product and wrote a hypothetical system design.

This is a four-part series. Part 0 covers my...

ChatGPT Fast Answers Part 3: Operations and Continuous Improvement

Coming soon.

ChatGPT Fast Answers Part 2: Production Architecture

TL;DR

• This post designs how the ML system from Part 1 could be implemented in production.
• It focuses on the key production architecture decisions specific to a Fast Answer system.

This is a four-part series:
Part 0. Product Teardown and Related Work
Part 1. Business Context and ML System Design
<a...

ChatGPT Fast Answers Part 1: Business Context and ML System Design

TL;DR

• Part 1 turns the product observations from Part 0 into a business objective and an ML system design.
• This post is less about sophisticated model architectures and more about how to translate a business objective into concrete ML problems.

This is a four-part series:
Part 0. Product Teardown and Related Work
<a...

What If I Had Launched ChatGPT Fast Answers?

TL;DR

• ChatGPT has a feature called Fast answers.¹ It appears to serve ready-made answers to frequently asked, information-seeking questions.
• What if I were the person responsible for launching it? For fun, I worked backward from the product and wrote a hypothetical system design.

This is a four-part series. Part 0 covers my...

ChatGPT Fast Answers Part 1: Business Context and ML System Design

TL;DR

• Part 1 turns the product observations from Part 0 into a business objective and an ML system design.
• This post is less about sophisticated model architectures and more about how to translate a business objective into concrete ML problems.

This is a four-part series:
Part 0. Product Teardown and Related Work
<a...

ChatGPT Fast Answers Part 2: Production Architecture

TL;DR

• This post designs how the ML system from Part 1 could be implemented in production.
• It focuses on the key production architecture decisions specific to a Fast Answer system.

This is a four-part series:
Part 0. Product Teardown and Related Work
Part 1. Business Context and ML System Design
<a...

Two Questions I Had While Exploring Mixedbread Search

TL;DR

• Mixedbread Search is an API service that builds searchable multimodal indexes from files such as PDFs, images, audio, and video.
• I was curious about how the product might address two problems.
• Based on several assumptions, I wrote down how I would approach them through a few small experiments (customer-specific retrieval signals and web retrieval...