PostHog Used Claude to Write a 70x Faster SQL Parser

PostHog's new SQL parser hits 70x speedup over the ANTLR-generated C++ version, and it was written almost entirely by Claude Code in parallel sessions.

Why PostHog Needed a Faster Parser

PostHog transpiles user SQL to ClickHouse SQL. That requires a parser to turn raw queries into an AST before transformation. The old parser used ANTLR, a powerful parser generator that compiles a grammar into an ATN (an NFA-with-a-stack) and walks that graph at runtime. ANTLR supports arbitrary dynamic lookahead and linear approximation simulation, but a generic graph-walking interpreter can never match a hand-rolled recursive-descent parser for speed.

The parser operates on untrusted input. Everything downstream -- access controls, optimizations, transpilation -- depends on the AST it produces. A faster parser reduces p95 latency for every query that hits PostHog.

The Approach: Recursive-Descent with Pratt and Property-Based Testing

Robbie Coomber ran two parallel Claude Code sessions. One focused on pure performance: a recursive-descent parser with a Pratt expression loop, adding lookahead and backtracking only where necessary. The other tried to mirror ANTLR's behavior but with explicit code instead of generic graph traversal. Both approaches worked equally well, but that wasn't clear until days in.

Key to the effort was having an "oracle" -- the existing C++ parser. Coomber used property-based testing via Hypothesis to generate SQL queries and compare the new parser's output to the oracle. He had to teach Hypothesis how to generate interesting SQL, so he (with Claude) wrote a tool that codegens an SQL generator from the ANTLR grammar file. That led to writing a parser for .g4 files too.

The final output: 16K lines of parser code, 5K lines of tooling, a few thousand lines of tests, and a ~70x speedup. The new parser only disagrees with the oracle on deliberately pathological SQL like SELECT SELECT FROM FROM WHERE WHERE AND AND -- which is valid SQL but never appears in practice.

Key Lessons and the Future

Coomber tried a naive prompt of "write a new parser in Rust, make no mistakes." That didn't work. Claude made many mistakes, doubted the feasibility, and wanted to stop after each round. The real workflow was iterative: find disagreement via property-based testing, fix the new parser to agree, repeat.

The success suggests that AI-assisted coding can replace parser generators for performance-critical components. ANTLR is extremely flexible, but its abstraction cost is real. Hand-rolled, AI-generated parsers can recover that performance while remaining maintainable through property-based testing against an oracle.

PostHog already used this technique to improve query performance through "autoresearch." The SQL parser rewrite shows the pattern scales: multiple long-running AI sessions in parallel, combined with automated test generation, can produce production-ready code that would have taken months to write manually.

Source: I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code
Domain: posthog.com