Evaluating Agents Is a Different Problem
Every component eval passed. The agent pipeline still failed. What changes when you move from evaluating single calls to evaluating trajectories.
All Posts
Why LLMs Are Brilliantly Stupid
LLMs pass the bar exam but can't count letters. The failures aren't random. They're architectural fingerprints, and understanding them changes how you use these systems.
The Reward Design Problem: When Getting What You Asked For Is the Problem
The hardest part of reinforcement learning isn't the algorithm. It's knowing what you actually want, and whether that can even be formalized.
Debugging as a Window into How AI Thinks
Watching AI tools solve the same problem in radically different ways reveals something about their cognitive architecture, and about the nature of problem-solving itself.
Systems That Build Systems: The Meta-Engineering Shift
My job quietly changed underneath me. I used to write code. Now I write instructions for systems that write code. Tracing a 50-year abstraction trend to understand what engineering is becoming.
Building the Thing That Replaces You
What it's like to build AI systems that automate your own skills, and the quiet fear that doesn't fit neatly into hype or doom.
Distribution Is the Only Moat: An Engineer's Reluctant Reckoning
AI is collapsing the cost of building software, and with it, the value of technical advantages. If building is cheap, distribution, the ability to reach and retain people, becomes the only defensible position. An engineer wrestles with what that means.
Building Artificial Beings: The Wonder and the Work
Most of building AI agents is debugging JSON. But sometimes you remember what you're actually building, and the gap between those two realities is where the real questions live.
The Memory Illusion: How Stateless Systems Fake Remembering
LLMs don't remember anything, yet agents built on them seem to learn and retain information. The gap between these facts reveals something surprising about what memory actually is.
The Art of Saying No: Engineering Judgment in the Age of AI Generation
As AI handles more code generation, the human skill shifts from creation to curation. What is engineering taste, why can't AI have it, and what does that mean for us?
What It Means to Learn
Children forget everything yet learn faster than any AI. What are they exhibiting about learning that we've failed to capture in our models?
Evals Are Hypotheses, Not Tests
Most AI engineers build evals like unit tests. That's why they fail. What changes when you treat evals as hypotheses about what matters instead of tests of model quality.
MCP Discoverability: The Hidden Cost of Scale
As the agentic ecosystem matures, tools are no longer scarce. They're everywhere. The hard part now isn't wiring up tools — it's helping models discover which ones to use.
Context Engineering: The Hidden Lever Behind Agent Performance
In the past year, agent architectures have gone from niche experiments to front-page product strategies. But one area remains dramatically under-discussed: context engineering.
Building Better AI Evals: Lessons from the Trenches
Evaluation has quietly become the backbone of modern AI products. It's what separates a system that 'looks cool in demos' from one that actually works.
The Future of Generative AI Work: 5 Layers That Will Define the Next Decade
Over the next 10 years, the GenAI landscape won't be shaped by prompt hacks or viral demos. It will be defined by who builds the infrastructure, systems, safety nets, and experiences that actually ship and scale.
The Production Reality of LLMs: A Comprehensive Analysis of Use Cases in 2023
A deep dive into how companies are actually using large language models in production, from GitHub Copilot writing 46% of code to enterprises struggling with hallucination rates of 27%
The Great AI Game: Mapping the Players, Power, and Politics of the LLM Revolution
An in-depth analysis of the LLM ecosystem in May 2023, from Geoffrey Hinton's dramatic Google exit to the $50 billion funding frenzy reshaping Silicon Valley's power structure
What are Large Language Models?
This is the first of a multi-part series exploring exciting new developments in AI. A deep dive into the models that power ChatGPT.