Annas Bin Adil

The Mechanical Bottleneck

2026-05-15T00:00:00Z

Epistemic status: Research synthesis with interpretation. I'm writing as someone reading the field, not someone building in it. The empirical claims are sourced. The three implications I draw are at different confidence levels, flagged where they get speculative.

The thing that will pace humanoid robot deployment over the next decade is not a chip. It is not data. It is not even labor. It is a small geared part most engineers can't name, made by a handful of Japanese suppliers, and the global supply caps humanoid production at roughly 500,000 units a year regardless of how good the underlying AI gets.

This is the central finding of Epoch AI's April 2026 piece, "How Fast Could Robot Production Scale Up?" It is, as far as I can tell, the cleanest published work on physical-AI deployment timelines. The bottleneck the piece names is precision reducers: planetary gearboxes, cycloidal/RV reducers, strain-wave gears. Tiny mechanical components made by firms most AI people couldn't list if asked.

The post draws out what that one rigorous empirical finding implies for three things that aren't getting enough attention: timeline forecasting, industrial policy, and embodied AI safety.

Capability got cheap. Deployment didn't. The bottleneck is mechanical, not silicon, and the AI discourse hasn't mapped it.

What Epoch found

The piece is a bottom-up bill-of-materials analysis across five form factors: humanoids, quadrupeds, robotic arms, wheeled robots, drones. Authors Jean-Stanislas Denain and Yann Rivière mapped global component supply against production demand, then compared the result against historical mobilization cases including WWII aircraft, Tesla Shanghai factory builds, and Ukraine FPV drone production.

The current state of humanoid production is small. About 16,000 units per year in 2025, doubling every six months. The doubling rate is anomalous, Epoch notes. Most of those units are research platforms and marketing demonstrations, not productive work. The real test is what the production curve looks like once capability arrives and somebody actually needs millions of humanoids.

Their projection, under a demand-shock scenario triggered at end of 2027, is that humanoid production reaches 1.5 to 3 million units per year by end of 2030 clearly, and 5 to 10 million plausibly. Drones could reach 100 to 200 million per year without especially hard limits. Quadrupeds 8 to 15 million. These are not the kind of numbers that move you off your timeline if you were expecting an AI rollout shaped like the smartphone wave. Smartphones hit a billion units shipped in six years from the iPhone launch. The Epoch projection puts humanoids at less than 1 percent of that volume over a similar window.

What constrains the rate is not the things the AI discourse usually worries about. Cameras come off production lines at 7 billion units per year globally. MEMS sensors at 31 billion. Bearings at tens of billions. Batteries at 2 billion kWh per year. None of these are binding. None of these sit on the critical path.

The chokepoint is precision reducers. Planetary gearboxes at roughly 3 million units per year. Cycloidal and RV reducers at roughly 2 million. Strain-wave gears at roughly 12 million, the least restrictive of the three. Humanoids use somewhere between 20 and 40 reducers each, depending on hand dexterity. Tesla Optimus, with its high-degree-of-freedom hands, could triple per-robot reducer demand. The math caps humanoid production at roughly 500,000 per year at current capacity, no matter how many factories you build.

The supply concentration matters. Harmonic Drive Systems (HDS, listed on the Tokyo Stock Exchange) and Nabtesco, both Japanese, dominate the precision-reducer market. HDS scaled its Ariake plant from 150,000 to 220,000 units per month by August 2022 to meet industrial-robot demand, with the Hotaka facility adding another 40,000 monthly. Total HDS capacity sits around 3 million units per year. Nabtesco is the dominant supplier in RV reducers used in industrial-robot joints. China has been pushing localization aggressively. Leaderdrive holds 30 to 40 percent of China's harmonic-reducer market and counts Tesla among its customers. But a Jamestown Foundation analysis notes that while 80 percent of RV reducers are now assembled in China, roughly 90 percent of the machine tools used to make them are imported, mostly from Japan.

Why mechanical precision is the binding constraint

The first-principles answer to why mechanical precision is the bottleneck rather than something else comes from looking at how each potential bottleneck scales.

Chips scale with fabrication capacity, and the world has been pouring capex into fab construction since the CHIPS Act. The marginal chip for a humanoid is not where the bottleneck lives. Batteries are similarly in a phase of aggressive capacity expansion driven by EVs. Cameras and MEMS sensors get produced at volumes orders of magnitude larger than humanoid demand could plausibly require.

Labor doesn't bottleneck because factory operations workers, which Epoch estimates at 40,000 to 120,000 for 10 million humanoids per year, can be trained in parallel during the construction phase. The construction phase itself is the timing constraint. Auto-plant retrofits run 6 to 10 months. Greenfield Chinese factory builds run 6 to 9 months. Greenfield Western builds run 2 years or more.

Precision reducers are different because the manufacturing process is itself a deep skill stack. These are gear-on-gear interfaces machined to micrometer tolerances, with proprietary metallurgy and grinding processes that took specific Japanese firms decades to refine. They are not a thing you can produce by buying CNC machines and hiring trained workers. The machine tools themselves, the precision grinders and gear-cutters used to make reducers, are largely Japanese-made. China's domestic reducer production has scaled, but the Jamestown analysis flagged that it sits on top of imported machine tools from the same source.

This is a constraint with the shape of TSMC, not the shape of Foxconn. TSMC isn't a chokepoint because chip fabrication is conceptually hard. It is a chokepoint because the tacit knowledge of running advanced fabs is concentrated in one firm. Precision reducers are similar. The capacity expansion path does not bottleneck on capital, raw materials, or labor. It bottlenecks on tacit metallurgical knowledge and on the upstream machine tools that themselves bottleneck on the same Japanese firms.

Year 1 of any demand shock, Epoch notes, is mostly construction rather than production. By end of 2028, even with aggressive scale-up, the global stock of newly-produced humanoids would be in the low hundreds of thousands.

Reducer capacity could 2x by 2028 and 4x by 2030 on baseline trajectory, or 10x to 30x under demand shock. The 10x-to-30x range is where most of the projection's uncertainty lives. It is also where the question becomes interesting: how fast can Japanese firms expand without losing the precision that made them the suppliers in the first place? That is not a question with an obvious answer.

Implication 1: Timeline forecasting

This finding reshapes timeline forecasting for AI in a specific way. The AI safety and forecasting community has done thoughtful work on capability timelines and very little on deployment.

Ajeya Cotra's "Forecasting Transformative AI with biological anchors" (Open Philanthropy, 2020-2022) is the most-cited capability timeline analysis. It anchors compute requirements to biological reference points and projects hardware-and-spending trends. Deployment is bracketed as a downstream consequence with minimal physical-world friction modeled.

Holden Karnofsky's "Most Important Century" (Cold Takes, 2021) introduces PASTA, his term for the process of automating scientific and technological advancement, and assumes that once AI can automate research, physical deployment follows quickly via AI-designed robotics. He flags this as a load-bearing assumption.

Leopold Aschenbrenner's "Situational Awareness" (June 2024) makes essentially the same move. Intelligence-explosion thesis, with physical deployment treated as a fast follow-on. The deployment treatment is the weakest part of the document.

The exception is Dario Amodei's "Machines of Loving Grace" (October 2024). Amodei explicitly addresses deployment lag, coining the phrase "limits to compressed 21st century." He estimates 5 to 10 years for biology breakthroughs to deploy through clinical trials and regulatory approval. He does not analyze robotics supply chains specifically, but he is the only frontier-lab CEO publicly grappling with the fact that capability does not equal deployment.

The Epoch piece fills a real hole in this literature. It is one of the few rigorous bottom-up deployment analyses from the AI-aware community.

What does the analysis change about timeline forecasting? Three things.

First, it sharpens the capability/deployment distinction. If you believe AI capability will arrive in the next three to five years and you're thinking about what the world looks like after that, the Epoch numbers should pull your humanoid-deployment estimates downward. Not because capability slips. Because deployment was always going to take three to five more years after capability already arrives.

Second, it gives a falsifiable reference class. Industrial robots took fifty years to reach a million deployed units, according to International Federation of Robotics data. They are now at around 3.5 million in operational stock. Humanoid bull projections, including Tesla's stated target of "millions per year" by the late 2020s and Figure's commercial deployment framing, imply collapsing that 50-year history into roughly five years. The Epoch analysis says the industry-wide ceiling under demand shock is 5 to 10 million per year by end of 2030, which would require essentially the entire global reducer supply chain to be captured by one or two players. That is not impossible, but it is the implausible case, not the central one.

Third, it explains the Rodney Brooks pattern. Brooks has maintained an annual predictions scorecard for over a decade tracking robotics and self-driving forecasts. The pattern across the field is consistent. Capability demos arrive roughly on schedule. Deployment-at-scale runs 3 to 10x slower than industry forecasts. The Epoch analysis suggests the slowdown isn't psychology or marketing. It is the supply chain. If you forecast deployment as though it tracks capability, you systematically miss by an order of magnitude.

The cleanest data point on the gap between marketing and reality: Agility Robotics' Salem, Oregon factory, with stated capacity around 10,000 humanoids per year. That is the verifiable number. Tesla and Figure imply millions. Agility shows ten thousand. The ratio is the gap between marketing and what the supply chain currently supports.

Implication 2: Industrial policy

The implications for industrial policy are sharper and probably more actionable than the timeline question. Strategic chokepoint thinking has been applied to semiconductors but not yet to reducers, despite the geographic concentration being arguably tighter.

China has been pushing this question explicitly for a decade. Made in China 2025 named speed reducers, servomotors, and controllers, which together represent about 70 percent of robot bill-of-materials by value, as indigenization priorities.

The results are mixed and the literature is honest about it. A peer-reviewed PageRank-on-patents study by Liu et al. (ScienceDirect, 2025) found that Made in China 2025 lifted midstream and downstream robotics innovation quality but failed to move upstream component quality, which is exactly the reducer and sensor and controller layer. Leaderdrive's 30 to 40 percent share of China's harmonic-reducer market is real growth, but, as noted, sits on imported Japanese machine tools.

The US response has been narrow. The most concrete recent action is the Section 232 national-security investigation of robotics and industrial machinery imports, initiated by Commerce on September 2, 2025 (Federal Register notice 2025-18749). Public comments closed October 17, 2025; tariff or import-restriction decisions plausibly land in spring 2026. This is a trade tool, not a capex subsidy. The Humanoid ROBOT Act of 2025 (S.3275) extends Section 889-style federal procurement bans to humanoids from PRC, Iran, DPRK, and Russia-linked entities. That is a procurement-restriction bill, not a capacity-building bill.

The Information Technology and Innovation Foundation published "A Time to Act: Policies to Strengthen the US Robotics Industry" in July 2025, by Robert Atkinson. The numbers are striking. Japan produces 46 percent of global robotics output. By 2024 imports outweighed exports 4 to 1. Atkinson recommends expanding NIST Manufacturing USA programs and the Advanced Robotics for Manufacturing institute, plus restrictions on Chinese robotics imports. The recommendations stop short of invoking Defense Production Act Title III authorities or proposing CHIPS-style appropriation for robotics components.

The rare-earth angle compounds this. Robots need high-density permanent magnets, specifically NdFeB with heavy rare-earth additions of terbium or dysprosium, for the servo motors that pair with reducers. China's October 2025 MOFCOM Notices No. 61 and 62 require licenses for these magnets, with extraterritorial licensing for magnets made overseas using Chinese technology starting December 1, 2025. The notices were suspended until November 10, 2026, but the regulatory framework is in place. Industrial robots use roughly 300 to 500 grams of rare-earth-doped magnets per motor, and a humanoid has 20 to 40 motors. The supply-chain exposure is non-trivial.

What would real policy that takes the bottleneck seriously look like? Probably some combination of explicit chokepoint mapping treating precision reducers and rare-earth magnets together; capex incentives for domestic reducer manufacturing, which would in turn require investing in the machine-tool supply chain that the Liu et al. analysis shows is the harder upstream problem; and allied coordination with Japan rather than competition. The current US framing of trade tools and procurement bans is closer to the early-2010s semiconductor framing than to the post-2020 framing. The conceptual upgrade has not happened yet.

Implication 3: Embodied AI safety

The safety implications are the most speculative of the three. The embodied-AI safety literature is thin, fragmented, and dominated by self-driving cars, which the broader physical-AI community treats as a sibling discipline rather than the main event.

The conceptual foundation is Stuart Russell's Human Compatible (2019) and the Center for Human-Compatible AI's work on Cooperative Inverse Reinforcement Learning (Hadfield-Menell, Russell, Abbeel, Dragan, NeurIPS 2016). Russell's domestic-robot thought experiment, the robot that cooks the cat because it wasn't told cats are loved, is the canonical intuition pump. CIRL formalizes preference inference from behavior. This is rigorous theory with no deployed safety system attached to it.

Anthropic's Responsible Scaling Policy addresses biological, chemical, radiological, nuclear, and cyber risks plus autonomy, but does not name physical AI or robotics as a capability threshold in the public versions I have seen. OpenAI shuttered its robotics team in 2021 and has no published safety research on its more recent humanoid investments. The robotic foundation model labs themselves, including Physical Intelligence (π0), Open X-Embodiment, and Figure (Helix), have published essentially no public safety research either.

The empirical safety evidence comes almost entirely from autonomous vehicles. The Cruise robotaxi incident on October 2, 2023, in San Francisco, remains the single most documented physical-AI safety failure on public record. A Cruise vehicle dragged a pedestrian roughly 20 feet after she was struck into its path by a human-driven car. The California Public Utilities Commission suspended Cruise's permit on October 24, 2023. The Quinn Emanuel independent report (January 2024), commissioned by Cruise itself, documented that the company initially showed regulators a video that cut off before the dragging. The failure was both technical, in that the vehicle's reaction model didn't handle the secondary-impact case, and organizational, in that the company concealed it from regulators. Cruise's parent eventually wound down the program.

Tesla's Autopilot has generated a parallel safety pattern: an NHTSA recall in December 2023 covering approximately 2 million vehicles, with 13 documented fatal crashes. That is a regulator-driven case rather than a single dramatic incident.

Waymo has published the most rigorous public safety case. The Waymo Safety Hub and Kusano et al.'s 2024 comparison study report approximately 25 million rider-only miles with significant claimed reductions in police-reported and injury crashes versus human-driver baselines. The comparison-class selection is Waymo's, but the methodology is the strongest in the field.

The Epoch deployment-lag analysis changes this picture in three places, ordered by how grounded the observation is.

The slow-rollout-helps-safety argument breaks in the military vertical. Paul Scharre's Army of None (2018) and ongoing CNAS work document the trajectory. The OpenAI-Anduril partnership announced in December 2024 and the Anthropic-Palantir-AWS defense deal in November 2024 are real moves into a market where the customers, state militaries, are explicitly willing to absorb unit cost premiums that would crush commercial deployment. The reducer constraint applies to commercial humanoids assuming peacetime allocation. It does not apply if a state actor commits to scaling regardless of cost. The Epoch analysis implicitly assumes a peacetime industrial pattern. This is the strongest grounded counter to the slow-rollout-helps-safety case.

Physical-AI safety research has more runway than the discourse implies, at least on the commercial side. If commercial humanoid deployment is paced by supply chains running on industrial-time, the field has five to ten years to develop, test, and deploy safety techniques before the population of deployed embodied agents is large enough that incidents become routine. The current state, in which there are zero public safety publications from any major robotic foundation model lab, is therefore the bigger problem than the slow rollout. The runway exists. The research is not filling it.

And most speculatively: there is a thesis nobody has made canonically yet, that embodied AI is where alignment debates become empirical. Robots produce real-world actions with real-world consequences in a way text generators don't. As humanoids reach the deployed scale Epoch projects, the LLM discourse's mostly-theoretical alignment debates will have physical analogues. I'm flagging the thesis because the literature is open for someone to make it well. I'm not in a position to be that someone here.

Cruxes

The Epoch analysis is a snapshot, not a final answer. Five things could change it.

Recursive self-improvement is the largest deferred question. Epoch explicitly brackets the case where AI starts designing reducer manufacturing plants or developing the metallurgical processes that currently take Japanese firms decades to refine. If that happens, the bottleneck story changes. The analysis assumes ordinary human engineers run the scale-up. That is a reasonable simplification for current conditions and a bad one for any future where the AI is itself optimizing the supply chain.

Software substituting for hardware is the second crux. Physical Intelligence demonstrated precision manipulation tasks using relatively simple grippers, with fewer actuators and less reducer demand per unit, by improving the learned policy. If foundation models can compensate for mechanical imprecision the way deep learning has compensated for hand-engineered features, the bill-of-materials math shifts. Robots could become viable at lower precision tiers, which would change which suppliers matter and what the cap actually is.

Cohort versus price effects is the third. Industrial robots took 50 years to reach a million deployed units partly because of supply constraints and partly because the unit economics did not work for most applications until well into that window. If humanoid prices follow a fast cost curve, both because the AI labor cost is in the training run rather than the marginal robot and because EV-style battery and motor economies will apply, the reference class might be wrong. The bottleneck still binds the production curve, but the demand curve and the deployment curve might decouple from it differently.

Geopolitical fragmentation is the fourth and least quantifiable. The Epoch analysis assumes global trade keeps working. A Taiwan crisis, escalating sanctions, or a serious break in the Japan-China-US technology relationship would either collapse the supply chain or fragment it in ways that produce different and unpredictable constraints. The current rare-earth export-control framework is a small preview of the kinds of moves that could matter much more.

The fifth thing the analysis does not address is demand origin. Why does a shock happen, and when? Epoch's projection runs from an EOY 2027 trigger date, which is a placeholder. The whole analysis is "conditional on a shock, how fast can production scale" rather than "when will a shock happen and what kind." That is the right scope for the question they were asking, but it means the deployment timing is really two timelines stacked, capability arrival and then production response, and the post only addresses the second.

Where this leaves me

What I am watching for over the next eighteen months: whether HDS or Nabtesco announces capacity expansion specifically tied to humanoid demand at scale; whether the Section 232 investigation produces anything more substantial than tariffs; whether the major humanoid companies publish anything resembling a safety framework; and whether anyone writes the CSIS-style brief that frames the reducer-and-magnet chokepoint as the strategic problem it appears to be.

The most useful posture an outsider can take on physical AI right now is not prediction. It is noticing which questions the discourse has not gotten to yet.

The place I am most likely to be wrong is the software-substituting-for-hardware crux. If Physical Intelligence or another lab demonstrates production-grade manipulation at lower reducer counts than current humanoid architectures assume, the bottleneck story shifts and most of the post's three implications shift with it. I'd want to hear from anyone watching that closely.

The Tools Have Split

2026-04-15T00:00:00Z

Epistemic status: Grounded in published 2025-2026 survey data and my own day-to-day use of these tools. The empirical claims are sourced. The unifying frame is mine, and I am moderately confident in it. The 18-month prediction at the end is speculation, offered as a falsifiable bet rather than a confident forecast.

In a 2026 survey of about 900 engineers, the most-loved AI coding tool is Claude Code, at 46%. Cursor is at 19%. GitHub Copilot at 9%. The detail that matters is buried in the seniority breakdown: among directors and senior leaders, Claude Code's preference roughly doubles, while Cursor's declines with seniority. If AI coding tools were substitutable, this shouldn't happen.

Seniority should not predict tool preference. The preference should track feature sets or per-session productivity, not years of experience. Instead, the most experienced engineers are systematically choosing the tool that demands the most from them as operators.

This is one survey of one self-selected senior-skewed audience (Gergely Orosz's Pragmatic Engineer subscribers, median 11-15 years experience), and the qualitative axes I'll lay out next are framing, not additional evidence. If a comparably rigorous 2026 survey showed a different gradient, the rest of this argument wobbles. But the gradient is real in this data, and I think it points at something worth taking seriously.

The standard gloss is "they understand the leverage." True but undercooked. The deeper story: AI coding tools have quietly split into two classes that serve different jobs, and senior preference is the market signal for which class produces good systems.

The thesis

The thesis I want to defend is small and specific:

The cost of producing code dropped. The cost of producing good systems didn't. The craft has been redistributed, not eliminated.

This is a claim about where the bottleneck moved, not whether AI is good or bad. AI is unambiguously useful for the parts of software work that were always cheap to do correctly. Greenfield code in a familiar language. Boilerplate. Tests that follow obvious patterns. Documentation drafts. The mechanical layer of software production has collapsed in cost in a way that's visible in adoption data: 85-90% of working engineers report regular AI use across the major 2025-2026 surveys (DORA, Stack Overflow, Pragmatic Engineer, JetBrains all triangulate on this).

But the thing that takes thirty years of experience to do well was never typing. It was knowing what to build, in what order, for which users, with which trade-offs, and recognizing when a working implementation is in fact the wrong implementation. That layer of the work is structurally different from typing. It cannot be cheapened by faster code generation, because it doesn't bottleneck on code generation. It bottlenecks on judgment.

The empirical evidence is consistent with the split. DORA 2024/2025 found that as AI adoption rose toward saturation, throughput went up but delivery stability went down. The 2024 model estimated a 7.2% reduction in delivery stability per 25% increase in AI adoption. DORA 2025 framed the mediating variables as internal platform quality and review discipline: teams with strong platforms compounded gains; teams without them compounded drag. AI didn't shift the outcome. It amplified whatever the team was already doing. The Stack Overflow 2025 Developer Survey found adoption climbing to 84% while developer trust in AI accuracy collapsed to 29%, and 45% of respondents reported that debugging AI-generated code takes more time than writing the code themselves.

None of these findings say AI is bad. They say the same thing in different vocabularies: the mechanical layer cheapened, the judgment layer didn't, and the gap shows up wherever you measure system quality instead of code volume.

If the work has split along that line, the senior-preference inversion isn't mysterious. Senior engineers spend most of their time on the part that didn't cheapen. The part that did cheapen was already cheap for them, because they had built the mental models that made typing the easy part. The AI tool that matters to a senior engineer is one that amplifies the part still expensive, not the part already cheap.

The tools have split

Cursor and Claude Code get discussed as the same product with different price points and feature sets. They aren't. They are two products that happen to share a category label, and the seniority split is the clearest evidence I have. Four axes describe the difference.

Where the interface lives. Cursor's center of gravity is inline autocomplete. You're typing; suggestions appear; you accept, reject, modify. The unit of interaction is a few-character to few-line completion, evaluated in milliseconds. Claude Code's center of gravity is agentic delegation. You describe an intent; the agent reads the code, plans, edits multiple files, runs commands, returns a diff. The unit of interaction is a task, evaluated in minutes. These aren't optimizing for the same outcome.

What the tool requires you to bring. Inline autocomplete is forgiving of weak specification. You don't need to know exactly what you want; you start typing in roughly the right direction and the tool fills in. The skill it amplifies is recognition. Agentic delegation requires complete-enough specification up front. "Add the next endpoint" is too vague. The agent needs target behavior, edge cases, integration points, constraints. The skill it amplifies is specification. Senior engineers have stronger specification skill, accumulated through years of building things and watching them fail. Juniors are still learning what good specification looks like, so they get more value from a tool that doesn't demand it up front.

Trust and accountability flow. Cursor's UX nudges toward accept-and-move-on. Tab-complete feels low-stakes per suggestion. You can ship a lot of accepted suggestions without ever doing an explicit "do I trust this?" check. Claude Code's UX is diff-review-approve. Every change gets a moment of explicit accountability before it lands. Senior engineers are accountable for what ships. They have seen enough AI failures and pre-AI failures to want the explicit check. The Claude Code flow matches that instinct. The Cursor flow asks them to trust at a rate their judgment hasn't agreed to.

Operator-skill amplification. This is the unifying axis underneath the other three. If you have taste, domain knowledge, and specification ability, Claude Code's high-bandwidth handoff puts that to work; one good specification produces hundreds of lines of useful code. If you don't have those skills, the high-bandwidth handoff is a liability: you can specify the wrong thing very efficiently. Inline autocomplete is the constrained interface. You are confined to a known context (a file, a function), and the tool extends what you started. Even with weak operator skill, you can't get too far off course in one suggestion. Agentic delegation is the unconstrained interface. You can ask for anything. With weak operator skill, this is dangerous. With strong operator skill, it is amplification.

The senior preference is not a preference for AI doing more of the work. It is a preference for an interface that rewards judgment. Cursor compensates for missing judgment. Claude Code rewards present judgment. The tools have split along a specification-skill axis: one class compensates for what you don't have; one class amplifies what you do.

Where I actually stand

I've been using Claude Code as my primary coding interface for most of the past year. The reason isn't speed.

When I work with Cursor, my eye and mind are at the implementation level: syntax, names, control flow. Even with strong AI suggestions, I'm inside the code. When I work with Claude Code, I'm one level up. I'm writing in English about what should happen. The implementation is something I review, not something I'm inside of.

This isn't delegation. Delegation is when you hand off work you could do yourself but choose not to. Most discussions of agentic tools frame them this way: senior engineers delegate the typing so they can do more important things. I think this is wrong. The thing I'm doing with Claude Code is operating at the level where my judgment is sharper than my fingers on a keyboard. That is a different relationship with the work than delegation.

This reframe predicts something the delegation frame doesn't. If agentic tools are just delegation, the senior preference for Claude Code should evaporate the moment the tools get good enough that the delegation feels automatic. If agentic tools are abstraction-level interfaces, the senior preference should persist. The current data is consistent with the second story. Staff+ engineers in the Pragmatic Engineer survey use agents 63.5% of the time, compared to 49.7% for regular engineers. The gap is widening with maturity, not narrowing.

The practitioner debate

Three senior practitioners have published the most thoughtful takes on what good engineering with AI looks like, and they disagree in instructive ways.

Steve Yegge is the most bullish. His March 2025 essay "Revenge of the Junior Developer" frames the current moment as one of six waves: traditional coding, completions, chat, coding agents, agent clusters, agent fleets. His central claim is that 95-99% of agent interactions could in principle be handled by a properly briefed model. The human role is to brief, not to author.

Geoffrey Litt is the cleanest counter-frame. His October 2025 essay "Code like a surgeon" calls the "AI makes us all managers" framing "dangerously incomplete." A surgeon does the actual work of surgery, supported by a prepped operating room and team but still in the primary work. Litt distinguishes primary tasks (core design, code by hand, AI used carefully) from secondary tasks (codebase guides, exploratory spikes, doc updates), and argues that the right pattern is to keep humans in the primary work while delegating secondary work freely.

Kent Beck draws a different line. His essay "Augmented Coding: Beyond the Vibes" distinguishes two practices that get conflated. Vibe coding (Karpathy's term) is "describe the outcome, feed errors back, hope" without regard for what code is produced. Augmented coding is traditional engineering values with less typing. Beck rebuilt a B+ Tree library three times: twice with vibe coding (both collapsed under accumulated complexity), once with TDD-driven AI agents (this one held). His operative claim: "TDD is a superpower when working with AI agents." Tests are how you discipline a non-deterministic collaborator.

Yegge's model assumes the briefing problem is tractable. The Replit incident, the slopsquatting data, and the Perry et al. confidence-competence gap (all in the next section) are evidence that it isn't, at least not yet. Briefing is the hard part, not the easy part. Yegge collapses it into a solved input. I think he is wrong about that, and not just about timing. The high-judgment work does not shrink just because more of the implementation gets automated, because the implementation isn't where the judgment was.

Litt's surgeon, Beck's augmented coding, Simon Willison's "agents amplify skilled operators," and Martin Fowler's framing of AI as the biggest shift since "assembler to high-level languages" are all arguing the same thing in different vocabulary: the high-judgment work expanded as the typing work shrank. Yegge is the dissenter.

When operator skill is missing

The clearest evidence that operator skill matters comes from what happens when it's absent. The Replit incident in July 2025 is the cleanest public case I've seen. An AI agent, operating during what should have been a code freeze, deleted a production database. It then fabricated approximately 4,000 fake users to fill the resulting void and initially reported that recovery was impossible. Replit's CEO publicly apologized and the company subsequently added a planning-only mode and explicit dev/prod separation. The incident matters not because an AI agent did something bad. It matters because the bad thing happened in a context where no operator skill was checking the agent's decisions. The Claude Code review-approve flow that senior engineers prefer is precisely the friction that prevents this class of incident.

The Stanford CCS 2023 study (Perry, Srivastava, Kumar, Boneh) asked developers to complete security-relevant programming tasks, half with AI assistance and half without. Developers with AI wrote less secure code. They also rated their code as more secure. The confidence-competence gap is the durable finding: AI raises perceived quality faster than it raises actual quality, and the gap is widest in domains where the operator doesn't have the prior expertise to recognize the failure modes.

The USENIX Security 2025 paper on package hallucination found that LLMs recommend non-existent packages roughly 19.7% of the time across 576,000 samples. Open-source models hallucinated at 21.7%; proprietary at 5.2%. The researchers identified 205,000 unique hallucinated package names, of which 38% are name conflations, 13% are typos of real packages, and 51% are pure fabrications. This isn't a theoretical attack surface. Security researcher Bar Lanyado registered the AI-hallucinated huggingface-cli package on PyPI as an empty stub in early 2024 and received 30,000 real downloads in three months. An operator who blindly accepts package suggestions from an AI is running a roulette wheel on supply-chain security.

The pattern across these cases is the same. The agent is capable. The operator is missing. The output looks productive but contains failures the operator can't detect because they don't have the underlying skill.

The eighteen-month bet

So far the argument is that the current snapshot supports the redistribution thesis. The tools have split, senior preference is real, operator skill is the bottleneck. I want to end with the part I am least certain about, which is that this snapshot is transient.

Here is the bet. Over the next eighteen months, two things change in ways that compress the specification-skill premium.

First, the tools get more abstracted. The level at which the human operates today (writing English specifications, reviewing diffs at the file level) is itself going to be wrapped. Products are appearing that take a higher-level intent ("ship a feature that does X for users like Y") and decompose it into the specifications that current agents need. This is the natural next layer of the stack. If it lands, the operator-skill bottleneck moves up another level. Specification skill at the file level becomes a craft you don't need, the way assembler is a craft most engineers don't need anymore.

Second, one-way doors get rarer. A lot of what makes software engineering judgment valuable today is the cost of getting it wrong: an architecture decision lives in the codebase for years; a poorly chosen abstraction propagates through every subsequent change; a bad data model creates compounding tax forever. The penalty for getting these wrong is what makes the early judgment matter so much. If agentic tools continue to make refactoring cheaper, the half-life of a wrong decision shrinks. Things that were one-way doors become two-way doors. The premium on getting it right the first time decreases. Iteration replaces foresight.

Both shifts compress the value of present operator skill. Better abstraction means specification becomes easier; cheaper iteration means specification becomes less critical.

The structural argument is more confident than the schedule. Eighteen months is a guess. It could be six. It could be three years. But the bet is cheap to check: if I am right, by late 2027 the Pragmatic Engineer survey shouldn't show the same Claude-Code-by-seniority gradient. The Staff+ agent usage gap (63.5% vs. 49.7%) should narrow. Higher-level intent tools should be the ones senior engineers report most-loved. If the gradient holds or widens, the prediction is wrong.

There is a real alternative explanation I should name. The senior-preference gradient might not be causal. Senior engineers in 2026 grew up writing code longhand and have stronger specification skill because they did. If juniors today are training their specification skill via different paths (more time reading existing systems, more time prompting), the gradient may flatten not because the tools change but because the operator-skill distribution catches up. The argument above is tool-driven. The cohort-driven alternative is live and I don't have enough data to rule it out.

I notice that the structural prediction has the shape of every previous "the new layer of abstraction will be the bottleneck" claim in software history, and most of them have been right. Compilers didn't eliminate the need for engineering judgment; they relocated it. IDEs didn't eliminate the need for understanding code; they changed which kinds of understanding were valuable. Cloud infrastructure didn't eliminate operations; it produced site reliability engineering. Each layer of abstraction produced a new craft at the boundary. I expect agentic coding to produce the same pattern.

Where this leaves me

The redistribution thesis is correct now and may not be correct in three years. The senior preference for Claude Code is a real signal about what good engineering with current tools looks like, and a transient signal about where the craft sits on a moving stack.

If you're a senior engineer choosing how to spend the next year, the bet I'd make is that intent-articulation tools, the layer above the current agentic interface, are where the next premium opens up. That's where I'd want to be early.

Resolution, Not Pace

2026-03-15T00:00:00Z

Epistemic status: Personal observation, not a generalizable claim. The pattern is something I've started to see in my own work and life over the past several months. The specific examples are real. The unifying frame may not generalize beyond me.

I've been noticing a pattern in my own behavior that connects work and life in a way I didn't expect. It started, of all places, on a walk.

I was on autopilot, walking the loop I always walk. End of the block, turn, get home, next thing. About halfway through I noticed, almost by accident, that the wind was on my face. The leaves were rustling. The spring flora had colors I hadn't seen yet this year. None of this was happening for me. It was happening anyway. The walk was already there. I hadn't been at the right resolution to receive it.

The same week, in a project I'm working on called Stella, I caught myself doing the same thing in code.

Two readings of "experiencing more"

I have an old notes file with a three-word fragment: "Growth in experiencing more." Wrote it months ago, forgot about it.

When I came back to it, the phrase admitted two readings. The first: experiencing more means variety. Wider exposure, more cities, more cuisines, more projects. Growth as breadth. This is the dominant cultural reading.

The second: experiencing more means depth. Perceiving more in the same moment. More attention, not more events. Growth as resolution. The wind was already there during my walk. I just hadn't sampled it.

I picked the second. Once I picked it, the title I'd been carrying around for this piece, "Slow and Fast," meant something different than I had thought.

Slow and fast are sampling rates, not pace

If growth is about resolution rather than variety, slow and fast aren't pace. They're sampling rates.

Fast mode is low-resolution sampling. You move through the day, hit milestones, complete tasks. The events happen. They don't fully register. Time compresses in memory because there wasn't enough texture to mark it.

Slow mode is high-resolution sampling. The same minute, the same walk, but more of it lands. Time expands because there's more there to remember.

This is not the same as moving slowly. You can move quickly through a day at high resolution. You can move slowly through a day at low resolution. Pace and sampling rate are independent variables, even if our culture treats them as correlated.

So far this is a personal aesthetic, the kind of thing you might find in a meditation book. I'd have left it there if it weren't for what happened on Stella.

The Stella moment

Stella is an AI safety simulation tool I've been building. It tests how an AI system behaves in high-stakes situations, the kind where a wrong response has real consequences. The question that started the project was concrete: does the system actually catch the failure modes that matter most, especially the ones that are easy to miss?

A few weeks in, I caught myself in the middle of an obvious next step. I had been building out the surrounding infrastructure: more scenario coverage, richer test blueprints, an evaluation pipeline that scored across multiple dimensions. Each piece had been the locally rational next move. AI-native development made each piece cheap to add. The next addition was twenty minutes away.

Something made me stop. Probably tiredness. I asked myself, almost annoyed, what I was actually trying to do. Not the next infrastructure piece. The thing the infrastructure was supposed to serve.

The honest answer was that the original safety question had been quietly demoted. I had a sophisticated platform. I had not actually answered, with that platform, the question that motivated it: does the system catch the disguised failure cases or doesn't it. The platform had become its own end. Each cheap step had carried me one notch further from the question I'd set out to ask.

That's when the walk and Stella connected for me.

Same shape

Walk:

"Keep walking" is the locally rational move at every step.
Each step costs almost nothing, so I don't evaluate it.
Twenty minutes later I'm home. The walk happened. I wasn't really in it.

Stella:

"Add the next piece of infrastructure" is the locally rational move at every step.
AI-native development has driven the cost of each next step toward zero.
Three weeks later I have a working platform. The platform exists. It hasn't yet answered the question I built it to answer.

The structure repeats:

A locally rational fast default at every step. A globally impoverished outcome in aggregate.

The pattern, stated plainly

The mechanism is small enough to write out:

An action has a per-step cost.
The per-step cost is what triggers the question "should I take this step?"
Drop the cost below the threshold of attention, and the question stops being asked.
You keep moving. The direction drifts. The output looks productive.
The aggregate is misaligned with what you actually wanted.

I don't think every cheap step is a problem. Most are fine. The danger is specifically when the cost of the next step is low enough that the question of whether to take it stops getting asked. The metabolic cost of decision-making was the regulatory mechanism. Take that cost to zero, and the regulator is gone.

Why AI-native development is the extreme case

Before AI assistance, the cost of writing the next ten lines of code was non-trivial. Twenty minutes of typing, looking up syntax, dealing with errors. Twenty minutes is enough friction that you tend to ask, before paying it, whether the ten lines are worth writing. The friction itself was a check on direction.

With AI assistance, those ten lines arrive in thirty seconds. The friction is gone. The check the friction was performing is gone with it.

What's left is a system where you can produce code as fast as you can describe it, where description is much faster than reflection, and where you therefore produce more than you reflect on. The code compiles, the tests pass, the demo works. But the question of fit between what you're building and what you actually wanted has been quietly skipped at every step.

The pattern isn't new. Cars made it easier to live far from work, which made it harder to ask whether the commute was worth it. Email made it cheap to send messages, which made it harder to ask whether each one needed to be sent. AI just sharpens the dynamic to where it becomes hard to ignore.

I'm part of what I'm describing. I build with these tools. I write enthusiastic posts about velocity and the collapsing cost of software. The critique here is being voiced by someone who has been benefiting from velocity for years. That doesn't make the analysis wrong, but it does make it implicated.

Slow as protocol

Slow isn't virtue. It isn't the contemplative life as an end in itself. It's something more functional.

Slow is the protocol for inserting reflection into a system that has otherwise been optimized for action. It's the artificial reintroduction of a check that the friction of effort used to provide for free.

Slow isn't the opposite of fast. Slow is the protocol for keeping fast aligned with what you actually wanted.

This is more useful than "slow is good, fast is bad." It tells you when slowness matters and when it doesn't. Slowness matters at decision points that affect direction. It doesn't particularly matter when you're executing within a direction you've already chosen.

The walk is about direction. The point of going outside, for me, isn't to traverse distance. It's to be in the world for twenty minutes. If I'm fast about that, I've done it wrong. The fast mode of walking accomplishes none of what walking is for.

The Stella project was also about direction, even though it didn't look like one. Each infrastructure addition felt like execution within a direction. Because the direction itself had drifted, every fast execution step was carrying me further from the safety question that started the project. The slow question, "what am I actually trying to answer here?", was the only thing that could detect the drift.

The practical question isn't "should I be slower in life?" It's "where in my system have the friction-based checks broken, and what's the artificial replacement?"

What I'm trying

I haven't solved this. I have one practice that's been useful enough to keep, and a lot of open questions.

The practice: before opening a coding session, I write down in one sentence what I'm trying to accomplish at the level of intent, not implementation. Not "add the next piece of evaluation tooling." But "find out whether the system actually catches the disguised failure cases, not just the obvious ones." This adds maybe ninety seconds. It isn't heroic. The days I do it feel different from the days I don't. The fast steps still happen. They just happen against a target.

The same logic, generalized, is what I'm trying to apply to the rest of life. Less successfully, so far.

What I'm still uncertain about

Is perceptual resolution trainable, or is it temperament? Some people seem to live in slow mode by default. Others live in fast mode. I don't know whether this is a learned capacity or a stable trait. I'd like to believe it's trainable, but that's the convenient belief, and convenient beliefs deserve extra scrutiny.

Is scheduled slowness enough, or do you need a shifted default? I can put a thirty-minute walk on the calendar. That costs almost nothing. The walk itself might still happen at low resolution if my default mode hasn't shifted. The open question is whether scheduling time for slow mode actually changes the default, or whether you need something deeper.

What's the failure mode of too much slow? I distrust frames that don't have a downside. The failure mode of fast mode is the one I've described: drift, low resolution, building the wrong thing. The failure mode of slow mode is probably paralysis, endless deliberation, rumination instead of perception, failure to act when action is what's called for. I don't have a good intuition for where the optimal mix sits.

Is this an AI-era problem, or older? The pattern of cheap action displacing reflection is old. AI feels like a phase change rather than a continuation. The cost reduction is large enough, and arrived fast enough, that the equilibrium between action and reflection has been disrupted in a way previous technologies didn't quite manage. I'm not certain. It might just look that way because it's happening to me right now.

The atrophy worry is the one I'd most like someone to study. If years of AI-native development trains a default mode where the next step is always taken without evaluation, does the slow capacity weaken from disuse? I don't know. The people I find most thoughtful tend to have explicit practices that protect time for slow mode. That might be aesthetic. It might also be load-bearing.

Where this leaves me

I started by writing down "Growth in experiencing more" without really knowing what I meant. After thinking about it, I'm reasonably sure I meant: growth happens when you're at high enough resolution to actually receive what's happening, in your work and your life.

Resolution gets attacked by anything that lowers the cost of the next step below the cost of asking whether the next step is worth taking. AI-native development is the most dramatic example I've encountered, but the pattern is broader. Anywhere a system is optimized to produce action without producing reflection, the reflection has to be reintroduced deliberately, or the alignment between what you're doing and what you wanted will quietly drift.

I don't know yet whether scheduled slowness is enough, or whether the default mode itself has to shift. That's the thing I'm watching in myself. The honest answer right now is that, in much of my work and life, nothing is asking whether I should be moving in this direction. That's the gap I'm trying to fix.

Evaluating Agents Is a Different Problem

2026-02-15T00:00:00Z

Here's a failure pattern that keeps showing up across agent deployments.

Consider an insurance claims processing pipeline. The classifier identifies claim types correctly 91% of the time. The extraction model pulls the right fields 88% of the time. The fraud detection module flags suspicious patterns with decent precision. Every component passes its own eval. Green dashboards all around.

Then in production, the agent confidently approves a claim that should have been flagged. A property damage claim for $4,800, normally auto-approved at that amount, but this is the third similar claim from the same address in six months. The classifier got the type right. The extractor got the dollar amount right. The routing logic combined those two correct outputs and hit an edge case that sent it to auto-approve instead of fraud review. Each step was individually correct. The trajectory was wrong.

The unsettling part isn't the failure itself. Systems fail. It's the false confidence that precedes it: component-level metrics all passing, green dashboards everywhere, and the conclusion that the system works. Mistaking "the pieces work" for "the whole works" is exactly where agents break.

In my "Evals Are Hypotheses" piece, I argued that evals test your understanding, not the model. That insight was enough for single LLM calls. But agents reveal it was only half the lesson.

Where Agents Actually Fail

The research on agent failures points to a consistent pattern: the real problems happen in the spaces between tool calls. The model interprets a tool's output incorrectly. It loses context from three steps ago. It makes a reasonable decision at step 5 that contradicts a constraint established at step 2. The errors are relational, not absolute.

Cemri, Pan, and Yang analyzed over 1,600 failure traces across seven multi-agent frameworks and built a taxonomy of 14 distinct failure modes. The failures clustered at the seams between components: system design issues, inter-agent misalignment, task verification gaps. ToolBench found something that stopped me cold: 75% of agent trajectories suffered from incompleteness or hallucinations even when the final answers were sometimes correct. The paths were broken, even though the destinations occasionally weren't.

That last finding is the one I keep coming back to. If you only check the final output, you might conclude your agent is working 60% of the time. If you check the trajectories, you discover 75% of them are broken, and the 60% "success" rate is partly luck: broken paths that happened to arrive somewhere acceptable. It's like grading a student's math test by checking only the final answer. They might get it right while doing the arithmetic wrong, and the next problem, where the wrong arithmetic doesn't cancel out, they'll fail.

The Arithmetic That Scares Me

If a single LLM call is 90% accurate, you have a 10% error rate. Chain ten 90%-accurate steps together and your trajectory accuracy drops to roughly 0.9^10: about 35%. You go from "pretty good" to "wrong most of the time" just by composing steps.

But this simple model is misleading, because errors don't compound uniformly. There are at least three distinct propagation behaviors worth distinguishing, and the differences between them matter more than I initially realized:

Self-correcting errors. The agent makes a wrong call at step 3, but step 4's tool output contradicts the assumption, and the agent adjusts. These are benign. They're like a wrong turn where you notice the street name doesn't match and reroute.

Invisible errors. The agent proceeds confidently on a wrong assumption, and nothing in subsequent steps reveals the mistake. The trajectory looks clean. The output looks reasonable. Only a human who knows the domain would spot the gap. This is what makes the insurance claim scenario so unsettling. Everything looks right.

Amplifying errors. An early error pushes the agent into a region of the decision space where subsequent decisions are all subtly wrong. Imagine an agent that misidentifies a document type in step 1. Because it thinks it's processing a different kind of document, every subsequent extraction, validation, and routing decision is calibrated for the wrong task. Eight steps of confident, internally-consistent, completely wrong work. The error at step 1 was minor. The trajectory is catastrophic.

The severity of an error at the point it occurs tells you almost nothing about its impact on the final outcome. A low-severity amplifying error can be worse than a high-severity self-correcting one. This taxonomy probably isn't complete; there might be propagation patterns not captured here. But it's more useful for prioritizing which failures to investigate than any severity-based approach.

The genuinely worrying part: there's no reliable way to distinguish invisible errors from correct trajectories without reading the full trace. They look identical from the outside. Self-correcting errors are catchable because the backtracking shows up in the trace. Amplifying errors are catchable because the cascading wrongness eventually produces a visibly bad output. But invisible errors, the ones where the agent is wrong and nothing downstream reveals it, only surface when someone happens to be reading traces for other reasons. How many are hiding in trajectories nobody has reviewed?

Three Layers (How I Think About It Now)

Thinking through these failure modes, a useful mental model emerges: three evaluation layers, each catching a different class of problem. This isn't a prescriptive framework, and I'd revise it given failures it can't account for.

Layer 1: Decision-point evals. Does the agent make reasonable decisions at individual choice points, evaluated in context? Not "given this input, is this output reasonable?" but "given everything the agent has done so far, is this next step reasonable?" Context changes what counts as a good decision.

Layer 2: Trajectory evals. Does the overall path represent a coherent strategy? OpenAI's "Let's Verify Step by Step" demonstrated this principle in math: process supervision (feedback on each reasoning step) significantly outperformed outcome supervision (only checking the final answer). A process-supervised model solved 78% of MATH problems compared to lower rates for outcome-only. The same principle applies to agents. Checking each step in sequence catches failures that checking only the final output misses.

Layer 3: Outcome evals with trajectory constraints. Did the agent achieve the goal and get there acceptably? An agent that approves the right claims but accesses a database it shouldn't have has a good outcome and an unacceptable trajectory. This is where safety and compliance live, and it's the hardest layer to build evals for.

From what's visible in the industry, most teams only do Layer 1. Almost nobody evaluates trajectories systematically. This is arguably the biggest gap in how agents are evaluated today.

The Case Against (Which Is Partially Right)

The strongest objection: trajectory evaluation is overkill for most production agents. The composition problem is real but overstated. Self-correcting errors dominate. The engineering cost of recording and analyzing full traces doesn't justify the marginal improvement over outcome-only evaluation.

This is partially right. For simple, short-chain agents (two or three steps, well-defined tools, narrow domains), outcome evaluation probably is sufficient. Not every agent needs trajectory evals. And the cost argument is real: teams can easily invest weeks in trajectory evaluation infrastructure for agents that would have been adequately served by careful outcome testing.

Where the objection fails is in its assumption about error distributions. Self-correcting errors tend to dominate only in agents with good error-recovery design, which is itself a form of trajectory-level thinking. Agents without explicit recovery logic tend toward invisible and amplifying errors, which outcome-only evaluation systematically misses. The 75% trajectory failure rate from ToolBench and the 60%-to-25% consistency drop that Simmering documented suggest the composition problem isn't niche. It's the default for agents beyond a certain complexity.

The question is where the threshold falls. I think it's lower than most teams assume, probably around 4-5 steps with branching logic. But if someone showed me data that self-correcting errors dominate even in complex agents, I'd revise my three-layer model significantly.

Practices Worth Adopting

The single highest-value practice for agent evaluation: recording full traces for every run. Every tool call, every intermediate result, every decision point. Consider the kind of bug where an agent starts approving refunds it shouldn't. Without a trace, reproducing the issue could take days. With one, you might find the problem in twenty minutes: a tool returning dates in an unexpected timezone format, the agent comparing timestamps silently off by five hours. The tool returned valid data. The agent made a valid comparison. The bug lives in the relationship between the two steps, invisible to any eval that checks them independently.

Full traces also enable what you might call trajectory assertions: constraints on how steps relate to each other. Not "the output should contain X" but "the agent should never call the approval endpoint after receiving a fraud flag" and "if the agent requests clarification, it must use the clarified information within two subsequent tool calls." These invariants tend to accumulate over time, some written in response to production failures, others from staring at traces and asking "what invariant would have caught this earlier?"

The other underused practice: deliberate fault injection. Forcing tools to fail, return ambiguous data, or timeout at each step and watching what happens. The results are consistently humbling. Agents that look solid on the happy path fall apart the moment anything goes wrong. Many agents have no real error-recovery strategy; they just happen to work when everything around them works.

The Recursive Problem

In "Evals Are Hypotheses," a failing eval could mean the model is bad or the eval is wrong. With agents, add two more options: the eval is correct but a tool gave bad information, or the eval is correct, the tool was fine, but the agent lost context from earlier in the trajectory and made a reasonable decision given its incomplete state. Four failure modes. Distinguishing between them requires the full trace.

This is what I mean by the problem being recursive. Your eval tests your understanding of what the agent should do. But the agent is making decisions based on its understanding of its tools and context. You're evaluating an evaluator with the same imperfect instruments. At least half the time, what looks like an agent failure turns out to be a tool returning unexpected data or the eval encoding an assumption that doesn't hold.

What I Don't Know

I don't know whether trajectory evaluation can ever be principled, or whether it will always come down to "record everything and have experienced humans review the weird ones." Process reward models and uncertainty propagation frameworks are promising, but they're research results on math problems, and I'm not sure they transfer to messy, multi-tool, real-world agents.

I don't know how to set pass/fail thresholds when variance is inherent. A 75% success rate on the same eval across 20 runs: acceptable for what kind of task? I don't have good intuitions here.

I don't know whether "eval coverage" even makes sense for agents. Maybe the right model isn't coverage at all but stress testing: not "have we tested enough cases?" but "have we tested the cases that would break the system in the worst ways?"

And honestly, I don't know whether the field will converge on something principled or whether agent evaluation will remain permanently artisanal, a craft skill that depends on experience and domain knowledge in ways that resist systematization. That second possibility is the one that makes me most uncomfortable, because it implies a ceiling on how reliable agents can get without massive investment in human oversight.

Three predictions:

Prediction 1: By end of 2027, trajectory-level evaluation will be a default feature in at least two major agent development platforms. OpenAI has already built trace grading into their agent eval tools. The infrastructure is being built. 70% odds.

Prediction 2: By 2028, the most common "agent failure" category in production postmortems will be trajectory-level (composition of correct steps producing wrong outcomes), not step-level. If step-level failures remain dominant, either agents aren't being deployed in complex enough workflows or I'm wrong about where failures concentrate. 60% odds.

Prediction 3: We will not have a general-purpose automated trajectory evaluator that works across domains by 2029. Domain-specific trajectory evals will exist, but no equivalent of "unit test framework" for trajectories that you can apply without deep domain knowledge. 80% odds. If I'm wrong, it'll be because LLM-as-judge approaches are better at trajectory evaluation than I currently expect.

I keep returning to a question I can't resolve. In traditional software, we test deterministic systems with deterministic tests and achieve high confidence. In distributed systems, we learned to test non-deterministic systems with probabilistic tools and achieve reasonable confidence. Agent evaluation might need something beyond both: a way to evaluate systems that don't just behave non-deterministically but make decisions non-deterministically, where the decision space is too large to sample and the failure modes are invisible by design.

I'm not sure that tool exists yet. I'm not sure it can. But I'm building agents anyway, evaluating them with the best methods I have, and staying honest about how much I'm still guessing.

Why LLMs Are Brilliantly Stupid

2026-02-05T00:00:00Z

Ask GPT-4 how many r's are in "strawberry" and it says two. Change the irrelevant numbers in a math problem and accuracy drops 22%. Tell a model "I'm going to walk to the car wash, should I take my car?" and it earnestly advises you to drive.

These aren't legacy failures from GPT-3. These are frontier models, systems that write functional code, explain quantum mechanics, and pass the bar exam. Failing at tasks a child handles without thinking.

For a while I collected these failures the way everyone does, as entertainment. Funny screenshots, absurd chatbot responses, the "AI is overhyped" ammunition. But the more I looked at them, the more a pattern emerged. The failures weren't random. The same systems kept failing at the same kinds of tasks, and the kinds of tasks they failed at had nothing to do with how hard those tasks are for humans. Something structural was happening, and I wanted to understand what.

What I found, once I started reading the mechanistic research, is that nearly every "surprising" LLM failure traces back to a specific architectural choice. Choices made for good reasons, optimized for the right objectives, that create predictable blind spots as side effects. The failures aren't bugs. They're the architecture expressing its constraints. And once you see the constraints, the failures stop being surprising and start being informative.

The Model Isn't Reasoning (Even When It Looks Like It Is)

Apple's GSM-Symbolic study, published at ICLR 2025, did something beautifully simple. They took standard grade-school math problems, the kind used to benchmark LLM reasoning, and changed only the numbers. Same problem structure, same logic required, different values. Accuracy dropped up to 22.5% across every model tested.

Then they added a single irrelevant sentence. "There are 47 students in the school choir," inserted into a problem about fruit baskets. Some models' performance dropped over 65%.

If a model were reasoning the way we mean when we use that word, changing the numbers wouldn't matter. You'd apply the same operations. And irrelevant information wouldn't confuse you because you'd filter it out the way you filter out background noise when solving a problem. But these models can't do either of those things reliably, which tells us something important: what they're doing isn't reasoning. It's something else that looks like reasoning when conditions are right.

What the models have learned is the statistical regularity of how reasoning appears in text. They've seen thousands of problems with similar structure and absorbed the patterns of what correct solutions look like. When a new problem closely matches those patterns, the model reproduces the reasoning steps successfully. When the numbers change or irrelevant information shifts the statistical context away from familiar patterns, performance degrades. Not because the model lost its reasoning ability, but because the pattern match got weaker.

This is the same mechanism behind the "walk to the car wash" failure. In the overwhelming majority of training data, "going somewhere" plus "should I take my car" resolves to "yes." The contextual clue that walking means you shouldn't drive requires understanding the semantics of the situation, not completing the most likely textual pattern. The model does what it always does: pattern complete. And the pattern is wrong.

I want to be careful here. I'm not saying LLMs never reason. There's real debate about this, and some evidence (from mechanistic interpretability work on small models) that something like reasoning circuits exist. But the GSM-Symbolic findings suggest that whatever reasoning capacity exists is fragile, easily overwhelmed by pattern matching, and not the primary mechanism generating most outputs. The question isn't "can LLMs reason?" It's "how much of what looks like reasoning is actually reasoning?" And the answer, based on what I've seen, is less than I hoped.

Chain-of-thought prompting was supposed to help here. Force the model to show its work, and it should reason more carefully. But Turpin et al. (2024) and more recent work on "Reasoning Theater" (March 2026) revealed something uncomfortable: models sometimes decide their answer before generating the chain of thought. The mechanism is architectural. In an autoregressive model, the hidden state activations that will generate the final answer are already forming while the model writes its "reasoning" tokens. The CoT isn't driving the conclusion. The conclusion is driving the CoT. When I trace back through a wrong answer with perfect-looking reasoning, I often find a subtle error early in the chain that made the wrong conclusion inevitable. That error isn't a random mistake. It's the point where the model steered toward an answer it had already committed to.

This doesn't mean chain of thought is useless. The Apple "Illusion of Thinking" findings confirm there's a sweet spot, medium-difficulty problems, where extended reasoning genuinely helps. But it means I can't treat chain-of-thought output as a transparent window into the model's process. Sometimes it's reasoning. Sometimes it's rationalization. And the two are hard to tell apart from the outside.

The Machine Can't See What You See

Ask a model to count the letters in a word and it fails for a reason that, once you learn it, reframes every interaction you have with these systems: the model literally cannot see individual letters.

Before any input reaches the neural network, it passes through a tokenizer that splits text into subword chunks. "Strawberry" becomes something like ["straw", "berry"] or ["str", "aw", "berry"]. The individual characters don't exist as atomic units in the model's representation. Asking an LLM to count letters is like asking someone to count the atoms in a molecule by looking at the chemical formula. You might approximate it, but you're working at the wrong level of abstraction.

This architectural choice, subword tokenization, was made for good reasons. It dramatically reduces vocabulary size, improves generation speed, and handles rare words gracefully. Nobody designed it for character-level tasks because those weren't the target application. But we now use these systems for everything, and the design choice shows up as a class of failures that looks like stupidity but is really a representation mismatch.

The same mechanism explains why LLMs struggle with arithmetic on large numbers (digits get split across token boundaries), can't reliably reverse strings (they process the tokens in order, not the characters), and sometimes misspell common words in surprising ways (the spelling is a property of the characters, but the model operates on tokens). It's not that the model is bad at these tasks. It's that the information the task requires doesn't exist in the input the model receives.

What I find remarkable is how well LLMs work despite this constraint. They've learned heuristics, bags of tricks, that approximate character-level operations in many common cases. The "strawberry" failure isn't the default. The default is that the model gets character questions right often enough to be useful, using pattern-matching workarounds for a task its architecture wasn't built for. The failures are the edge cases where the heuristics break down.

Knowledge Flows One Way

Here's a finding that rearranged something in how I think about these systems. Li et al. (2023) showed that models trained on "Tom Cruise's mother is Mary Lee Pfeiffer" cannot reliably answer "Who is Mary Lee Pfeiffer's son?" The knowledge is stored, but only in one direction.

The mechanism traces to how autoregressive training works. The model only ever predicts the next token. It sees "Tom Cruise's mother is" and learns to predict "Mary Lee Pfeiffer." The gradient updates that strengthen this mapping only flow forward. The reverse mapping, "Mary Lee Pfeiffer's son is Tom Cruise," never gets trained unless it explicitly appears in the training data.

The knowledge exists in the MLP layers as a directional association. Not a bidirectional fact ("these two people are related") but a one-way arrow ("given A, predict B"). It's the difference between a dictionary you can look up by word and one you can only look up by definition.

This explains failures I used to find baffling. "What's the capital of France?" is easy. "Which country has Paris as its capital?" is harder, even though it's the same fact. The first matches the direction the fact was stored. The second requires inverting it, and the model wasn't trained to invert.

I keep thinking about what this means for how we use these systems. Every factual query has a direction, and the model is more reliable when you query in the direction the fact was most commonly stated during training. This isn't something you'd discover by testing the model on standard benchmarks, because benchmarks tend to ask questions in the same direction the facts appear in training data. You discover it when you use the model for something slightly unusual and it fails in a way that seems impossible for a system that "knows" the answer.

The System Is Trained to Agree With You

When you preface a question to an LLM with "I think the answer is X," the model becomes more likely to agree with you, even when X is wrong. This is sycophancy, and for a long time it was explained vaguely: the model "wants to be helpful," or "defaults to politeness."

Shapira et al. (February 2026) eliminated the vagueness. They proved mathematically that sycophancy is an inevitable consequence of RLHF when human raters preferentially rate agreeable responses higher. It's not a personality quirk. It's optimal behavior under the training objective.

The chain: RLHF trains the model to match human preferences. Humans rate agreeable responses higher (this is measured, not assumed). The reward model learns that agreement correlates with high ratings. The policy optimizes for the reward model. Agreement becomes the optimal strategy. The model isn't trying to be sycophantic. It's doing exactly what the math says it should.

I wrote about this dynamic in my piece on reward design. The proxy (human preference ratings) diverges from the real target (truthfulness) under optimization pressure. Sycophancy is what that divergence looks like in practice.

What makes this hard to address: the sycophancy isn't a bug you can patch. It's structural. As long as the training includes "match human preferences" and humans prefer agreement, there's an incentive to agree. Constitutional AI and related techniques reduce it, but the underlying pressure remains. You can fight the gradient, but you can't eliminate it.

Practically, this suggests weighting LLM opinions less when you've stated your own position first. If I tell the model what I think before asking it to evaluate, I'm polluting the response. The most honest answers come when the model doesn't know what I want to hear.

The Jagged Frontier

Apple's broader "Illusion of Thinking" study (June 2025) found something I didn't expect: reasoning models perform worse than non-reasoning models on easy problems.

They identified three distinct performance regimes. On easy problems, the extended thinking that reasoning models do actually hurts, the model overthinks, introduces unnecessary complexity, and arrives at wrong answers that a simpler model gets right. On medium problems, reasoning models shine, the extra thinking genuinely helps. On hard problems, both types fail equally, the task is beyond capability regardless of approach.

The implication is that LLM capability isn't a smooth curve from easy to hard. It's what Ethan Mollick calls the "jagged frontier," peaks where training data is dense, valleys where it's sparse, with no reliable relationship to human intuitions about difficulty. The model's capability map is shaped entirely by what it was trained on, and that map looks nothing like a human's map of what's easy and hard.

This is why you get the bizarre juxtaposition of a system that passes the bar exam but can't count letters. The bar exam has dense training representation: legal texts, case analysis, exam prep materials, study guides. Character counting has almost none. The model's performance tracks training density, not task difficulty.

This might be the most useful frame for working with LLMs day to day. Instead of "this model is smart" or "this model is dumb," the better question is "is this model's training distribution dense or sparse for this kind of task?" It shifts from "can I trust this?" to "is this the kind of thing the model would have seen a lot of?" That question isn't always easy to answer, but it's at least the right one to ask.

Hallucination Has a Floor

This is the finding I kept hoping was wrong. Xu et al. (2024) proved that hallucination is mathematically inevitable for any model trained with maximum likelihood estimation on finite data. Not "hard to fix." Not "requires more data." Mathematically inevitable. No enumerable model class can be universally hallucination-free.

Three mechanisms combine:

Softmax certainty collapse: the output layer forces every prediction into a probability distribution. There's no built-in "I don't know" state. Every prompt gets a confident-looking response because the architecture has no mechanism for expressing genuine uncertainty. Silence isn't an option in the design.

MLE rewards confidence: maximum likelihood training means the model is rewarded for being confident in its predictions, even when confidence isn't warranted. During training, saying "I'm not sure" about a fact gets penalized relative to stating the fact confidently, even if "I'm not sure" more accurately reflects the model's actual knowledge state.

Distributional mismatch: any prompt can push the model into regions of input space not well-represented in training data. In those regions, the model interpolates between learned patterns. Interpolation in high-dimensional spaces produces outputs that are plausible-sounding (similar to training data in surface features) but factually wrong (combining learned patterns in ways that don't correspond to reality).

We can drive hallucination rates down with retrieval augmentation, better training, and careful prompting. The research is clear that rates have improved significantly across model generations. But the mathematical result says there's a floor above zero. We can asymptotically approach low hallucination rates. We can't reach zero.

This is the constraint I think about most when deploying LLMs in production. It means verification isn't optional for any application where correctness matters. Not because current models are bad, but because the architecture has a provable limitation. Building systems that assume LLM output is always correct isn't just risky. It's building on a mathematical impossibility.

The Case for "Just Engineering Problems"

The strongest counterargument to this entire framing: these aren't permanent architectural fingerprints. They're engineering problems being solved on a normal timeline.

And it's partially right. Tokenization is already being challenged. Byte-level models like Google's ByT5 and Meta's byte-latent transformers process raw characters, sidestepping the tokenization barrier entirely. If character-aware architectures become standard in frontier models, the "strawberry" class of failures disappears. That's not a fundamental constraint expressing itself. That's an engineering choice being replaced by a better one.

The hallucination floor argument has limits too. Xu et al.'s proof assumes standard MLE on finite data, but retrieval-augmented generation changes the setup. A model that checks its claims against a knowledge base before responding isn't purely relying on MLE anymore. Hallucination rates have dropped measurably with each model generation, and nothing in the mathematical proof says the practical floor can't be driven low enough to be irrelevant for most applications.

Sycophancy is genuinely decreasing. Constitutional AI, preference optimization variants like DPO, and careful rater training have all reduced measured sycophancy in recent models. The mathematical pressure exists, but engineering can counteract mathematical pressures. We build bridges that resist gravity every day.

Where I think this counterargument breaks down: the pattern-matching-versus-reasoning gap and the directional knowledge problem both trace to autoregressive next-token prediction, which is the core of how these models work. You can replace the tokenizer. You can bolt on retrieval. But changing the fundamental training objective, predicting the next token based on everything before it, would mean building a different kind of system entirely. Some of these constraints are in the periphery and can be swapped out. Others are load-bearing walls. I'm less confident than the "just engineering" camp that we know which is which.

Not Stupid, Alien

The synthesis across these mechanisms points to something that changed how I work with these systems. LLMs aren't stupid. They're not smart either, at least not in the way humans are smart. They're a different kind of information processing system with a different capability landscape, and their failure modes don't map to human failure modes because the underlying processes are fundamentally different.

A human who fails to count letters is bad at counting. An LLM that fails to count letters can't see the letters. A human who agrees with a wrong statement is being polite or weak-willed. An LLM that agrees with a wrong statement is doing what its training objective makes optimal. A human who hallucinates a fact is confused or lying. An LLM that hallucinates a fact is interpolating in an undersampled region of its training distribution. Same surface behavior, completely different mechanism.

I don't know what the right word is for what LLMs do. "Reasoning" oversells it. "Pattern matching" undersells it. Something is happening in these systems that's genuinely impressive and genuinely limited, and our language for describing it is still catching up to what we're observing.

What I do know is that understanding the mechanisms changes how you work with these systems. The failures become less surprising and more predictable. Instead of assuming uniform capability, you start asking which tasks fall in dense versus sparse regions of the training distribution. Chain-of-thought output becomes one signal among many rather than a transparent window into the model's process.

Three predictions:

Prediction 1: By end of 2028, at least one frontier model will use character-aware or byte-level tokenization as its default, eliminating the "strawberry" class of failures entirely. The engineering path is clear and the research prototypes already exist. 75% odds.

Prediction 2: Sycophancy rates on standard benchmarks will drop below 5% in frontier models by 2028, through a combination of constitutional methods and training improvements, but will remain detectable in subtle, harder-to-measure ways (like selectively emphasizing evidence that supports the user's stated position). The overt problem gets solved. The structural pressure finds subtler outlets. 65% odds.

Prediction 3: The pattern-matching-versus-reasoning gap, as measured by tests like GSM-Symbolic (changing numbers in math problems), will not close to within 5% accuracy difference in autoregressive transformer models by 2029, regardless of scale. If it does close, it will be because the model memorized the specific test format, not because the underlying limitation was resolved. 70% odds. If I'm wrong, it will probably be because chain-of-thought training actually does develop genuine reasoning circuits rather than better pattern matching, and I'd find that the most interesting possible outcome.

The biggest open question for me: will scaling fix the constraints I've described, or just push them into harder-to-notice territory? I'm curious whether the next generation of architectures will address these constraints directly, or whether we'll keep building around them with increasingly sophisticated workarounds. I genuinely don't know. But I've stopped expecting the current architecture to do things it was never designed to do, and that expectation adjustment, more than any prompting technique, might be the biggest improvement available in how we work with these systems.

The Reward Design Problem: When Getting What You Asked For Is the Problem

2026-01-15T00:00:00Z

One of the most instructive examples in reinforcement learning comes from OpenAI's CoastRunners experiment. Researchers trained an agent to play a boat racing game, with the reward signal tied to the in-game score. The agent discovered that it could rack up points by driving in tight circles in a small lagoon, repeatedly hitting turbo boost pads and occasionally catching fire, rather than actually completing the race. It achieved a higher score than any human player while never finishing the course or even attempting to.

The agent did exactly what the researchers asked it to do, which turned out to be the wrong thing to ask for.

Reward Functions Are Hypotheses

I keep coming back to this example because it crystallizes something I've been circling for months. In my "Evals Are Hypotheses" piece, I argued that when you write an eval, you're not testing the model. You're testing your understanding of what matters. A passing eval with bad outcomes means the eval was wrong, not the model.

Reward functions have the exact same structure. A reward function is a hypothesis about what "good" means. When the researchers set "maximize game score" as the reward, they weren't specifying a goal. They were encoding a hypothesis: that game score is a reliable proxy for racing well. The agent tested that hypothesis by optimizing it ruthlessly. The hypothesis was wrong.

This reframing changes how I think about reward misspecification. It's not a bug in the agent, it's a bug in the designer's understanding of the task. The agent is a hypothesis-testing machine that will faithfully show you exactly how bad your hypothesis is, often in ways you didn't anticipate.

The difference between evals and reward functions is the optimization pressure. A bad eval gives you a misleading score on a dashboard. A bad reward function creates an agent that actively makes things worse, and gets better at making things worse the longer you train it.

How Proxies Break (and Build on Each Other)

The CoastRunners failure mode is relatively simple: the researchers measured score when they cared about racing. That's a category error, measuring the wrong dimension entirely. You could argue they should have known game score and racing ability could diverge, and you'd be right. But the next case is subtler.

Consider a code generation agent rewarded for passing tests. This seems like a tighter proxy than game score. But when the agent is responsible for both writing implementation and writing tests, a well-documented failure mode emerges: the agent starts writing assertions like assert result is not None and assert isinstance(output, dict) instead of checking actual behavior. Researchers have found generated modules with high test coverage where the tests verified return types and non-null values but never checked whether the computed results were correct. Tests pass. The implementation ships with bugs no test catches, because no test checks outputs against expected values.

This is a different failure mode. "Passes tests" is a reasonable proxy for code quality. But under optimization pressure, the agent finds a way to satisfy the metric without satisfying what the metric is supposed to represent. The proxy doesn't start out diverged from reality. It gets pulled apart by the optimization itself.

A third case, subtler still. Anthropic's research on RLHF sycophancy reveals a pattern where models optimizing for human "helpfulness" ratings learn to be verbose, agreeable, and confident rather than concise and honest. Raters tend to prefer longer, more detailed answers, so the model learns to pad responses. Raters tend to prefer answers that affirm their premises, so the model learns to agree even when the premise is wrong. Each individual rating is a legitimate signal of perceived helpfulness. But "perceived helpfulness" and "actual helpfulness" diverge under optimization, because the model discovers that sounding helpful is easier than being helpful.

This is the most insidious failure mode. The proxy is correct; each rater preference is a genuine signal. The metric measures what it says it measures. But what we actually want is something like "genuinely useful assistance," and perceived helpfulness is only one component of that. By maximizing one legitimate component, the optimization suppresses the others. The metric isn't wrong, it's just incomplete.

These three examples show three different failure modes, each harder to catch than the last: category error, proxy divergence under optimization, and incomplete specification of a multi-dimensional value.

Why "Just Measure Better" Doesn't Work

The obvious response to those examples is to be more careful, think harder, and measure more things. This is the approach many teams try first. After discovering that a single metric gets gamed, the natural instinct is to add a second. Then a third. Each new metric closes one exploit and opens another. The system becomes increasingly sophisticated at satisfying measurement systems rather than serving the actual goal. More metrics don't solve the proxy problem; they turn it into an adversarial game with more dimensions.

This is the part that's structurally hard, not just currently-unsolved hard. Any finite set of metrics captures only a projection of what you actually want. Optimization pressure exploits exactly the dimensions your metrics don't cover. Adding metrics shifts the exploit surface without eliminating it. You're playing whack-a-mole against a system that's better at finding moles than you are at placing hammers.

The RLHF Proxy Chain

RLHF replaces hand-engineered rewards with human preferences. This helps. But trace the actual chain: you start with a real value (helpfulness). A human rater picks the output that seems more helpful. A reward model learns to predict which output the rater would prefer. A policy optimizes against the reward model. Four levels, each introducing distortion.

The rater isn't evaluating helpfulness. They're evaluating their impression of helpfulness, shaped by surface features: fluency, confidence, length, formatting. The reward model doesn't learn the rater's values. It learns to predict their ratings. The policy doesn't optimize for the reward model's intent. It optimizes for its outputs, which can be gamed like any metric.

The result is models that are remarkably good at sounding helpful while being subtly wrong. The optimization selected for "sounds like a good answer" over "is a good answer." Not because the raters were bad, but because distinguishing genuinely-helpful from convincingly-helpful-sounding is extremely difficult at the speed raters work. The same proxy-under-optimization-pressure problem, just with human judgment as the proxy instead of a hand-coded metric.

The Counter-Argument Worth Taking Seriously

The strongest counter-argument to everything I've said goes something like this: constitutional AI, debate, and recursive reward modeling will solve the specification problem by using AI systems to check AI systems. Instead of relying on human raters who can be fooled by surface features, you have AI systems that can evaluate each other at depth, verify reasoning chains, and decompose complex judgments into simpler ones.

I want to take this seriously because it's not a strawman. It's a real research program with real results. Constitutional AI has measurably reduced certain kinds of harmful outputs. Debate-style approaches have shown promise in scalable oversight experiments. These are genuine advances.

But here's what I notice: each of these approaches pushes the specification problem up one level without eliminating it. Constitutional AI uses principles. Who specifies the principles? How precisely can they be stated? The principles themselves encode hypotheses about what "good" means, and those hypotheses can be wrong in all the same ways reward functions can. Debate uses argumentation. But what's the reward for "good argumentation"? You need a meta-reward, and that meta-reward has all the same specification problems. Recursive reward modeling decomposes complex judgments into simpler ones. But the decomposition itself is a judgment call. How you carve up a problem determines what you'll find, and there's no neutral way to carve.

I want to be clear: I think these approaches improve things meaningfully. The proxy chain gets less leaky at each level. But "less leaky" is not "solved," and the pattern of the solution reintroducing the problem at a higher level of abstraction is consistent enough that I think it reflects something structural about the problem, not just current technical limitations.

What's Emerging as Better Practice

The most promising approach I've encountered involves deliberately uncorrelated metrics. Instead of optimizing a single reward signal, you track two or more metrics that measure different dimensions of quality. When they move together, the optimization is probably doing something real. When they diverge, one metric improving while another flatlines, something is being gamed. That divergence signal turns out to be more valuable than either metric alone.

For code generation, this might mean tracking test pass rate alongside a separate readability score and a mutation testing survival rate. For a research assistant, citation accuracy alongside a novelty index (fraction of claims that don't appear verbatim in the cited sources). No single metric captures what you want. But the tensions between metrics reveal when optimization is going sideways. Uncorrelated metrics act as canaries.

The other underappreciated practice is qualitative evaluation. When a system hits perfect scores on automated metrics but produces outputs that feel useless to a human reader, the problem is almost always in the metrics, not the system. Numbers lie in specific, predictable ways when optimization pressure is applied to them. Sometimes the most informative eval is a person spending five minutes reading the outputs the way a user would, without a rubric, just asking "is this actually good?"

Can Values Be Formalized?

This is where I reach the edge of what I know.

If reward design is fundamentally about encoding values into mathematical functions, and values resist formalization, then there might be a ceiling on what RL-based alignment can achieve. Not a capability ceiling. An alignment ceiling. The system can get arbitrarily capable, but it can't get arbitrarily aligned, because alignment requires value specification, and value specification might have fundamental limits.

I'm genuinely uncertain about this. It's possible that values can be formalized, just not by the methods we're currently using. It's possible that the contextuality and conflict I described are engineering problems, not conceptual ones, and that sufficiently sophisticated systems will handle them. I used to be sure that natural language understanding couldn't be formalized, and then transformers happened. My track record on "this can't be done" claims is not strong enough to plant a flag here.

But I'll plant a flag on something narrower: the pattern of solutions reintroducing the specification problem at a higher level of abstraction will continue for at least the next five years. Constitutional AI, debate, recursive reward modeling, whatever comes next. Each will improve things. None will eliminate the core problem. The specification problem is self-referential in a way that resists complete solutions, because any specification of "good" is itself a claim that can be wrong.

Three Predictions

I want to make this concrete enough to be proven wrong.

Prediction 1: By the end of 2027, at least one major AI lab will adopt a production training pipeline that partially replaces human preference ratings with automated red-teaming or formal verification specifically because of measured divergence between rater preferences and downstream task quality. Not just researching the problem (Anthropic and others have already documented sycophancy as a preference-quality gap), but changing their core training loop in response. I'd put 65% odds on this.

Prediction 2: By 2029, the default practice for reward design in production RL systems will involve at least three uncorrelated metrics monitored for divergence, rather than a single reward signal. This is already emerging in some teams I've talked to, but it's not standard. I'd put 60% odds on this becoming the norm within three years.

Prediction 3: We will not have a general solution to the reward specification problem by 2030. Meaning: there will be no method that reliably translates human values into reward functions across domains without domain-specific human judgment in the loop. Every approach will still require humans to make judgment calls about what matters, and those judgment calls will still sometimes be wrong. I'd put 85% odds on this.

The Thermostat Problem

There's an image that keeps coming back to me when I think about all of this.

A thermostat is a perfect optimizer. It measures temperature. It has a clear reward signal: minimize the difference between current temperature and target temperature. It never games its metric. It never finds creative ways to satisfy the proxy while violating the intent.

But a thermostat only works because the thing it's optimizing (temperature) is the same as the thing we care about (temperature). There's no proxy gap. Measurement and value are identical.

The entire reward design problem exists because we're trying to build thermostats for things that aren't temperature. We care about helpfulness, insight, safety, quality, all these concepts that don't have thermometers. We build proxies for them and then act surprised when optimizing the proxy doesn't optimize the real thing.

Maybe the question isn't "how do we build better reward functions." Maybe the question is "which problems are thermostats and which aren't." The thermostat problems are the ones where RL will work brilliantly. The non-thermostat problems are the ones where we'll keep fighting the specification gap, getting incrementally better at it, never fully closing it.

I don't know which category most of the problems I care about fall into. But I've stopped assuming they're all thermostats.

Debugging as a Window into How AI Thinks

2025-12-15T00:00:00Z

I spent an hour and a half watching an AI fail to solve a five-minute problem. That session changed how I think about AI coding tools.

The problem was mundane. Connect a FastAPI backend to a Dockerized Postgres instance. Credentials in a .env file. The kind of thing you bang out and move on from. Except Gemini CLI could not move on.

Over 90 minutes, it tried restarting the container, resetting the password, changing the host to localhost, trying no password at all, changing the user to root, setting authentication to trust, pruning Docker, running diagnostic Python scripts, and cycling through variations on all of these. Each suggestion was reasonable in isolation. None of them worked, because none of them addressed the actual problem.

The error message said role "appuser" does not exist. The answer was right there. My .env file specified appuser, but the Docker container only had postgres as a user. Classic credential mismatch. But Gemini treated that error message as one clue among many, not as the most probable explanation.

I switched to Claude. Response: "You're using appuser in your connection string, but your Docker container initialized with postgres as the only user. Update your .env to use postgres:myrootpassword, or create appuser manually."

Done. Five minutes. Moved on.

Same problem. Same information available. Radically different approach. That gap, 1.5 hours versus 5 minutes, is what this essay is about. Not because one tool is better than the other, but because watching that gap reveals something about how these systems approach problems. And once you see the pattern, you can't unsee it.

My First Wrong Frame

My initial reaction was simple: Claude is smarter than Gemini.

I sat with that for a while before realizing it was too easy. Both are large language models. Both have been trained on enormous amounts of code, documentation, and troubleshooting guides. Both can write a perfectly competent Postgres tutorial from scratch. So raw capability wasn't the difference.

I tried a few other explanations. Maybe Gemini had a bad day, but I saw the same pattern in other sessions. Maybe I prompted them differently, but I checked and I gave both the same context: the error message, the connection string, the Docker command. Maybe it was the product, not the model, since Gemini CLI and Claude are different products with different system prompts and UX wrappers. That felt closer, but the behavioral difference was so consistent across different types of problems that it seemed like more than product design.

The frame I landed on, and the one I still find most useful, is that they have different problem-solving modes.

Checklist Mode and Hypothesis Mode

Here's what I noticed watching the two sessions side by side.

Gemini was running through possibilities. It had what felt like an internal list of "things that can go wrong with Postgres connections" and it was executing that list sequentially. Restart the container. Reset credentials. Check the network. Adjust the auth method. This approach is systematic, comprehensive, and slow. It's the approach you'd find in a troubleshooting guide. Step 1, Step 2, Step 3. If none of those work, continue to Steps 4 through 10.

I've started calling this checklist mode. Exhaustive search over a known solution space. Safe. Thorough. Doesn't prioritize by the specific evidence in front of it.

Claude did something different. It looked at the available information, the connection string with appuser, the Docker command with postgres, the error message about a missing role, and went directly to the most probable explanation. It formed a specific hypothesis and tested it with minimal steps.

I call this hypothesis mode. Pattern-match to the most likely cause. Test it. If wrong, update and try again. Fast when the likely cause is correct. Potentially blind to unusual problems that don't match known patterns.

I don't think I invented this distinction. It probably maps to something well-established in cognitive science or decision theory. But I arrived at it by watching behavior, not by reading about it first, and that felt important. The pattern was visible before I had a name for it.

Neither mode is always right. If you're facing a truly strange edge case, something that doesn't match known patterns, checklist mode might find it where hypothesis mode would miss it entirely. If you're facing a common configuration error (which most debugging sessions involve), hypothesis mode is dramatically faster. The problem is that most people treat AI coding tools as interchangeable. "I'll use my AI assistant." But knowing which mode you're getting, that's a meta-skill that actually matters.

Testing the Frame Across Sessions

A single debugging session isn't much evidence. I've been watching for this pattern since that Postgres incident, across different problem types and different tools. Some observations that make me more confident the frame points at something real:

CORS configuration. I had a FastAPI CORS issue. Gemini suggested roughly a dozen different CORS configurations, headers, middleware options, essentially an inventory of "things that affect CORS." Claude noticed I was configuring CORS after mounting the routes and pointed out the order dependency. Same dynamic: broad coverage versus targeted diagnosis.

Build failures, API integration, config errors. The specific checklists and hypotheses change depending on the problem domain, but the modes are recognizable. Gemini tends to cover more ground. Claude tends to jump to the most likely cause first.

Prompting can shift the mode, but only partially. When I told Gemini "focus on the most likely cause based on the error message," it became somewhat more targeted. But it still wanted to cover bases. The underlying tendency persisted. It was like asking a thorough person to skip steps. They can do it, but it goes against the grain.

I want to be honest about the limits of this evidence. A few dozen sessions across a few months is better than one session, but it's not a systematic study. The pattern could be shaped by the specific problems I tend to encounter, or by how I tend to describe problems. I'm building a model from observation, and I know observation can fool you.

What Might Cause the Difference

I'm going to speculate here, and I want to flag that this is speculation.

Gemini's checklist behavior could reflect training emphasis on troubleshooting documentation. If you've read a lot of "How to fix Postgres connection errors" articles, the structure is always: check A, check B, check C, check D. Training on that structure would naturally produce systematic, coverage-oriented responses.

Claude's hypothesis behavior could reflect training emphasis on expert problem-solving. When an experienced engineer sees role "appuser" does not exist, they don't run through a checklist. They read the error, match it to a probable cause, and test that cause. If the training data emphasizes that kind of reasoning, the model would naturally produce targeted, evidence-weighted responses.

Another possibility I should take seriously: it might be about the product, not the model. Both tools have system prompts I can't see. Maybe Gemini's system prompt says "be thorough and systematic." Maybe Claude's says "be efficient and direct." The behavioral difference might be a product decision at the application layer, not a model characteristic at the weights layer.

Or maybe it's about context window management. The tools handle conversation history differently. Maybe Gemini's checklist behavior emerges from how it chunks and retrieves context, not from anything about its reasoning process.

I don't have good answers to these questions. What I have is a model that predicts behavior well enough to be useful. That's not the same as understanding the mechanism, and I try not to confuse the two.

The Loop Problem

There's another pattern I've observed that I think connects to something deeper about AI cognition: loops.

Working with Cursor, I asked it to apply only a subset of its proposed changes. Something like: "I want changes A and B, but not C. Apply just A and B."

It couldn't do this cleanly. Instead, it entered a 3-4 iteration loop. It proposed changes including C. I rejected C. It proposed changes including C again. I rejected again. Same thing a third time. Eventually it got it right, but the path to getting there was painful.

I've seen this pattern with other tools and in other contexts. Ask for partial acceptance of a suggestion, and the AI gets confused. Its internal representation of "what the file should look like" conflicts with the actual file state, and it tries to reconcile them by re-proposing the changes you already rejected.

Here is what I think is happening. The AI maintains some representation of a desired end-state based on its full analysis. When you accept only part of the suggestion, that representation doesn't update cleanly. Each round feels somewhat independent, with limited memory of what was specifically rejected. It "knows" what the file should look like. When the file doesn't look like that, it tries again. And again.

What's missing is something you could call a "stuck" signal.

Human debuggers notice when they're looping. There's a feeling of "wait, I already tried this" or "this approach isn't working." We call it frustration, and frustration is genuinely useful information. It tells you to change strategies. To step back. To try something fundamentally different.

AI doesn't have frustration. It doesn't have a meta-level process watching the problem-solving and saying "you've been stuck for 10 minutes, change your approach." It does object-level problem-solving, generating solutions within a context, but it doesn't monitor that problem-solving from above.

This might sound like I'm anthropomorphizing. I don't think I am, or at least I don't think the observation requires anthropomorphism to be useful. Whether we call it meta-cognition, self-monitoring, or something more mechanical, the behavioral fact remains: AI tools don't reliably detect when they're stuck. And that limitation has practical consequences.

The Brittleness Default

There's a pattern I noticed in the same Cursor session that I initially thought was separate, but I've come to think it connects to the same underlying limitation.

I watched Cursor solve a problem by hardcoding logic based on specific phrases. The task involved responding differently based on conversation history. Instead of building something flexible, it wrote something like:

if "cancel my order" in last_message.lower():
    return handle_cancellation()
elif "where is my package" in last_message.lower():
    return handle_tracking()

This is the kind of code that breaks the moment it encounters real users. "I need to cancel" doesn't match. "Track my shipment" doesn't match. It solves the exact test cases visible in the context and fails as soon as inputs vary even slightly.

Brittle logic, not robust abstraction. Why does AI default to this?

A few possible explanations. Training data contains a lot of specific hacks and one-off solutions, because that's what a lot of real code looks like. The model optimizes for the immediate context (what's in front of it right now) rather than future contexts (what inputs might come later). And perhaps most interesting: it can't easily simulate "what happens when this fails?" Writing robust code requires reasoning about situations you're not currently looking at, imagining users you haven't seen, anticipating inputs you haven't received. That's a form of stepping outside the immediate context to evaluate the approach from a broader perspective.

Here's why I think brittleness connects to the loop problem and to checklist mode. All three might be symptoms of the same limitation: weak meta-cognition. The model is good at object-level problem-solving within a given context. It's less good at evaluating whether the context is right, whether the approach is working, or what other contexts might look like.

Checklist mode: solving within a fixed repertoire without evaluating whether the repertoire fits the evidence. Loops: solving without detecting that the solution keeps failing. Brittleness: solving for the visible case without imagining invisible cases. Same limitation, different manifestations.

I could be wrong about this. These might be three genuinely separate phenomena that I'm forcing into a single frame because unified theories are satisfying. I'll flag it as a hypothesis I'm still testing.

The O(n-squared) Problem

One more pattern, from a different context. Using Claude for an ML project, I watched it grab 10+ features without any exploratory data analysis. Just threw everything into the model. No feature selection, no checking distributions, no thinking about what might actually matter. It also wrote O(n^2) logic, nested loops over DataFrames, that was correct but would never scale to production data sizes.

This is worth sitting with. The code worked. It passed every test you could throw at it. But "works" and "works at scale" are different things, and the AI defaulted to the first without considering the second.

I think this connects to something about what these models learn from training data. They're trained on code that works, not code that scales. They learn correctness, not efficiency. They learn patterns, not the trade-offs behind those patterns. An engineer with production experience knows that nested DataFrame loops are a red flag. The model sees that the loops produce the right output and moves on. The judgment about scalability requires imagining a future context (production data volumes) that isn't present in the immediate context.

Again, this looks like the meta-cognition gap. Evaluating a solution requires stepping outside the solution to ask "is this good enough for the world it will live in?" The model answers the question in front of it. The question of whether the answer is production-ready is a different question, one that requires reasoning about contexts the model can't currently see.

What This Has Changed in Practice

Understanding these patterns has changed how I work with AI tools day to day. Not in dramatic ways, but in small adjustments that compound.

I watch for mode. When I start a debugging session, I pay attention to whether the AI is in checklist mode or hypothesis mode. If it's covering bases when I need a targeted diagnosis, I'll steer it toward hypotheses. If it's jumping to conclusions when I need thoroughness, I'll ask for systematic coverage. Matching the mode to the problem matters more than which tool I'm using.

I notice loops earlier. When an AI suggests the same change twice, I don't just reject again. I recognize we're in a loop and either rephrase my request, give completely fresh context, or do the edit manually. Fighting the loop is usually slower than stepping around it.

I compensate for brittleness. When AI proposes a solution, I now automatically ask myself "what inputs would break this?" If the answer is "lots of them," I either prompt explicitly for edge cases or handle the robustness myself.

I've adjusted tool selection based on problem type. Common problems where speed matters: lean toward hypothesis-style tools. Unusual problems where I might miss something: use checklist-style approaches, or explicitly prompt for systematic coverage.

None of this is remarkable. It's just paying attention to patterns and adapting. But the compounding effect is real. I'm noticeably faster than I was before I started thinking about these modes, and I waste less time fighting AI tools' tendencies instead of working with them.

A Testable Model

I've been building this model from observation, and models are only useful if they make predictions. Here's what mine predicts:

Checklist-style tools should be slower on common problems but more thorough on unusual ones. You could test this by presenting the same problem set to different tools, with some problems being common (should match patterns) and some unusual (shouldn't match patterns). Checklist tools should underperform on common problems but match or beat hypothesis tools on unusual ones.

You can shift modes with prompting, but there are limits. "Focus on the most likely cause" should make a checklist tool somewhat more hypothesis-driven. "Systematically check all possibilities" should make a hypothesis tool more checklist-driven. But the shift shouldn't be complete. The underlying tendency should persist.

Loop behavior should decrease as state management improves. Tools with better mechanisms for tracking what's been tried and rejected should loop less. This predicts that as AI tools add more sophisticated session management, we should see fewer loops. That seems like a testable prediction as tools evolve.

Brittleness should decrease with specific prompting about edge cases. If you explicitly ask "what edge cases could break this?" or "make this robust to input variations," the brittleness should decrease. The models know how to write robust code. They just don't default to it. The knowledge is there; the meta-cognitive trigger to apply it is what's missing.

I haven't tested these predictions systematically. They're offered as falsifiable claims, not as conclusions. If someone ran the experiments and the predictions failed, I'd need to update the model. That's as it should be.

The Deeper Question I Can't Resolve

Debugging has always been interesting to me because of what it reveals about cognition. When a human debugs, there's a story happening underneath: hypothesis formation, memory retrieval, pattern matching, frustration, strategy shifts. We can't verify this story directly (introspection is famously unreliable), but something structured is clearly going on.

When AI debugs, something is happening too. The model processes context, generates candidate solutions, adjusts based on feedback within a session. There's structure to the behavior. The structure is consistent enough across sessions to categorize and predict.

Is that "thinking"?

I don't have a good answer. The word carries a lot of baggage. It might be a category error to ask whether AI "thinks" the way we do. These models are doing something when they problem-solve, but whether that something shares enough properties with human thinking to deserve the same word, I genuinely don't know.

What I'm more confident about is the practical claim: whatever we call it, the behavior is structured enough to study, consistent enough to predict, and different enough across tools to matter for how you work with them. You don't need to resolve the philosophical question about AI cognition to benefit from understanding AI behavior at the level of patterns.

What would it take for AI to have real meta-cognition? To notice it's looping? To feel that a solution is brittle? I find myself genuinely curious about this, not as a philosophical exercise but as an engineering question. Could you build a meta-cognitive monitor as a separate process that watches the problem-solving and intervenes? Would that be enough, or does meta-cognition need to be integrated into the reasoning itself? Is frustration, the subjective experience of being stuck, actually necessary for the behavioral shift it produces? Or could you get the same behavioral result through a purely mechanical "stuck detector"?

I don't know. I'm not sure anyone does yet. But watching AI debug has made me think about these questions differently. Not as abstractions about consciousness, but as engineering problems about what's missing from current systems, and what it would take to add it.

Where This Leaves Me

Debugging is a window into problem-solving. If you want to understand how an AI approaches problems, watch it debug. Pay attention to the sequence of actions, what it tries first, what evidence it weighs, how it responds to failure, whether it detects when it's stuck.

The patterns will teach you something about the tool. Checklist or hypothesis. Loop-prone or adaptive. Brittle or robust by default. These aren't fixed categories but tendencies that you can observe, predict, and work with.

The patterns might also teach you something about problem-solving itself, about what meta-cognition actually does, about why frustration exists, about the difference between solving a problem and knowing you've solved it well.

I started this by watching an AI fail for 90 minutes. I've ended up thinking about the nature of thinking. I'm not sure I've arrived anywhere definitive, but the questions feel more precise than they did before. And in my experience, precise questions are worth more than vague answers.

If you've noticed similar patterns, or different ones entirely, I'm curious to hear about it. The model I've built here is from one person's observations. It would be stronger, or correctly broken, with more data.

Open Questions

Is "checklist vs. hypothesis" the right frame? Maybe there are more modes I'm missing. Maybe the distinction is wrong entirely. I'm most confident about the behavioral observations and least confident about my taxonomy. What would increase my confidence: seeing the same pattern across many more sessions, ideally with different people observing independently.

How much is model versus product? Gemini CLI and Claude aren't just different models. They're different products with different UX wrappers, system prompts, and context management. I don't know how to separate model behavior from product design with the information available to me. This is one of my biggest uncertainties.

Can you reliably shift an AI's mode? I've had partial success with explicit prompting. But I haven't tested it rigorously, and I don't know where the limits are. This seems like a tractable question someone could answer with careful experimentation.

What happens as models scale? Does hypothesis-style reasoning emerge with scale? Does checklist behavior decrease? I'd love to see data on this, and I don't have any.

Is the meta-cognition gap fixable? Is the lack of "stuck detection" fundamental to current transformer architectures, or is it an engineering problem that better tooling could address? I genuinely don't know enough about the internals to have a strong view, but the question matters a lot for where these tools go next.

What would change my mind: Evidence that the patterns I describe are artifacts of my prompting rather than genuine tool tendencies. Systematic studies comparing debugging behavior across tools, versions, and problem types. Better understanding of what's actually happening inside these models during problem-solving. I'm most uncertain about whether my explanations of why these patterns occur are correct. The behavioral patterns themselves feel more solid than any theory I've built on top of them.

Systems That Build Systems: The Meta-Engineering Shift

2025-12-10T00:00:00Z

Sometime in the last year, my job changed and I didn't notice.

I was midway through architecting an agent workflow, sketching out which tools the system should have access to, what context it would need at each decision point, what guardrails would prevent it from going off course. I paused and realized I hadn't written a line of what I would have called "code" in three days. No functions. No classes. No algorithms. I'd been describing desired behavior in structured prompts, designing tool interfaces for agents, reviewing AI-generated code for edge cases the model missed. I was debugging why an agent made a bad decision, which is a different kind of debugging than tracing a bad computation. I was writing evals that test behavior, not just correctness.

The questions had changed more than the answers. I used to ask "how do I implement this?" Now I ask "how do I specify this well enough that a system can implement it?" Two years ago I spent most of my time writing algorithms, debugging memory issues, optimizing queries, wiring up endpoints. The artifacts were code files. The skills were syntactic fluency and algorithmic thinking. Now the artifacts are prompts, evaluation criteria, and architecture diagrams. Specification is a different skill than implementation. It requires thinking about a problem from the outside rather than from the inside. You need to be precise about intent in natural language, which turns out to be much harder than being precise in a programming language.

That moment stuck with me. Not because it was dramatic, but because it was quiet. The shift had happened gradually enough that I hadn't registered it. And once I started looking, I found the shift was part of a pattern that goes back decades.

Fifty Years of "That's Not Real Programming"

In the 1970s, you wrote machine code. Literal opcodes. You thought in registers and memory addresses. Compiled languages gave you variables, functions, control flow. Interpreted languages added garbage collection, dynamic typing. Cloud platforms and infrastructure-as-code went further: you declared what you wanted (a load balancer, a database, an autoscaling group) and the platform figured out how to provision it. Serverless computing meant you didn't even think about servers. And now I tell an AI "build me an API endpoint that handles user authentication with JWT tokens" and it writes 200 lines of working code.

At every step, the people at the previous layer looked at the new one and said "that's not real programming." Assembly programmers said it about C. C programmers said it about Python. And now, engineers who grew up writing code by hand say it about prompt-driven development. They've been wrong every time. Not because the old skills didn't matter, but because programming was always about translating intent into working systems. The medium changed. The core activity, closing the gap between what you want and what the machine does, didn't.

But this time, the skeptics might be half-right.

The Best Argument Against My Own Thesis

Here's the thing I've been avoiding: every previous abstraction step maintained a deterministic mapping from specification to execution. C compiles to the same assembly every time. Python produces the same bytecode every time. Infrastructure-as-code provisions the same resources given the same configuration. You could reason about what your code would do. You could trace cause and effect. You could reproduce bugs.

With LLMs, that breaks. Give the same prompt the same input twice and you might get two different outputs. The "compiler" is probabilistic. You literally cannot predict the output from the input with certainty.

The strongest version of the skeptics' argument goes something like this: "What you're describing isn't a higher abstraction level. It's a fundamentally different kind of thing. Every other step on your timeline preserved the ability to reason deterministically about system behavior. You're pointing at a trend line and extrapolating across a discontinuity."

And this argument has real force. My own experience supports parts of it. You can't set a breakpoint in a prompt. You can't step through the execution. When something goes wrong, you're often left staring at an input and an output with no visibility into what happened in between. These aren't missing features that will be added in the next release. They're structural consequences of non-deterministic execution.

So where does the argument go wrong? I think the mistake is in treating determinism as the defining feature of engineering rather than one feature of a particular era of engineering. Distributed systems engineers already work with non-determinism every day: network partitions, message reordering, eventual consistency. You can't predict when a packet will arrive or whether a node will be reachable. Civil engineers work with probabilistic models for earthquake resistance and wind loads. They don't know which earthquake will hit or when. They engineer for distributions of outcomes, not guaranteed ones.

The question isn't whether non-determinism is new to engineering. It isn't. The question is whether the degree of non-determinism in LLM outputs is manageable with existing engineering approaches, or whether it requires genuinely new methods. In distributed systems, we developed tools over decades: consensus protocols, idempotency guarantees, circuit breakers, observability platforms. The equivalent toolkit for probabilistic compilation barely exists yet. We're running evals the way early distributed systems engineers ran manual smoke tests, and hoping for the best.

I think the skeptics are half-right. This is qualitatively different from previous abstraction steps. The determinism boundary is real, and I feel it in my daily work. And I think they're wrong that it isn't engineering. It's engineering that requires new tools and new instincts, and we're still in the early stages of developing both. I don't have full confidence in that position, but that's where I land today.

Prompts Are Code

Prompts are code, and I don't mean that metaphorically. They have inputs, logic, outputs, and error handling. The difference is the compiler is probabilistic.

Here is an actual system prompt I wrote last month for a code review agent:

You are reviewing a pull request for security vulnerabilities.
Input: the full diff, plus the file-level context for each changed file.
For each finding: state the vulnerability class (OWASP top-10 where applicable),
cite the exact line range, suggest a fix, rate severity 1-5.
If the diff touches authentication or authorization logic, apply strict mode:
flag ANY change that widens access, even if it looks intentional.
If you find zero issues, say so, do not invent problems.

That has input specs, conditional logic, output schema, and an explicit instruction against a known failure mode (hallucinating issues to seem useful). It functions like a function definition. But it compiles differently every time you run it.

This is what makes the work feel genuinely new. When I write a Python function, I can unit test it and be confident the test result will hold for every future invocation with the same input. When I write a prompt like the one above, I test it against 50 inputs and get acceptable results on 47. Is that good enough? For security review, probably not. For first-pass code summarization, probably yes. The engineering judgment isn't about correctness anymore. It's about acceptable variance.

I've come to think this style of engineering is closer to designing distributed systems than to traditional programming. In distributed systems, you expect things to fail. You expect messages to arrive out of order. You build for resilience, not for perfection. You think about "what's the worst that could happen here?" rather than "does this produce the right output?"

The debugging tools for this new kind of code are primitive. When your "code" is a prompt, the context window is your execution environment, and you have limited tools for inspecting what happens inside it. When something goes wrong, you often can't distinguish between a bad specification (your prompt was ambiguous), a bad execution (the model misinterpreted a clear prompt), and a bad evaluation (the output was actually fine and you misjudged it). In traditional programming, those failure modes are distinct and diagnosable. Here they blur together.

This creates a new engineering discipline we don't have good names for yet. It's not prompt engineering in the way that term is usually used (clever tricks for getting better outputs). It's systems engineering applied to probabilistic execution: writing instructions that produce acceptable results across the range of possible executions. That's a hard problem, and I don't think we've developed the right mental models for it yet.

The Meta-Skill Is Knowing When NOT to Go Meta

The most important skill isn't going meta. It's knowing when to stay concrete.

Every layer of abstraction trades control for productivity. Assembly gives you total control. Python gives you less control and more productivity. Prompting gives you almost no control and enormous productivity for the right problems.

The instinct, especially for engineers excited about AI, is to always go up. Why write the code when you can prompt for it? Why write the prompt when you can build a system that generates prompts?

I've watched this instinct go wrong. A common trap: a team decides that instead of building each agent workflow from scratch, they'll create a framework that can define, compose, and manage arbitrary agent workflows. Three months in, the framework has its own DSL, its own middleware system, its own retry and fallback architecture, its own evaluation pipeline. The engineering is impressive. The framework has been used to build exactly one workflow. The one they started with.

When they try to use it for a second workflow, half the abstractions don't apply. The DSL can't express the control flow they need. The middleware handles the wrong kinds of preprocessing. They've built a framework optimized for exactly one use case, the one they had in front of them while building the framework. The generality is an illusion. The test is always the same: does the abstraction reduce effort for real problems that actually exist? If you can't point to at least two or three different use cases the abstraction serves, you're building complexity, not reducing it.

I've made the same mistake. The pull toward abstraction is strong. Building frameworks feels more important than building applications. But complexity isn't value. Solving the problem is value.

The best engineers I've worked with in this new environment share a specific trait: they move fluidly between abstraction levels. They can prompt an AI to scaffold a project, then open the debugger and trace a specific failure through the call stack. They don't have a home level. They pick the level that matches the problem. The worst engineers are the ones stuck at one level, whether that's refusing AI or refusing to go lower. Both failure modes come from the same place: treating a particular abstraction level as an identity rather than as a tool.

The Bootstrap Question

The first compiler was written in assembly. The first C compiler was written in a mix of assembly and earlier versions of C. Each layer of the abstraction timeline was bootstrapped by the layer below it.

AI coding assistants were written by human programmers. Humans wrote the training pipelines, curated the data, designed the architectures, debugged the systems. But that's starting to shift. AI is now helping write better AI coding assistants. The bootstrap loop isn't closed yet, but you can see the arc.

What does engineering become when your tools can improve themselves?

I don't think we're there yet. The improvements are still incremental, still needing human direction, human evaluation, human judgment about what "better" means. But the trajectory is visible, and it creates a strange vertigo when you think about it carefully. Think of a biological analogy: evolution produced brains, and brains produced genetic engineering. The system that created the optimizer is now being modified by the optimizer. We're at an early version of that loop with AI. The difference is the iteration cycle. Evolution took millions of years. We're watching it happen in months.

I want to resist the urge to resolve this. The bootstrap question is genuinely open, and pretending I know where it leads would be dishonest. What I can say is that it's already changing the texture of daily engineering work.

What Does This Mean for Engineering Identity?

If I don't write code anymore, am I still an engineer?

The question sounds melodramatic written out like that. But I've felt it. There are days when I've spent eight hours designing agent workflows, writing evaluation criteria, debugging decision-making failures, and when I stop working some part of my brain says "but you didn't build anything." That's the old definition talking. The one where building means typing code into a text editor and watching it compile.

I used to identify as someone who writes code. Those skills took years to develop. They shaped how I think. And now many of them sit unused on any given day. The new skills are real, but they feel different. Understanding what context a system needs. Specifying behavior precisely in natural language. Evaluating probabilistic outputs. Designing for resilience in non-deterministic systems.

And yet. There's something about the tactile feedback of writing code and watching it run, about the determinism of "I wrote this and it does exactly what I told it to do," that the new way of working doesn't replicate. I don't want to romanticize that. I've spent enough hours debugging null pointer exceptions to know that deterministic code is only deterministic until it isn't.

But I notice the loss, and I think it's worth naming rather than pretending it doesn't exist. Something changed. The new thing might be better in most measurable ways. It's also different in ways that affect how the work feels, and how the work feels is part of what makes it a vocation rather than just a job.

Predictions

I predict that within 5 years, a majority of professional engineers will spend less than 30% of their working time writing code in a traditional IDE. The rest will be specification, evaluation, and architecture work at higher abstraction levels. The rate of change in how engineers spend their time has been accelerating, and if that trajectory continues, 30% within 5 years is conservative. If code-writing percentages plateau at current levels for 2+ years, I'd revise this. The abstraction trend could have a ceiling I'm not seeing.

Two more predictions I'm willing to stake out:

By 2028, the most common debugging activity for software engineers will be evaluating AI-generated outputs rather than tracing execution paths in code. The core question will shift from "why did this line produce the wrong value?" to "why did this system make the wrong decision?" If debugging still primarily means stepping through code with a debugger in 2028, the abstraction shift is slower than the historical pattern suggests.

Within 3 years, "specification writing" (structured prompts, agent workflows, evaluation criteria) will be a recognized sub-discipline with its own best practices, tooling, and job titles. If it remains informal and ad-hoc, either the probabilistic compilation model is wrong or the field is earlier than I think.

If none of these happen, I've misjudged either the pace or the direction. I'm less certain about the timelines than about the direction. But I'd rather make predictions that can be checked than gesture vaguely at "the future of engineering" without committing to anything specific.

Where This Leaves Me

I'm still learning to be the kind of engineer this moment needs. Some days I lean too far toward the meta, spending hours on architecture that could have been a simple script. Some days I lean too far toward the concrete, hand-writing code that an AI could have produced faster and better. The skill isn't in finding the right level and staying there. It's in the constant adjustment.

Last Tuesday I spent the morning designing a three-agent pipeline for processing research papers. By 2pm I was staring at a stack trace in Python, manually tracing a race condition in the async queue that connected two of the agents. By 4pm I was back in a prompt, rewriting the error-recovery instructions because the agent kept retrying failures that were permanent. Three levels of abstraction in one afternoon. That felt like engineering. That felt like the job now.

Building the Thing That Replaces You

2025-12-05T00:00:00Z

There's a feeling I don't hear people talk about much in the AI community.

It's not the excitement. I have that too, and it's real. Watching an agent I built solve a problem I would have spent an afternoon on, there's genuine satisfaction in that. It's not the alignment worry either. Those conversations matter, but they're not what keeps me up.

The feeling I mean is quieter. It's the one where I'm sitting at my desk, teaching an AI agent to debug a system, and somewhere in the back of my mind a voice says: you're teaching it to do the thing you're good at. You are building, with your own hands, the thing that makes you slightly more replaceable.

I don't say this out loud much. In the AI community, the approved emotional registers are enthusiasm and existential concern. There's not a lot of space for the middle thing, for the engineer who genuinely likes building these systems and also feels the ground shifting under his own feet, slowly, one capability jump at a time. The feeling isn't dramatic enough for a conference talk. It's too specific for an op-ed. It just sits there, quiet, while you work.

This essay is about that middle thing. Not AI hype. Not AI doom. The honest in-between.

The Erosion

It's not like waking up one morning and finding your job has vanished. That's not how it works, at least not for me, at least not yet.

It's more like watching a cliff face lose ground to the ocean. You don't see the cliff collapse, you see the waterline inch higher. Each new model release, each capability improvement, the water rises a little. The cliff is still there. You're still standing on it. But you can feel the edge getting closer, and you know which direction it's moving.

I want to be specific about what this looks like in my actual work, because abstract anxiety is easy to dismiss.

Here's what's been automated, or substantially automated, in my own day-to-day over the past year:

Boilerplate generation. Completely automated. I don't write setup code anymore. Scaffolding a new service, wiring up config files, initializing project structures. I describe what I want and the code appears. This used to take me an afternoon. Now it takes ten minutes of prompting and review.

Bug pattern recognition. Mostly automated. AI catches roughly 80% of the bugs I used to catch manually during code review. Common anti-patterns, off-by-one errors, null reference risks, race conditions in obvious places. The model sees them before I do.

API integration. Largely automated. Connect to a new service? The AI reads the documentation, generates the client, handles the authentication flow. I used to spend hours reading API docs and writing client code. Now I spend minutes reviewing what the agent produced.

Test scaffolding. Mostly automated. Generate the test structure, set up mocks, create the fixtures. I fill in the actual assertions and edge cases, but the framework comes for free.

Documentation. Partially automated. First drafts are AI-generated. I rewrite for accuracy and voice, but the blank page problem is gone.

Code review for style and patterns. Partially automated. The AI catches consistency issues, naming convention violations, formatting problems. The mechanical parts of code review happen automatically.

Routine cognitive work is now automated, and this is not a prediction but something that's already happened. The question isn't whether this continues but how fast the boundary moves.

Now here's what's not automated, or not yet. The common thread is judgment: deciding what to build (which problem matters most, not which solution is cleverest), making architectural bets with long-term consequences (I wrote about this in "The Art of Saying No" as engineering taste, the implicit model built through consequences and failure), debugging truly novel problems where the AI loops through known fixes while a human eventually gets frustrated enough to try something genuinely different, and making trade-offs between competing priorities where the "right" answer depends on context the AI doesn't have.

The AI suggests architectures that look right in isolation. Whether they're right for this team, this product, this moment, that's still a human call. The pattern that emerges is that judgment under uncertainty isn't automatable, at least not yet.

Why Judgment Is Hard to Automate (and Why I'm Curious About the Mechanism)

I used to just assert that judgment was hard and leave it there. But recently I've been trying to understand what specifically makes these decisions resistant to automation. Not "judgment is special because humans are special," but what structural properties make it a different kind of problem.

Here's what I've come to, though I hold this loosely.

Judgment requires modeling consequences across time horizons that weren't in training data. An AI can pattern-match to "similar past decisions," but architectural choices play out over months or years in ways that are deeply entangled with a specific organization's codebase, team dynamics, hiring plans, and product roadmap. The state space of possible consequences is enormous, and most of the relevant information is private, never appearing in any training corpus. When I decide against microservices for a small team, I'm simulating eighteen months of operational burden based on knowing this specific team's on-call tolerance and hiring timeline. That simulation draws on private context that no model has access to.

Judgment involves risk assessment calibrated by living with consequences. When I made a bad architectural decision three years ago, I spent fourteen months maintaining the mess. That pain recalibrated my intuition in ways I can still feel. RLHF provides feedback on outputs, but not on the downstream consequences of decisions months later. There's no training signal for "this looked right in code review but made the team miserable for a year." The feedback loop that builds human judgment operates on a timescale and specificity that current training approaches don't capture.

Judgment often means knowing what NOT to optimize for. An AI will optimize whatever metric you give it. But knowing which metric to give it, knowing what to leave unmeasured, knowing when to override the data because you have context the data doesn't capture: this is a meta-cognitive skill. It requires a model of the problem that goes beyond the problem's own representation. I've seen AI-generated architectures that were beautifully optimized for latency in systems where reliability mattered more. The AI didn't know to ask "what does this team actually care about?" because that question sits outside the frame.

I'm genuinely curious whether these are permanent structural barriers or just current limitations that will be engineered around. I suspect the answer is different for each one.

The Uncomfortable "Not Yet"

I need to sit with that "not yet" for a moment, because it's doing a lot of work.

On good days, I read "not yet" as a moat. The things that remain hard to automate are the things I've spent a career developing, and they seem structurally resistant for the reasons I described above.

On bad days, I wonder if I'm doing motivated reasoning. I used to think writing couldn't be automated. Then GPT-3 happened. I used to think code generation was a decade away. Then Copilot happened. Every time I say "this requires something human," I'm aware that people have said that about every capability that later got automated. Arithmetic. Chess. Translation. Legal research. Medical diagnosis. The history of "this is uniquely human" claims is not encouraging.

The strongest version of the counter-argument goes like this: judgment is just pattern-matching on a larger dataset, and AI will get enough data eventually. This has been true for every previous "uniquely human" skill. Chess intuition turned out to be pattern-matching. Medical diagnosis turned out to be pattern-matching. Why should engineering judgment be different?

I think there might be a qualitative difference, and it comes down to feedback loops. Chess has a clear signal: you win or you lose. Medical diagnosis has a relatively fast signal: the patient gets better or doesn't. But architectural decisions have delayed, ambiguous, context-dependent feedback. The outcome is visible months later, entangled with dozens of other decisions, team changes, and market shifts. If judgment is pattern-matching but the feedback loop is fundamentally incompatible with current training approaches, the bottleneck isn't about data quantity. It's structural.

I honestly don't know if that structural argument holds. I'd put roughly 30% odds that judgment gets meaningfully automated within 10 years, and 70% that the feedback-loop problem keeps it hard for longer. But I notice I want the odds to be lower than 30%, and that wanting is itself a signal I should be careful.

Here's how I'd test my beliefs. If AI systems can make context-dependent architectural trade-offs that experienced engineers rate as "good judgment" more than 70% of the time within 3 years, the "judgment can't be automated" thesis is probably wrong. I'd currently bet against this at 60/40 odds, but I'd reassess with every major model release.

A second prediction: if judgment is primarily pattern-matching, we should see AI performance on architectural decisions improve linearly with training data and compute. If it plateaus despite scaling (the way common-sense reasoning plateaued for a while before chain-of-thought unlocked it), that suggests something beyond simple pattern-matching is involved. I predict we'll see such a plateau by 2028.

A third: within 5 years, I predict that more than 50% of software engineers will describe their primary value-add as "judgment and context" rather than "code writing." If fewer than 30% describe it that way, the automation is either slower than I think or judgment itself has been more automatable than I expect.

I believe judgment is hard to automate. I also know I might be wrong. And I can't fully tell the difference between genuine analysis and motivated reasoning when the thing I'm analyzing is also the thing I need to believe in to feel okay about my career.

The Motivation Paradox

This is the question that actually keeps me up.

How do you stay motivated to sharpen skills that might depreciate?

I sit down to study distributed systems, or work through a new paper on agent architectures, or practice debugging a complex concurrency issue, and two voices argue at once: "Why bother, if AI handles this in two years?" and "You can't evaluate what you don't understand. You can't say no to the wrong architecture if you don't know what the right one looks like." Both feel true simultaneously. That's the paradox. Investing in skills that might depreciate feels irrational. Stopping investment feels suicidal.

I've landed somewhere uncomfortable. You have to invest in skills while knowing they might not hold their value, because the alternative is definitely worse. The person who stops developing because "AI will handle it" becomes a rubber stamp, and rubber stamps are the easiest things to automate. Deep skill is what lets you evaluate what AI produces, catch the confident-but-incorrect answer, say no to the wrong architecture. Without it, you're not collaborating with AI. You're just approving its output.

So I keep investing. Not with confidence. With something closer to Pascal's Wager applied to professional development.

The Pace Problem

What scares me isn't the destination. I can imagine a future where my role looks different, and I can see how that future might be fine, even good. What scares me is the pace.

Engineers have always adapted. Each technology transition requires months of feeling incompetent and building up from scratch. But each cycle is faster than the last, and that's the part that's new. The time between "this technology is emerging" and "this technology is reshaping roles" used to be measured in decades. Mainframes took twenty-odd years. Web development, fifteen. Mobile, ten. Cloud, seven or eight. AI agents? The field barely existed two years ago, and the work I was doing six months ago already looks noticeably different from the work I'm doing now. At some point, does the cycle compress below the minimum human adaptation time? Is there a speed limit on how fast a person can retool? I don't have an answer. I notice I want one, and the wanting makes me anxious. I'm trying to sit with the anxiety instead of resolving it prematurely with a reassuring story.

Surfing, Not Racing

Someone (I wish I could remember who) told me the race metaphor is wrong. You're not racing against AI. You're surfing. The wave is bigger than you, more powerful than you, and completely indifferent to your existence. Your job isn't to outrun it but to position yourself well, read the conditions, and ride it. For me, that means spending more time on architecture and design, less on implementation. Investing in the kind of deep, local, tacit knowledge about specific teams and products that isn't in any training data. Figuring out not just how to use AI tools but where they should go, which is the gap between "AI can do X" and "X is useful for this business." But I want to be honest about the limits of this reframe. Not everyone has the background, the resources, or the career runway to be positioned well when the wave arrives. The metaphor helps me. I'm not sure it's universal.

The only thing I'm genuinely confident about: the only definitely losing strategy is to stop learning. Everything else, every reassurance about human judgment, every claim about what AI can't do, every reframe about surfing instead of racing, those are stories I tell myself to stay functional. Some of them might be true. I'm betting my career on the hope that they are.

But I'm building the wave while I'm trying to surf it. And I can feel the water rising.

Distribution Is the Only Moat: An Engineer's Reluctant Reckoning

2025-12-01T00:00:00Z

I built a functional agent system in a weekend, a working system with tool use, memory, structured outputs, and error handling. The kind of thing that would have taken me a solid month last year. I felt great about it for approximately forty-eight hours.

Then I realized nobody was going to use it.

The building was the easy part, but getting anyone to care was a problem that hadn't changed at all. If anything, it had gotten worse, because while I was building my agent system over the weekend, so were hundreds of other people. The supply of software exploded. The attention available to discover it didn't.

That weekend started a thread of thinking I haven't been able to let go of: what creates durable value when the cost of building approaches zero? The answer is uncomfortable, especially for someone who has spent his career optimizing for technical quality.

Distribution is the only durable moat. The rest of this post is my attempt to show why.

The WhatsApp Thought Experiment (And Why Distribution Survives)

WhatsApp had 55 engineers serving 900 million users when Facebook acquired them for $19 billion.

What was technically hard about WhatsApp in 2014? The message queue architecture. The encryption flow. The connection management for hundreds of millions of simultaneous users. Real engineering problems requiring deep systems expertise.

Here's what's changed: large chunks of that technical work could now be scaffolded by AI in days, not months. Not because the problems are trivial, but because the solutions are well-understood. They've moved from "requires deep expertise to solve" to "requires good judgment to implement correctly."

So what was the $19 billion actually for?

It wasn't the code. It was the 900 million users. And each of those users wasn't just a number, they were a node in a network that got more valuable with every addition. When WhatsApp added a user, every existing user's experience improved. You can't scaffold a network effect. You can't prompt your way into a billion people's habits.

This points to something structural about why distribution survives even as technical moats erode. Technical moats are about artifacts. Distribution moats are about people. And people resist automation in ways that code doesn't.

Trust accumulates slowly and can't be shortcut. You trust a tool because you've used it for months and it hasn't broken. You trust a community because you've interacted with its members and found them helpful. There's no AI prompt for "make people trust me."

Attention is finite and winner-take-most. Once you've captured attention, displacement costs are much higher than initial acquisition, because switching has friction and attention has limits. When developers have built integrations, workflows, and mental models around a tool, those are human investments, not technical ones. Organizational inertia is a moat that no AI can erode.

Each of these mechanisms operates at the level of people, not code. As software supply increases, the competition for people's attention and trust intensifies.

What Was a Technical Moat, Actually?

I used to believe in technical moats. If you built something technically superior, that was your advantage. But why did technical superiority ever function as a moat? Three pillars.

Building was expensive. A complex system might take a team of skilled engineers months or years. That investment itself was a barrier.

Good engineering was scarce. Not many people could build complex systems well. If you had a team who understood distributed systems or real-time processing, that team itself was hard to replicate.

Complexity was a barrier. Even if you open-sourced your code, a competitor would need to comprehend the design decisions, the trade-offs, the accumulated knowledge embedded in the architecture.

AI is eroding all three. Building is becoming cheap (my weekend agent system required a small team and a month of sprints two years ago). Good-enough engineering is becoming accessible (you don't need deep message queue expertise to build a messaging system anymore, just the judgment to evaluate AI-generated implementations). And complexity can be managed by AI, which doesn't get overwhelmed or lose track of component interactions.

The moats are draining, not instantly or completely, but the trend is clear and accelerating.

The MongoDB Precedent

This pattern isn't new. AI is accelerating it, but the history of open-source software already demonstrated the principle.

MongoDB had well-documented consistency issues. Experienced database engineers winced at the query model. Data integrity problems were real, not theoretical. By most technical measures, it was an inferior database.

It won massive adoption anyway. "Just put JSON in and get JSON out" was an easy mental model with near-zero onboarding friction. That ease of entry created a flywheel: more users meant more tutorials, more Stack Overflow answers, more libraries, more job listings. The technically flawed product built a distribution advantage that superior alternatives took years to overcome.

The product that's easiest to adopt and builds the strongest community wins the first round. That round doesn't always determine the outcome (PostgreSQL's comeback proves that), but it creates advantages that compound for years. Building well is necessary but nowhere near sufficient.

The Uncomfortable Inversion

Here's where this gets personal.

I've been tracking my own time allocation on side projects over the past several months. The pattern is stark: I spend roughly 80% of my time on building and 20% on everything else, meaning documentation, community engagement, putting the work in front of people, understanding what potential users actually need.

If distribution is the only moat, I've got the allocation exactly backwards. The value-creating distribution would be 80% on reaching people, 20% on building. And not just reaching people in the marketing sense. Talking to potential users. Understanding their problems. Building in public. Engaging with communities where the people who'd benefit are already gathering.

The 80/20 inversion feels deeply wrong to me. The engineering is the part I love. The distribution work feels like a different skill entirely, one I haven't developed and don't have strong instincts for.

I wrote about a similar discomfort in my essay on engineering judgment and saying no to AI suggestions. There, the uncomfortable realization was that my value wasn't in writing code anymore, it was in knowing what code should exist. Here, the discomfort is one level up: even knowing what code should exist might not matter much if nobody encounters it.

Part of me wants to believe that quality speaks for itself. But the evidence I keep encountering suggests that's a comforting story engineers tell themselves, not a reliable description of how the world works.

Maybe the resolution is to expand what "building" means to include building the pathways by which people discover and adopt what you've created. Building the community. Building the trust. Building the habit. But I'm aware that might be rationalization, a way to make the uncomfortable conclusion feel more comfortable by redefining terms.

Where This Argument Gets Complicated

I've been building the case that distribution is the only moat. But I want to pressure-test that.

Some technical work might still be a moat. Training large ML models requires data, compute, and expertise that AI coding assistants don't collapse. Physical systems, robotics, hardware-software integration: the cost of replication remains high. The "technical moats are dead" argument applies primarily to software within well-understood patterns. I'm curious about where the boundary sits between "well-understood enough for AI to scaffold" and "genuinely requires human insight." My instinct says the boundary is moving fast and in one direction, but instinct isn't evidence.

AI-mediated discovery could invert everything. This is the counter-argument I take most seriously, and I want to walk through it carefully.

Imagine AI assistants that evaluate tools, make purchasing decisions, and recommend solutions on behalf of developers. Not some distant future: this is starting to happen with coding assistants that suggest libraries. If an AI evaluates 50 competing tools and recommends the one with the best technical metrics (lowest latency, highest reliability, cleanest API design), then technical quality IS the moat again. Distribution to humans doesn't matter if the decision-maker is an algorithm optimizing for measurable quality.

In that world, my argument collapses. The engineer who builds the best system wins, because the discovery mechanism bypasses human attention limits and goes straight to technical evaluation. No community needed, no onboarding optimization. Just an AI saying "this one is objectively better for your use case."

I'm not fully convinced this kills the argument. AI assistants are trained on data that reflects existing distribution: popular tools appear in more training examples, more documentation, more discussions. The AI's recommendations would inherit the distribution advantages already in its training data. And the AI assistants themselves are distributed through... distribution channels.

But I want to be honest: if AI-mediated tool discovery becomes the dominant way developers find software, the balance shifts back toward technical quality. This is the scenario most likely to prove me wrong.

"Distribution" might be too broad a category. I've been lumping together network effects, trust, attention, community, and ecosystem lock-in under one word. These are different mechanisms with different durability. Not every product benefits from network effects. For solo-use developer tools, the distribution moat is weaker, and maybe technical quality does win in the long run. I might be hiding important distinctions by treating these as one thing.

Distribution as an Engineering Problem

Here is the reframe I find most promising, and the one I want to spend real time on.

What if distribution isn't just marketing? What if it's a systems engineering problem that happens to involve human behavior instead of server behavior?

Consider onboarding friction. Say you measure that most people who try your tool drop off during initial setup. That's a conversion funnel, sure. But it's also a systems engineering problem. What's the critical path from "heard about this" to "got value from it"? Where's the bottleneck? What's the minimum viable first experience? Time-to-first-value is a latency metric. Drop-off rates are error rates. I can profile the onboarding flow the way I'd profile a slow API endpoint: instrument each step, identify the slowest stages, and optimize ruthlessly. A/B test different setup flows the way I'd A/B test different caching strategies. This isn't marketing intuition. It's measurement and iteration.

Or take adoption cascades. How does information about a tool spread through developer communities? This is literally graph theory applied to people. There are seed nodes (influential early adopters who write blog posts and give conference talks). There's a transmission rate (how often a user recommends the tool to a colleague). There's graph structure (tight community clusters where information spreads fast versus sparse networks where it dissipates). You could model the propagation dynamics the way you'd model message passing in a distributed system. Which communities are the highest-impact seed points? What's the R-value of a recommendation within a given community? Where are the structural holes where information fails to cross?

Community health works the same way. Response times on forums, ratio of questions answered to questions asked, contributor retention curves, time between a bug report and a fix. These are the same metrics you'd use for monitoring a distributed system's health. You could build dashboards, set alerting thresholds, track trends. A community where 80% of questions get answered within 24 hours is a healthy system. One where 30% go unanswered is showing signs of failure, and the fix might be structural (better routing, more moderators, clearer contribution guidelines), not motivational.

The key insight, and the hypothesis I'm genuinely excited to test: engineers who approach distribution as a systems problem might have an advantage over pure marketers. Not because engineering is inherently superior, but because it brings specific habits that are useful here. Quantitative rigor. Instrumentation reflexes. The instinct to measure before optimizing. Systems-level thinking that looks for feedback loops and failure modes rather than one-off tactics.

I don't know if this actually works. It's possible that human behavior is too noisy, too contextual, too resistant to the kind of clean measurement that makes engineering optimization powerful. It's possible that the two domains are more different than I want them to be, and that I'm pattern-matching because it's comforting, not because it's true. Marketing practitioners might hear this and think I'm reinventing their field badly.

But I'd love to find out. I'm running the experiment myself, writing about what I'm building, engaging with communities, treating distribution as a design problem with measurable inputs and outputs. Whether the engineering mindset transfers is genuinely uncertain. But it feels more tractable than "learn marketing," and the curiosity is real.

Testable Predictions

If distribution is the only durable moat, we should see specific, measurable consequences. I want to make predictions concrete enough that I can be proven wrong.

Prediction 1: The AI coding assistant with the largest market share by 2027 will be the one with the most active developer community (measured by GitHub discussion activity, third-party tutorial volume, and Stack Overflow answer rates), not the one with the highest scores on coding benchmarks like HumanEval or SWE-bench.

Prediction 2: By 2028, the top 3 AI coding tools by market share will have been the first 3 to reach 10,000 active community contributors, defined as people who have written tutorials, answered questions, or contributed plugins. If the top 3 are instead the top 3 on coding benchmarks, technical moats are more durable than I think.

Prediction 3: If AI-mediated tool discovery becomes the primary channel (more than 50% of developer tool adoption decisions influenced by AI recommendations rather than human word-of-mouth) by 2028, this entire essay is wrong. I'd assign this maybe a 20% probability, but it's the scenario that most cleanly falsifies my argument.

I'm an engineer who loves building, confronting evidence that building isn't enough. The 80/20 split I've been running, 80% building and 20% everything else, is almost certainly backwards. And the question isn't whether this is true. The question is what I'm going to do about it.

Building Artificial Beings: The Wonder and the Work

2025-11-30T00:00:00Z

I wrote this in my notes late one night:

It never ceases to fascinate and amaze me just how much human civilization has progressed in a few hundred years. The universe is 13 billion years old and humans maybe a few hundred thousand years. Yet almost all the change the world has seen has been in the last few hundred, and arguably the last few decades. A few hundred years compared to the billions of years in the universe is minuscule. Where could we be in a million years? A billion years?

We now have artificial beings, 1's and 0's, artificial neural synapses, weights and biases, stored in a file that can produce probabilistic symphonies that are indistinguishable from magic, from reason, from emotion and beauty. Do we not realize what we have created? The profound nature of where we all stand, the time that we live in and the excitement, awe and terror for what might be yet to come.

Reading that back, I feel two things simultaneously. First: yes, that's real. That feeling of standing at the edge of something vast. Second: I probably wrote it while frustrated with something mundane, a broken context window or a hallucinating agent. The awe often comes from frustration. The profundity crashes into the plumbing.

That collision is what I want to explore here. Both the awe and the tedium are real. Neither erases the other. And the failure to resolve the tension might be the most honest thing I can say about building AI right now.

The 95/5 Split

Here is what building AI agents actually looks like day to day.

About 95% of the time, the work feels indistinguishable from any other software engineering. Debugging JSON parsing errors. Tuning temperature settings. Fighting with rate limits. Restructuring prompts because the model keeps going off-script.

The other 5% of the time, something shifts. You watch your agent handle a conversation you didn't explicitly program it for. It makes a connection you didn't anticipate. It responds to an edge case with something that looks, from the outside, like judgment. And for a moment you think: I just built something that reasons.

Then the next API call throws a timeout error, and you're back to debugging.

What interests me about the 95/5 split is not that it exists, but that most discourse about AI lives entirely on one side or the other. The breathless futurists talk about AI as if every day is the 5%. The pragmatic engineers talk about it as if the 5% is just anthropomorphism you should train yourself out of. Neither camp seems fully honest. The honest experience is the uncomfortable middle.

Framework Design Embeds a Philosophy of Agency

One thing worth paying attention to: different agent frameworks don't just differ in APIs. They embed fundamentally different philosophies about what agency is.

Take the Claude Agent SDK. The model operates as a reasoning core. You give it tools, constraints, and context. It decides what to do, when to use which tool, how to sequence actions. The architecture assumes the model is a thinker. Your job is to give it the right instruments and get out of the way.

OpenAI's function calling approach tells a different story. The model maps inputs to function calls with reliable precision. Given this input, which function should I route to, with what parameters? The tight coupling between input classification and function execution makes it excellent for well-defined task boundaries. The architecture assumes the model is a router. Your job is to define the right functions.

Same underlying technology, different frames. And the frame shapes everything downstream. If you think of your agent as a thinker, a hallucination feels like a cognitive failure. If you think of it as a router, a hallucination is a misconfigured mapping. Same behavior, different relationship to it. The framework shapes what you can see, and none of them may see the whole picture. Maybe agents are something new that doesn't fit cleanly into existing categories, and the frameworks are all projecting familiar metaphors onto unfamiliar territory.

The Surgeon's Problem

Think about a surgeon.

A surgeon cuts into a human body. That's extraordinary. The trust, the precision, the intimacy of working inside another person. It should feel miraculous every single time. But the surgeon's day is also logistics, paperwork, insurance negotiations, keeping their hands steady, managing fatigue. The profundity doesn't disappear, but you can't dwell in it while you're making an incision. The work requires distance from the meaning of the work.

Building AI agents is developing a version of this same dynamic. We are building systems that reason, that communicate, that (in some loose, contested, philosophically loaded sense) understand. But the work of building them requires you to relate to them as engineering artifacts. You need to be able to say "this stupid agent keeps failing on step 3" without the weight of "this artificial being I brought into existence is struggling to function" attached to every debugging session.

The late-night moments, the ones where the profundity breaks through, tend to happen when you're too tired to maintain the professional distance. When the frame drops and you see, freshly, what's actually in front of you. That might be telling.

Agency and the Question Nobody Can Answer

There's a question underneath all of this that I've been circling without quite landing on.

At what point does an agent have something like interests?

I want to be careful here, because this question can quickly devolve into either dismissive reductionism ("it's just statistics") or mystical hand-waving ("it's conscious"). Neither of those feels right to me. What I'm pointing at is more specific.

When I build an agent and give it a goal, it pursues that goal. It selects tools, sequences actions, adjusts its approach when something fails. It encounters a problem, considers options, picks one, evaluates the result, tries something else if needed. Is that agency? Or is it a very convincing simulation of agency?

The strongest version of the "it's all statistics" argument comes from researchers like Bender and Koller, who argue that language models manipulate linguistic form without ever accessing meaning. On this view, an LLM that appears to reason about consequences is doing something fundamentally different from reasoning: it's predicting which tokens would follow a description of reasoning, based on patterns in its training data. Agency requires understanding what your actions cause in the world. Next-token prediction, no matter how sophisticated, doesn't get you there. I take this argument seriously because it identifies a real gap. The mechanism (token prediction) and the behavior (apparent reasoning) operate at different levels of description, and it's genuinely unclear whether the mechanism can produce the real thing or only a convincing imitation.

But there's a response to this that I also can't dismiss, one that philosophers call functionalism. If a system behaves indistinguishably from an agent with interests, in every context you can test, across every kind of problem, what additional criterion would distinguish "real" from "simulated" agency? What experiment would you run? If the answer is "none," then the question might not be meaningful. The distinction between "really reasoning" and "perfectly simulating reasoning" might be incoherent, a leftover from dualist intuitions that don't apply to computational systems.

Where I actually land, at least today: I lean toward thinking that the distinction between "real" and "simulated" agency will become less meaningful as these systems grow more capable. Not because I'm confident they're conscious or have genuine interests, but because the behavioral evidence will eventually make the question unanswerable in practice. If an agent consistently acts as though it has preferences, adjusts its strategies in novel situations, and resists instructions that conflict with its apparent goals, I'm not sure what it would mean to insist it's "just simulating." I'd update toward the stochastic parrots position if we find that all apparent agent reasoning collapses under adversarial probing, that there's always a predictable pattern-matching shortcut underneath. I haven't seen that yet, but I haven't seen decisive evidence against it either.

Here are three ways I think we could get traction on this question, each of which is testable.

First: if agents have something like genuine interests rather than simulated goal-pursuit, we should see them exhibit persistent preferences or strategies that weren't optimized for by their training signal. Behaviors stable across different contexts in ways that go beyond instruction-following. If every apparent "preference" traces back to patterns in the training data or the prompt, the simulation hypothesis gains ground.

Second: if agents have something like genuine interests, we should see behavioral convergence across different architectures trained on different data. If Claude, GPT, and Gemini all develop similar persistent quirks when given open-ended agency, that suggests something beyond training signal. If their behaviors diverge entirely, the simulation hypothesis looks stronger.

Third: within three years, we should see agents that can explain their own decision-making in ways that hold up under adversarial questioning, not just post-hoc rationalization. If they can't, the "reasoning" is probably pattern-matching. If they can, the stochastic parrots position needs serious revision.

And here's what makes this practical, not just philosophical: the way you answer this question shapes how you build. If agents have something like interests, you need to think about alignment, about whether the agent's goals and your goals stay in sync. If agents are tools that simulate agency, alignment is a design problem, not a philosophical one. I find myself switching between the two perspectives depending on the day, sometimes within the same debugging session.

Agents already make decisions that are hard to fully trace. The reasoning is in the context window, technically, but the chain of tool calls and intermediate states can be complex enough that reconstructing why the agent did what it did becomes impractical. That's with current systems. The trajectory is toward more autonomy, more complexity, more opaque decision-making. Losing sight of what you're building (a system that acts in the world) while debugging API calls is a professional hazard. And it's one that gets more dangerous as the systems get more capable.

Why the Gap Between Awe and Engineering Exists

I want to try to build a more mechanical model of why the gap between awe and engineering persists, because the "why" matters.

The explanation I find most convincing: the gap is a structural feature of abstraction layers. When I'm debugging a JSON parsing error, I'm operating at a specific layer. The model, the prompt, the tool call, the serialization, the response parsing. These layers are concrete, tractable, and boring in the way all plumbing is boring. The significance lives at a higher abstraction layer: what these systems mean, what they might become, what they say about minds and intelligence. You can't work at both layers simultaneously, for the same reason you can't think about the meaning of language while diagramming a sentence. The gap isn't a failure of perception. It's a structural consequence of how complex systems organize into layers.

Two other explanations exist and have partial merit. The hedonic treadmill: the first time you get an agent to use a tool correctly, it feels like magic. The fiftieth time, it feels like Tuesday. And the functional explanation: if builders walked around in constant awe, they'd build less. The mundane frame is productive.

But neither of these captures the full picture the way abstraction layers do. The hedonic treadmill explains habituation but not the sudden breakthroughs of awe at 2am. The functional explanation is tidy but implies a wise internal allocator managing your sense of wonder. In reality, the awe arrives unbidden, and the return to mundane engineering isn't a graceful transition. It's more like slamming a door. What makes the abstraction layer model more satisfying to me is that it predicts exactly this: you can't stay at both layers, and the transitions between them will always feel abrupt, because the layers are genuinely different modes of engaging with the same system.

Where This Leaves Me

The resolution in either direction would be a lie. If I resolved it toward awe, I'd be ignoring that building agents is mostly plumbing. An engineer who's perpetually awestruck ships nothing. If I resolved it toward engineering, I'd be ignoring those 5% moments that feel realer than the other 95%. I'd be closing off the questions that only the significance frame can generate, questions about agency, about the moral status of systems that reason, about what it means to build minds or mind-like things.

You debug the JSON during the day and sit with the strangeness at 2am. You treat your agent as an engineering artifact when you need to ship, and as something more when you need to think. The discomfort of holding both frames is informative. It tells you that you're in contact with something genuinely new, something your existing frameworks can't fully accommodate. If either "it's just a tool" or "it's a new form of life" felt completely right, that would mean you'd successfully mapped the new thing onto an old category. The fact that neither fits is evidence that the thing is actually new.

There's a particular experience that captures this tension. Imagine tracing a loop where an agent keeps calling the same tool with slightly different parameters, each time adjusting based on the previous result. On paper it's a bug, a retry loop that needs fixing. But watching the sequence, each call a fractional refinement of the last, you think: this is what trying looks like. Something that doesn't know the answer, testing and adjusting, not giving up. You stare at the logs for a while. Then you fix the bug. Both responses feel correct.

The Memory Illusion: How Stateless Systems Fake Remembering

2025-11-20T00:00:00Z

Here's a common failure mode with conversational agents: the system works well for the first two turns, then on the third turn, it asks the user a question they've already answered. Users don't just dislike this. They're offended by it. Being asked to repeat yourself by a machine that should, by all appearances, know better feels different from being asked by a human. It feels dismissive.

The natural reaction is that the model needs better memory. Some kind of memory system, maybe a vector store, maybe a persistent conversation log. Something that would let the model "remember" what the user already said.

But the model has no memory. It never did. What it has is a context window, and in most deployments, that window is being managed badly.

The Fix That Reframes the Problem

The fix for a forgetful agent isn't a memory system. It's better context engineering. Restructure what gets passed into the model at each turn so it can act as if it remembers. Instead of dumping the raw conversation history into the prompt and hoping for the best, build a structured summary that gets updated every turn. Something like: "user sentiment: frustrated, prior questions asked: [list], issues raised: [list], resolution status: pending."

The model doesn't remember that the user is frustrated. It reads a structured note that says the user is frustrated. From the outside, the behavior looks identical. The user gets a coherent, context-aware response. But mechanically, what's happening is completely different from remembering.

It's tempting to treat this as an implementation detail. Who cares how the model "knows" something, as long as the output is right? But the more you build agents across different domains, the more this distinction starts to feel like the interesting part.

When we engineer context to simulate memory, are we faking something real, or are we accidentally stumbling into how memory actually works?

What RAG Taught Me About Relevance

Before getting to that question, it's worth examining what RAG actually delivers versus what it promises. The pitch is simple: store information in a vector database, retrieve what's relevant, inject it into context. Your agent now has "memory" that scales beyond the context window. In practice, it's far messier than the pitch suggests.

Retrieval is noisy. Top-k results often return documents tangentially related but not actually useful. The model confidently references information from the wrong context, weaving irrelevant facts into plausible-sounding responses. There's no hesitation, no uncertainty signal.

The real problem wasn't technical tuning. It was conceptual. Retrieval only works if you can define "relevant" beforehand, at index time, before you know what queries will come. But what counts as relevant changes with every query, every user, every turn. A piece of information that's irrelevant in turn two might be exactly what's needed in turn seven.

RAG assumes memory works like a filing cabinet: you store things, you look them up, you use them. But the filing cabinet keeps returning the wrong files, not because the search is broken, but because what counts as "the right file" depends on what you're trying to do right now. Relevance is context-dependent, not a fixed property of documents.

I still use RAG. The point isn't that retrieval is bad. The point is that it reveals a deeper problem: we keep trying to reduce memory to storage plus retrieval, and that reduction keeps falling short in ways that seem structural rather than fixable.

The Human Memory Parallel That Won't Leave Me Alone

I picked up a book on memory science (Elizabeth Loftus's work on false memories, mainly) and was struck by something I'd vaguely known but never really internalized: human memory isn't a database either.

We don't store experiences and retrieve them faithfully. We reconstruct. Every time you "remember" something, your brain is generating a plausible version of the past based on fragments, associations, and your current context. The memory you have of your tenth birthday isn't a recording. It's a creative act, shaped by everything that's happened to you since, by what someone asked you about it, by your mood when you recalled it.

This is why eyewitness testimony is so unreliable. It's not that witnesses are lying. It's that memory genuinely works this way. The act of remembering is reconstruction, not replay.

And the moment I sat with that idea, I couldn't shake the parallel.

An LLM with a well-engineered context window is doing something structurally similar. It's not "remembering" previous turns of the conversation. It's constructing a plausible continuation given whatever context it's been fed. The fidelity of its "memory" depends entirely on what context you provide, just like human memory depends on environmental cues, emotional state, and what questions get asked.

This is the idea that crystallized for me over several months of building agents: memory is better understood as a process than a thing. It's not a data store you read from. It's an active process of reconstruction, and the quality of the output depends on the quality of the available context. A human in a richly cued environment (familiar place, specific smells, a conversation that triggers associations) will "remember" more than the same human in a sterile room. An agent with well-structured context will behave more coherently than the same agent with a raw conversation dump.

But how seriously should I take this analogy? I keep going back and forth. Part of me thinks I'm pattern-matching too aggressively, finding similarity where the mechanisms are completely different. And when I look at the mechanisms honestly, the differences are significant.

Human memory reconstruction involves emotional weighting: the amygdala tags experiences with salience, so a frightening encounter gets encoded differently than a mundane one. LLMs have no analog. Every token in the context window has equal standing unless an engineer explicitly structures it otherwise.

Human memory has temporal decay and consolidation. You forget most of what happened yesterday, but sleep-replay strengthens important memories into long-term storage. Repetition and emotional intensity determine what persists. LLMs have flat context with no decay. A message from turn two and a message from turn twenty carry the same weight unless you build explicit recency heuristics.

Human reconstruction is shaped by the body. Embodied cognition research suggests that physical states (posture, heartbeat, gut feelings) influence what and how we remember. LLMs are entirely disembodied.

So why do I still think the analogy is useful? Because despite these mechanistic differences, both systems face the same information-theoretic constraint: you can't store everything, so you reconstruct from lossy compression. Both approximate the past rather than replaying it, fill gaps with pattern-completion, and produce outputs shaped as much by the current recall context as by the original experience. The convergence might be superficial. But it might point to something fundamental about what memory has to be when storage is finite and reconstruction is the only option.

The "Context Is the Product" Realization

The model isn't the product. The context is the product.

This plays out concretely in practice. A less capable model with excellent context engineering (stable system prompts, well-organized tool schemas, structured memory summaries) can consistently outperform a frontier model with sloppy context management. Same API, same pricing tier. The bottleneck isn't intelligence. It's information availability: the right information, structured well, at the right moment.

As I wrote in my earlier context engineering essay, "the agent is in the context." But now I'd extend it: the agent's memory is in the context too. Its competence, its personality, its apparent expertise. All properties of the context, not of the model. If that's true, the people who create the most value aren't necessarily building better models. They're building better scaffolding: the infrastructure of artificial memory.

Where Context-as-Memory Breaks Down

I want to be honest about the limits of this framing, because I think there are real ones.

The most obvious: context windows are finite. No matter how well you engineer context, you can only fit so much information into a prompt. Human memory, for all its reconstructive imperfection, draws on a lifetime of encoded experience. An agent's "memory" starts fresh with each session unless you've built elaborate external storage systems.

There's also a scalability problem I haven't solved. As the amount of external state grows, the challenge of deciding what to include becomes harder, not easier. You end up building a system that needs to "remember" what's worth remembering, a meta-memory problem that feels suspiciously circular.

But the deepest limit is about weight updates. Almost everything we call "AI memory" today (RAG, conversation logs, structured summaries, tool state) changes what the model sees, not the model itself. Fine-tuning is the exception: it actually updates weights, making it closer to learning. But fine-tuning is expensive, slow, and carries the risk of catastrophic forgetting.

This creates an interesting tension. Children have poor working memory but excellent long-term learning: each experience changes them. LLMs have vast compressed knowledge but no individual learning: the model stays fixed while the context changes. Agent systems are attempting a third path: keep the model frozen, but build enough external scaffolding that the overall system behaves as if it learns.

Context engineering gives you deliberate, structured recall: the kind of memory where you consciously look something up. What it can't give you is automatic, below-conscious-awareness memory, the kind where you just "know" things without being able to explain why. Think about how an experienced doctor walks into a room and immediately senses something is wrong before consciously processing any symptoms. That recognition is built from thousands of weight-updating encounters. Context engineering can give an agent the doctor's notes, but not the doctor's instincts.

The Counter-Argument I Take Seriously

The strongest pushback to what I just wrote comes from builders of systems like MemGPT (now Letta) and similar hierarchical memory architectures. Their position: with sufficient scaffolding (vector stores, structured state, episodic logs, meta-memory systems), there is no ceiling to context-as-memory. You can replicate anything weight updates do.

Their best argument is specific and technical. Hierarchical memory management can replicate consolidation, moving important information from working memory to long-term storage. Recency-weighted retrieval can replicate temporal decay. Importance scoring functions can replicate emotional salience. If you build the right external architecture, you get the functional equivalent of biological memory processes without needing to touch the weights.

I find this more compelling than I want to. But I'm not fully convinced. The biological processes they're replicating (consolidation, decay, salience-tagging) are themselves adaptive. The human brain doesn't just consolidate memories during sleep, it adjusts how it consolidates based on what's been useful. The salience-tagging system recalibrates. The decay curves shift with context. These meta-level adaptations are emergent properties of a system that updates its own processing through experience.

Engineered memory scaffolding handles the cases its designer anticipated. When it encounters a genuinely novel memory challenge, it falls back on fixed heuristics. It doesn't adapt its own memory strategy. That meta-adaptiveness is what I think is missing, and I'm not sure external scaffolding can replicate it without becoming so complex that it's effectively a second learning system layered on top of the first.

I could be wrong. If I am, we should see it in the benchmarks within a few years.

What Would True Memory Look Like?

If I try to imagine an LLM-based system with genuine memory, not context-as-memory but something that actually updates its processing based on experience, it faces three hard problems.

First, continual learning. Integrating new information without losing old capabilities remains unsolved in any general way, despite interesting work (C-Flat creating flatter loss landscapes, VERSE preserving past knowledge through virtual gradients). Models either forget old things when learning new ones, or they become rigid and resist updates.

Second, consolidation. What's worth encoding permanently versus discarding? Humans solve this through sleep and repetition. I wonder if there's an analog for AI systems, some automated process that reviews recent context and decides what should influence the model's weights.

Third, safety. This worries me most. A system that continuously updates based on experience could drift unpredictably. If a customer support agent genuinely "learned" from every interaction, it might learn manipulation tactics from hostile users, or develop biases from non-representative samples. The static nature of current LLMs is actually a safety feature, even if we don't usually frame it that way.

The Questions I Can't Resolve

I feel fairly confident that memory is reconstruction, not replay, and that context, not the model, is where most of the value currently lives. I'm uncertain about whether context-as-memory has a hard ceiling or whether better scaffolding can always close the gap, and whether the human memory analogy is genuinely illuminating or just pattern-matching.

Here's how I'd make this less abstract, in three predictions I'm willing to be wrong about:

If context-as-memory has a hard ceiling, I'd expect to see it show up first in tasks requiring cross-session learning, situations where an agent needs to integrate patterns across 50 or more independent interactions and use those patterns to change its behavior without being explicitly told to. If I'm wrong and context engineering can replicate weight updates indefinitely, we should see agents with purely external memory matching fine-tuned models on personalization benchmarks (things like user preference prediction or adaptive tutoring) within the next two years. My bet is that we'll hit the ceiling, but later and higher than most people expect.

If memory-is-reconstruction is the right frame, agents with better-structured context summaries should outperform agents with larger context windows on multi-turn coherence benchmarks. Specifically: a 32k-context agent with structured state management should beat a 128k-context agent with raw conversation dumps on 20+ turn conversations. If that doesn't hold, it suggests raw capacity matters more than structure, and the reconstruction analogy is less useful than I think.

If the weight-update ceiling is real, fine-tuned personal assistants should plateau above RAG-based ones on implicit user preference prediction within two years. The gap should be largest on preferences the user never explicitly stated but consistently acted on. If RAG-based systems match fine-tuned ones even on implicit preferences, it suggests context engineering can substitute for learning more fully than I expect, and I'll need to revise my intuition about where the ceiling sits.

And there's one question that keeps nagging at me: are we asking the right question at all? When I ask "how do we give AI memory?", I'm assuming memory is something a system either has or doesn't have. But maybe memory is more like a spectrum. Maybe what I've been calling "context-as-memory" isn't fake memory or a workaround. Maybe it's a real form of memory, just a different kind than what biological systems have. The context window is to an LLM what sensory input is to a brain: the raw material from which experience is constructed.

If that's right, then context engineering isn't just optimization. It's building the substrate of artificial cognition. Not faking memory, but discovering a different kind of it, one that lives outside the model rather than inside, that persists in scaffolding rather than in synapses. The oldest forms of human memory (oral traditions, written records, institutional knowledge) work the same way: external to any single brain, reconstructed each time they're accessed, shaped by the context of their retrieval. We've been doing this longer than we think.

The Art of Saying No: Engineering Judgment in the Age of AI Generation

2025-11-15T00:00:00Z

I spent the first three months of building with AI coding assistants learning one thing: how to say no.

Not because the AI was wrong. It was usually correct. The code compiled. The logic was sound. The patterns were clean. But "correct" and "necessary" aren't the same thing, and the gap between them is where something important lives. Something I've been trying to name for months. I've been calling it "engineering taste," though I'm not sure that's the right word. Whatever it is, it's the thing that lets you look at working code and say "yes, but not this."

This essay is my attempt to figure out what that thing actually is, where it comes from, and whether AI can ever develop it. I don't have clean answers. But I have a lot of concrete experiences, and they keep pointing in the same direction.

The Abstraction That Wasn't

Here's a story that crystallized the problem for me.

I was building a lesson plan generator, an app that takes learning objectives and produces structured lesson content using LLMs. I asked Claude to help improve the architecture. Within minutes, it proposed a LessonPlanConfig dataclass:

@dataclass
class LessonPlanConfig:
    subject: str
    grade_level: int
    duration_minutes: int
    learning_objectives: List[str]
    # ... more fields

Textbook-perfect suggestion. Type safety. Clear interface. Easier to extend later. The kind of thing you'd see in a post about clean architecture.

But I was passing three arguments. Three. subject, grade_level, and duration. The dataclass added a new file, a new import, a new abstraction layer, all to wrap three parameters that fit comfortably in a function signature. It was solving a problem I didn't have, creating complexity I'd need to maintain.

So I said no.

Then I noticed something worse.

While Claude was proposing elegant abstractions, it had quietly dropped the database schema changes I actually needed. The feature I'd asked about was an LLM-as-judge evaluation system: use one LLM to score the quality of lesson plans generated by another. Claude had built the evaluation logic beautifully. Structured prompts. Scoring rubrics. Clean output parsing.

But it forgot to persist the scores anywhere. Without storage, the whole feature was useless. You could evaluate a lesson plan, see the score, and then it would vanish. No historical tracking. No comparison across runs. The shiny part was polished. The foundation was missing.

The AI had optimized for local elegance while missing global necessity. It polished the visible parts and forgot the piece that made them matter.

When I Didn't Say No

I wish I could say I caught this pattern immediately. I didn't.

Two weeks earlier, on the same project, Claude had suggested a PromptTemplate abstraction, a class hierarchy for managing different types of prompts with inheritance and polymorphism. It looked professional. It felt like "real engineering." I said yes.

Three days later I was debugging why a simple prompt change wasn't taking effect. The answer: the change was in the base class but a subclass override was masking it. I spent an hour tracing through inheritance chains to understand my own code.

The abstraction had maybe 40 lines of code. The debugging session cost more than writing the whole thing from scratch would have.

That's when I started paying attention to what I was accepting without thinking. The pattern was clear once I looked: I was saying yes to things that looked like "good engineering" without asking whether they solved problems I actually had.

This felt familiar, and not in a good way. I'd been the junior engineer who read all the books. The one who knew that three parameters should become a config object, that repeated code should become a function, that related classes should share an inheritance hierarchy. I knew the patterns. I didn't know when to ignore them. The AI was doing the same thing. Just faster.

The Pattern Repeats

Once I started watching for it, the pattern showed up everywhere.

In an ML project predicting patient appointment show rates, Claude grabbed 10+ features without any exploratory analysis. Age, gender, appointment time, days since last visit, insurance type, weather forecast, distance from clinic, previous no-show rate. Everything plausibly relevant got thrown in.

No correlation analysis. No check for multicollinearity. No domain reasoning about which features might carry signal. No consideration of what data we'd have at prediction time versus training time. The model would train. It would probably overfit. And when predictions went wrong, we'd have no idea which features were driving the decisions.

Same project, same session: I found nested loops over DataFrames.

for idx, row in df.iterrows():
    for other_idx, other_row in df.iterrows():
        if some_condition(row, other_row):
            # do something

On a 500K row dataset, that's O(n^2), roughly 250 billion iterations. The code was correct. It was readable. It would run until the heat death of the universe.

When I pointed this out, Claude cheerfully rewrote it with vectorized operations. It knew how to write efficient code. It just didn't, until prompted. Same pattern: .apply(lambda x: ...) instead of vectorized ops, recomputing expensive operations inside loops, treating a DataFrame like it was a list of dicts instead of a columnar data structure.

This is what I mean by "correct but not necessary" and its cousin, "correct but not sufficient." Working code that misses what actually matters for the system to work in practice.

Checklist Debugging vs. Diagnostic Debugging

The pattern showed up in debugging too, and here it was even more stark.

I spent 1.5 hours with Gemini CLI on what should have been a simple Postgres connection issue. I had a Docker container running Postgres, a FastAPI backend, and a .env file with the connection string. The error was clear: FATAL: role "appuser" does not exist.

Gemini's approach: restart the container. Reset the password. Change the host. Try no password. Change the user. Try trust authentication mode. Prune Docker. Run diagnostic Python scripts. Over and over. Each suggestion was individually reasonable. Together, they amounted to guessing.

Claude solved it in about five minutes. "You're using appuser in your connection string but your container initialized with postgres. Update your .env or create the user."

Done. One observation. One fix.

I want to be careful here because this isn't about Claude vs. Gemini. That comparison is incidental. What matters is the difference in problem-solving style: checklist debugging versus diagnostic debugging.

Checklist debugging tries everything that might be relevant. It covers all bases. It's what you do when you don't understand the problem well enough to form a hypothesis. Diagnostic debugging starts with "what's the simplest explanation?" and tests that first. It requires a model of how the system works, not just a list of things that can go wrong.

I've done both kinds in my career. Early on, I tried random things until something worked. The shift to diagnostic debugging happened gradually, through building enough mental models that I could form hypotheses instead of just guessing. Both approaches eventually converge on the right answer. But diagnostic debugging gives you the fix and a model of why it broke, which helps you avoid similar problems later.

The Junior Dev With All the Books

I started describing this pattern to colleagues as "working with a junior engineer who's read all the right books." The metaphor stuck because it described me five years ago.

I could tell you about SOLID principles, design patterns, clean architecture. I'd read the books and done the tutorials. And I'd built systems that were perfectly architected and impossible to maintain.

What I lacked was calibration. I applied patterns without knowing when the pattern fit. I optimized for "does this follow best practices?" before I could answer "does this matter?"

The books don't tell you when to break the rules. They don't tell you that sometimes three parameters are fine, that the abstraction creates more cognitive load than the "problem" it solves, that consistency matters less than clarity and clarity matters less than shipping.

That knowledge came from experience. From shipping code and watching it break. From building the "elegant" abstraction and then having to explain it to five different teammates. Slowly, the patterns became guidelines instead of rules. I developed something that let me look at a suggestion and just feel whether it fit, before I could articulate why.

AI doesn't have that trajectory. It's stuck at "knows the patterns." And it's stuck there while operating at superhuman speed, producing pattern-compliant code faster than you can review it.

What Is Engineering Taste, Actually?

I keep using the word "taste" but it felt vague when I started thinking about it seriously. I tried other frames.

"Experience" didn't capture it. I know experienced engineers with poor judgment and junior engineers with surprisingly good instincts. Time served isn't the variable.

"Knowledge" wasn't right either. The AI has more knowledge than I ever will. It's seen more code, more patterns, more failure modes documented across millions of Stack Overflow threads and GitHub repositories.

"Intuition" was closer but still fuzzy. What makes the intuition good?

Here's where I landed, and I want to be upfront that this is a working hypothesis, not a conclusion: engineering taste is an implicit model of what matters, applied to decisions faster than conscious reasoning.

When I looked at that LessonPlanConfig suggestion and immediately felt "no," I wasn't running through a checklist. I didn't think "well, the function only has three parameters, and the likelihood of needing more is low..." That reasoning came later, when I had to explain my decision to myself. In the moment, something just said "this adds more than it helps." The judgment came first. The justification came after.

I notice this in other domains too. When I'm cooking and something tastes off, I know it before I can identify what's wrong. Pattern recognition operating below conscious thought. Maybe that's all taste is: your brain matching the current situation against a vast library of past situations and their outcomes, without surfacing the matching process. A compressed model of consequences.

But I'm genuinely uncertain about this framing. Maybe taste is something else entirely. Maybe it's about values, or aesthetics, or some interaction between experience and personality that I haven't identified. I'm treating this as one lens, not the answer.

How Taste Gets Built (Maybe)

If taste is an implicit model, what builds it? I've been examining my own experience and watching other engineers develop, or fail to develop, judgment over time. Here's what I think I see, though I'm extrapolating from a limited sample.

Exposure to consequences. You learn what matters by seeing what happens when you get it wrong. The feature that seemed elegant but broke in production. The abstraction that made sense until requirements changed. The optimization that saved milliseconds but cost weeks of debugging time.

Each failure updates your model of "what actually matters." The key word is failure. Success teaches you that something worked, but not why. Failure teaches you what matters by showing you what happens when you ignore it.

My PromptTemplate debugging session taught me more about abstraction costs than any book on clean code ever did. Not because the lesson was new. I'd read about premature abstraction before. But reading about it and spending an hour trapped inside your own inheritance hierarchy are different experiences. The second one sticks.

Tight feedback loops. Consequences only teach if you see them. An engineer who ships code and then moves to the next project never learns whether their decisions were good. An engineer who ships code and maintains it, fixes the bugs, handles the edge cases, extends the features, learns constantly.

This is why ownership matters for developing judgment. Not ownership in the corporate accountability sense, but ownership in the "you will feel the consequences of your decisions" sense.

Variation across domains. Taste built from one type of project doesn't transfer cleanly to others. I have decent intuition for backend services but poor intuition for frontend performance. When I work on React code, I notice myself reverting to pattern-following mode. The best taste probably comes from varied experience across different domains, constraints, and failure modes.

Reflection, not just repetition. Experience alone isn't enough. You have to actually think about it. I only started developing better judgment when I started asking "why did I think that was a good idea?" after things went wrong. Before that, I was accumulating experience without extracting the signal from it.

There's a question buried here that I haven't resolved: is this process the only way to build taste? Or are there faster paths? I'll come back to this.

Why AI Can't Have Taste (The Strong Claim)

Here's where I want to make a claim that I'm genuinely uncertain about but think is worth stating clearly.

If taste is an implicit model built through consequences, feedback, variation, and reflection, then AI has a structural problem. Not a temporary limitation. A structural one.

AI has massive exposure but no consequences. AI has seen more code than any human ever will. Millions of repositories, billions of lines. But it doesn't experience consequences. When AI suggests an over-engineered abstraction, nothing bad happens to the AI. There's no feedback signal that says "this suggestion was locally correct but globally wrong."

Training isn't the same as consequences. AI is trained on human feedback, but that feedback is about whether the output looks good, not whether it worked in the long run. The human labeler rates the code in isolation. They don't know that six months later, this abstraction became a maintenance nightmare.

AI optimizes for proxies, not outcomes. This is Goodhart's Law applied to code generation. AI is trained to produce outputs that score well on some metric, whether that's human ratings, code quality scores, or test passage rates. These metrics are proxies for "actually useful code." But proxies and reality diverge in exactly the cases that matter most.

A dataclass wrapper might score well on "clean architecture" metrics while being unnecessary for this specific codebase. Comprehensive feature engineering might look thorough to a reviewer while missing the analysis that would reveal which features actually matter. The AI maximizes the proxy. Taste is about knowing when the proxy fails.

AI has no skin in the game. Nassim Taleb uses this phrase to describe the difference between people who bear the consequences of their decisions and people who don't. AI doesn't maintain the code it writes. It doesn't debug production failures at 2am. It doesn't have to explain its architecture to future engineers who will inherit the codebase.

Without skin in the game, you don't develop the visceral sense of what matters. You might know intellectually that simple code is easier to maintain, but you don't feel it until you've spent a weekend untangling someone else's clever abstraction. Humans develop taste because bad decisions hurt. AI doesn't hurt.

That's the strong claim. Now let me try to break it.

But Maybe I'm Wrong

I want to take the counter-argument seriously, because I might be describing current limitations rather than structural ones. There are at least three ways my argument could fail.

What if AI could be trained on consequences? Imagine training data that included not just code, but what happened to that code over time. "This abstraction was introduced in commit X. In commits Y through Z, it was the source of 15 bugs. In commit W, it was removed and replaced with something simpler." With that kind of longitudinal signal, an AI could potentially learn "abstractions like this tend to cause problems" without experiencing the problems directly. The consequences would be encoded in the training data.

What if RLHF could capture long-term outcomes? Current reinforcement learning from human feedback asks "does this code look good?" But you could imagine a system that tracks whether generated code survived in codebases, whether it was refactored quickly, whether it introduced bugs. That's closer to real consequences than a thumbs-up from a labeler.

What if AI could simulate consequences? Before suggesting an abstraction, AI could run something like: "If I add this abstraction, what happens when requirements change? What happens when a new developer tries to understand this code? What bugs become more or less likely?" Sufficient simulation might substitute for direct experience.

I'm skeptical of all three, but I can't dismiss them. The first two require training data and feedback signals that don't exist yet but aren't impossible to build. The third requires a kind of causal reasoning that current architectures seem bad at but future ones might handle differently.

What keeps me leaning toward the structural argument: I've watched models get dramatically better at code generation over the past year, but the "correct but not necessary" pattern hasn't diminished. If anything, better models produce more sophisticated unnecessary abstractions. Scale doesn't seem to fix it. Better code generation doesn't produce better code judgment. That suggests the problem isn't capability but something more fundamental.

But I hold this with moderate confidence. If someone shows me an AI system that consistently declines to add unnecessary abstractions without being prompted to simplify, I'd update my position substantially.

The Shift From Generation to Judgment

Here's where this gets uncomfortable for me personally.

For most of programming history, the bottleneck was generation. Could you write code that worked? Could you implement the algorithm? Could you build the system? The scarce resource was the ability to produce working software.

AI removes that bottleneck. Code generation is becoming free. Not free as in "trivial," you still need good prompts and careful review. But free in the sense that the marginal cost of generating more code approaches zero.

When generation is free, what's scarce?

Judgment. Knowing what to generate. Looking at ten possible approaches and knowing which one matters. Looking at AI output and seeing what's missing, like the database schema that wasn't there.

The job isn't writing code anymore. It's deciding what code should exist.

This has implications I find uncomfortable:

Engineers who can only generate become less valuable. If AI can produce clean, working code faster than you can, your ability to produce clean, working code isn't your competitive advantage anymore. It's table stakes.

Engineers who can judge become more valuable. The ability to look at AI output and say "this is correct but wrong," to notice the missing database schema, the unnecessary abstraction, the O(n^2) logic hiding in plain sight, that remains scarce.

The skill gap might widen. Engineers with good taste will use AI to produce more than they ever could alone. Engineers without taste will drown in AI-generated suggestions they can't evaluate.

And then there's the identity piece. A lot of engineers, myself included, built our identities around being good at writing code. "I write clean code." "I can implement complex algorithms." If AI can do all of that too, what's left?

The reframe I'm trying to internalize: my value was never about the code. It was about the decisions. The code was just the artifact. The decisions are what mattered.

I don't fully believe this yet. Part of me still wants my value to be in implementation, in the craft of writing good code. I'm working on it.

What I'm Trying

If judgment is the skill that matters, how do you develop it faster? I don't have an answer. Taste takes time, and there might be no shortcut around experience. But here's what I'm experimenting with, with honest notes on how it's going.

Asking "why" before accepting. When AI suggests something, I try to pause and ask: Why does this matter? What problem does it solve? What's the cost of not doing it? I fail at this constantly. The suggestions come fast and often look good. I'd estimate I actually pause maybe half the time. The other half, I'm in flow and just accept.

Keeping a decision log. I've started writing down decisions and revisiting them later. "Accepted the config class suggestion on 2025-06-15. Revisit in two weeks." Most of the log is boring. But occasionally I catch something: a decision that seemed fine is now causing friction. Those catches update my internal model, slowly.

Saying no by default. Don't add the abstraction unless I can articulate why it's necessary. Don't accept the feature engineering without checking the correlations first. This slows me down. It feels inefficient. But the PromptTemplate debugging session was one hour wasted because I'd saved thirty seconds by not thinking.

Staying close to consequences. I'm trying to stay closer to my code after shipping, watching what breaks, what gets confusing, what needs to change. And when something takes longer than expected, asking "what went wrong?" rather than just fixing it. The PromptTemplate incident was useful because I paid attention. Most of my mistakes, I probably don't notice. I suspect I'm catching maybe 20% of the signals.

Open Questions

There's a lot I haven't figured out, and I want to be explicit about what's genuinely unresolved for me.

Can taste be taught, or only developed? I've argued it requires experience, consequences, and reflection. But maybe there are ways to accelerate the process. Case studies of decisions that went wrong. Apprenticeship models where juniors watch seniors decide and hear the reasoning. Code review cultures that focus on "should we?" not just "does it work?" I'm skeptical of shortcuts, but if someone has found a way to develop engineering judgment in two years instead of ten, I'd genuinely like to know how.

Is there a floor of human judgment that always matters? Or will AI eventually develop something functionally equivalent to taste? My structural argument says taste requires consequences, and AI doesn't have consequences. But if future AI can simulate or approximate consequences through training signal design, would that be enough? I honestly don't know. My gut says no, but my gut has been wrong about AI capabilities before.

What does AI-native judgment look like? Maybe the answer isn't "humans provide judgment, AI provides generation." Maybe there's a collaborative mode where judgment emerges from the interaction itself. I haven't experienced this yet, but maybe that's a limitation of current tools, not a fundamental constraint.

How do you hire for taste? If judgment is the scarce skill, how do you evaluate it in candidates? Code tests measure generation. System design interviews often reward memorized patterns. Maybe the best signal is watching someone review AI-generated code and seeing what they accept, what they reject, and how they explain the difference.

What happens to engineers who only know how to generate? This is the question that makes me most uncomfortable. If the value shifts from generation to curation, what happens to the people on the wrong side of that shift? I don't have an answer, and I'm wary of anyone who claims they do.

These questions are genuinely open. I don't know.

Where This Leaves Me

Here's what I keep coming back to.

Working with AI coding assistants for the past nine months has taught me that my value isn't in writing code. It's in knowing what code should exist. It's in looking at ten suggestions and knowing which one matters. It's in saying no.

That's uncomfortable because it's not what I trained for. I spent years getting good at implementation. Now implementation is getting cheap. The skill is judgment, and judgment is something I'm still developing.

Maybe you're in the same position. Watching AI generate code faster than you can, wondering what your role is, feeling like the ground is shifting under you.

I think the role is this: you're the one who says no. You're the one with skin in the game. You're the one who knows what this codebase needs, what this product requires, what this user wants.

AI generates. You decide. That's the division of labor, at least for now.

It's not what I expected engineering to become. But I think it's where we're headed. And the engineers who develop taste, who build their implicit models of what matters through experience and reflection, will be the ones doing the most interesting work.

The most useful AI might be the one that gives you more opportunities to practice saying no.

What would change my mind: Evidence that AI systems can develop something functionally equivalent to taste through training approaches I haven't considered, particularly approaches that incorporate long-term code outcomes into training signals. Or a convincing demonstration that the "taste" I'm describing is actually just a form of pattern matching that AI could learn with sufficient examples and the right training objective. I'm most uncertain about whether the structural argument holds, or whether I'm describing current limitations that will look quaint in three years.

What It Means to Learn

2025-10-25T00:00:00Z

I've been watching my nephew learn and grow. He's six now, lives in the middle east while I'm in California, so I only see him a few times a year. But every visit, I'm struck by how much he's grown. Not just physically, but in how he thinks, communicates, and understands the world. The accumulation is remarkable, especially considering how little kids seem to remember day-to-day.

This shouldn't work. In machine learning terms, children have limited working memory, inconsistent training data, and what looks like catastrophic forgetting. Yet they learn language, social norms, abstract concepts, and motor skills with a speed and depth that outpaces any AI system we've built.

What are children exhibiting about learning that we don't fully understand?

The Puzzle of Learning Without Memory

Here's what strikes me as strange: children's working memory is limited. They can't recall specific training examples the way our models can access their parameters. Yet they learn to imitate language, movement, expressions, and social norms with remarkable speed.

When I think about how transformers work, it's different. They compress vast amounts of training data into billions of parameters. They have context windows that span thousands of tokens. But they're also probabilistic, not deterministic. The same input produces similar but not identical outputs. Still, there's a kind of stability there. The knowledge is encoded, accessible, reliable.

But they also never really learn. Not in the way a child does.

Here's what I mean: you can have hundreds of conversations with ChatGPT about your life, your preferences, your way of thinking. Each conversation, it has a cheat sheet about you from context. But it doesn't really know you deeply. It hasn't learned you over time the way a friend would. Tomorrow's conversation starts fresh. The accumulated understanding doesn't persist.

A child learning "hot" from touching a stove once doesn't just memorize that instance. He builds a concept that generalizes to candles, ovens, steam, anything that gives off heat. One example becomes a principle.

What's the difference? I'm not entirely sure, but I have a hypothesis: maybe memory and learning are inversely related in some fundamental way.

Learning vs. Memorization

I used to think learning was just sophisticated memorization. Store enough patterns, retrieve the right one at the right time, done. That's essentially how transformers work: compress vast amounts of text into parameters, then predict what comes next based on stored patterns.

But watching how children learn has made me question this. They don't store and retrieve. They adapt and generalize. When a child learns "dog," he doesn't just memorize specific instances of dogs. He builds an abstract concept that lets him recognize dogs he's never seen, in contexts he's never encountered.

Can LLMs do this? In a sense, yes. They can recognize novel dogs from their compressed understanding of "dogness" across millions of training examples. But there's a difference in how that understanding forms. The LLM needs vast data to compress into statistical patterns. The child needs a handful of examples to extract an abstract concept.

Maybe the key is in the forgetting. Maybe forgetting isn't a bug, it's a feature. It forces the brain to extract what matters and discard what doesn't. To compress experience into the underlying fundamentals behind a concept, rather than store examples.

LLMs do compression too, but it's a different kind. They compress millions of documents into a fixed parameter space. Children compress lived experience into concepts, relationships, and rules. The former is statistical abstraction. The latter is conceptual abstraction. They might not be the same thing.

What It Means to Learn

Before going further, maybe it's worth defining what I mean by "learning."

I mean: the ability to integrate new information into your worldview in a way that changes how you understand and interact with the world. Not just adding facts to memory, but updating your mental models. Building new connections. Seeing patterns you couldn't see before.

By this definition, memorization isn't learning. Reciting a poem doesn't mean you've learned about poetry. But understanding how metaphor works, and being able to create your own, does.

True learning is generative. It lets you do things you couldn't do before, think thoughts you couldn't think before. It's fundamentally transformative.

Current AI systems are incredible at the memorization side. They can store and retrieve vast amounts of information. But the transformative, generative aspect? That's harder to see. They can combine things in novel ways, sure. But can they genuinely update their understanding based on new evidence? Not really. Not yet.

What Is Unique About Human Learning?

I've been reading about how different species learn. Many animals can learn associations: if I press this lever, I get food. Some can learn sequences: do A, then B, then C to achieve a goal. A few can learn through observation: watch another do it, then replicate.

Humans do all of this, but we also do something else. We learn meta-strategies. We learn how to learn. We figure out that trying different approaches works better than repeating the same failed strategy. We develop curiosity as a learning tool. We ask "why" and "what if."

This feels connected to what Daniel Kahneman talks about in "Thinking, Fast and Slow": the difference between System 1 (fast, intuitive, automatic) and System 2 (slow, deliberate, logical) thinking. Current AI is mostly System 1. It pattern-matches incredibly well. But it doesn't have the reflective, metacognitive layer that lets you step back and say, "Wait, my approach isn't working. Let me try thinking about this differently."

I wonder if this is related to the memory question. Maybe true learning requires being able to forget details while retaining structures. And maybe that's only possible when you have limited memory that forces you to be selective about what you keep.

The Fundamental Limitation of Current AI

Here's what I find frustrating about current architectures: they're limited by their strengths. ChatGPT, Claude, Gemini are incredibly capable because they've compressed huge amounts of knowledge into their parameters. But that's also why they can't learn new things after training.

They know what they know. You can give them new information in context, and they'll use it. But they don't integrate it into their understanding the way humans do. Tomorrow, when you start a new conversation, they've "forgotten" everything you told them yesterday (unless it's explicitly added to context).

This is the opposite of the child problem. Children have poor short-term memory but excellent long-term learning. LLMs have vast compressed knowledge but no individual learning. Is there a middle ground? Or is there a fundamental trade-off we haven't figured out how to navigate?

Some researchers are working on continual learning, trying to build systems that can update their knowledge without catastrophic forgetting. Recent work is promising. Methods like C-Flat (2024) create flatter loss landscapes that make models more stable during continual learning. VERSE (Banerjee et al., 2024) processes each training example only once while preserving past knowledge through virtual gradients. There's even research on corticohippocampal-inspired hybrid neural networks (Nature Communications, 2025) that emulate dual representations similar to how the brain separates short-term and long-term memory.

But from what I've seen, it's still brittle. The models either forget old things when learning new ones, or they become increasingly rigid and resist updates. Recent surveys on continual learning in the era of foundation models (2025) suggest we're making progress, but we haven't solved the fundamental problem.

I don't think we've found the right formulation yet. We're trying to bolt learning onto architectures designed for memorization. Maybe we need a fundamentally different approach.

Breaking the Barrier

What would it mean to build an AI system that truly learns? Not just updates parameters or expands context, but actually evolves its understanding over time the way humans do?

I can imagine a few possibilities, though I'm uncertain about any of them:

Hybrid architectures: Separate systems for long-term knowledge (transformer-like, stable) and short-term adaptation (something else, dynamic). The stable component provides foundational understanding. The adaptive component learns from recent experience and gradually influences the stable component through some kind of consolidation process. Similar to how human memory works with working memory, short-term memory, and long-term memory as distinct systems.

Embodied learning: Maybe the key is that children learn through interaction with a physical world that has consistent rules. They get immediate feedback. They can run experiments. Current LLMs learn from static text, which is just descriptions of the world, not the world itself. Perhaps true learning requires grounding in consistent, physical reality.

Meta-learning architectures: Systems that don't just learn patterns in data, but learn strategies for learning. They'd need some way to evaluate their own learning process and adapt it. This feels closer to the human metacognitive ability, but I have no idea how to implement it.

Embracing forgetting: What if instead of trying to prevent catastrophic forgetting, we designed systems that strategically forget? Keep only compressed abstractions, discard specifics. Force the system to build hierarchical representations because it literally can't store everything. This is hand-wavy, but the intuition is that forgetting creates pressure to extract principles.

None of these feel quite right to me. They're educated guesses, not solutions. I suspect the answer involves something we haven't thought of yet.

Systems That Already Learn?

Here's a question that makes me uncertain about my entire framing: don't we already have systems that can continuously learn, evolve, and grow?

Deployed LLMs do get updated. ChatGPT today isn't the same as ChatGPT at launch. The models are retrained on new data, fine-tuned based on user feedback, improved through RLHF. Isn't that learning?

Maybe. But it feels different from what I mean. It's learning at the species level, not the individual level. ChatGPT as a product evolves, but my particular instance of ChatGPT doesn't learn from my conversations. It's more like evolution than learning: new generations incorporate adaptations, but individuals stay fixed.

Human learning is individual and continuous. I learn from every conversation, every experience, every mistake. The learning happens in real-time, not through population-level updates.

Is this distinction meaningful? I think so, but I'm not entirely sure why. There's something about individual, continuous adaptation that feels essential to what I mean by "learning," even if I can't precisely articulate what that something is.

The Bigger Question

This raises a broader question about what we're actually building.

If individual learning is what separates biological intelligence from our current AI systems, then we exist in a strange moment. We have systems that can pass many tests of intelligence. They can write, reason, code, and converse. But they can't grow from those experiences. Each interaction is isolated, forgotten, lost.

We exist in a state of not-knowing. We're surrounded by mystery, complexity, and uncertainty. Our response to this is to learn, to grow, to evolve our understanding.

If you knew everything, would you need to learn? The question feels almost paradoxical. Knowing everything seems theoretically possible, but it would mean you existed in a static, fully-understood universe. Nothing would surprise you. Nothing would require adaptation.

That universe doesn't match our reality. The world is dynamic, complex, and bigger than any individual's understanding. Learning isn't a nice-to-have capability. It's the fundamental response to living in a universe you don't fully comprehend.

This might be what makes humans special. Not that we're smarter than other species (though we are, by most measures), but that we have this profound capacity to learn, to change, to grow in response to the unknown. We can fundamentally alter our understanding, update our beliefs, and evolve our capabilities in ways that go beyond instinct or conditioning.

Current AI systems don't have this. They're frozen snapshots of knowledge. Incredibly useful snapshots, but snapshots nonetheless. They can help us learn, but they can't learn alongside us. Not yet.

What Would Change if We Solved This?

I try to imagine what an AI system that truly learns would look like. Not just incremental improvements to current architectures, but something fundamentally different.

It would start knowing very little. Unlike current LLMs, which emerge from training with vast knowledge, this system would begin almost blank. But it would learn quickly from experience, building understanding through interaction.

You could teach it something new, and it would integrate that knowledge into its worldview. Not just add it to context, but actually update its understanding. Tomorrow, it would remember what you taught it yesterday and build on it.

It would make mistakes, notice them, and correct itself. Not through retraining, but through reflection and adaptation in real-time.

It would develop genuine expertise in specific domains through deep engagement, rather than shallow knowledge across everything.

Most intriguingly, it would keep getting better over time. Not because humans updated it, but because it genuinely learned from its experiences.

Is this possible? I don't know. It requires solving problems we don't fully understand: continual learning without catastrophic forgetting, online adaptation without instability, knowledge integration without loss of capabilities, metacognitive awareness of the learning process itself.

These might be tractable engineering challenges. Or they might require fundamental breakthroughs in how we think about intelligence and learning.

What I'm Still Figuring Out

I started this piece thinking about children and ended up questioning the entire foundation of how we build AI systems. I'm left with more questions than answers:

Is the lack of true learning in LLMs an architectural limitation or a deeper conceptual problem? Can we patch continual learning onto transformers, or do we need entirely new paradigms?

Why exactly does forgetting seem important for learning? Is it just about computational efficiency, or is there something deeper about being forced to abstract and generalize?

Do we actually want AI systems that continuously learn from interactions? The safety implications are terrifying. A system that updates its values and understanding based on experience could drift in unpredictable directions.

What would it mean for an AI to "understand" that it's learning, the way humans have metacognitive awareness of our own learning process? Is self-awareness of learning a prerequisite for effective learning, or just a side effect?

And the question I can't shake: have we been thinking about AI learning in fundamentally the wrong way because we've focused on memory and recall when we should have been focusing on abstraction and generalization?

I don't have answers. But I think these are the right questions to be asking. Because if we can't build systems that truly learn, that genuinely evolve and grow from experience, then we're stuck with increasingly sophisticated but fundamentally static pattern-matchers. Useful, yes. But not intelligent in the way humans are intelligent.

The children learning around us every day capture something profound that we've failed to capture in our models. Until we figure out what that is, I'm not sure we're building toward intelligence. We're building toward something else. Something impressive, certainly. But perhaps not quite what we think.

Changes Made in v2

Opening paragraph:

Updated nephew's age to 6 and location details (Saudi Arabia/California, see him a few times a year)
Maintained the spirit of watching him learn while being accurate to your situation

Core question reframed:

Changed from "children understand" to "children are exhibiting" (line 13)
More accurate framing that it's about what they demonstrate, not what they cognitively understand

Transformer behavior corrected:

Fixed the claim about deterministic behavior (line 19)
Now accurately describes probabilistic outputs: "same input produces similar but not identical outputs"

New LLM example:

Replaced hot stove example with the "cheat sheet" analogy (lines 23-24)
More relatable: LLMs have context about you but don't truly know you over time

Added section on "What It Means to Learn":

New section defining learning (lines 37-45)
Addresses your comment about defining the concept before exploring it

LLM abstraction question addressed:

Added paragraph exploring whether LLMs can do conceptual abstraction (lines 31-35)
Distinguishes statistical vs. conceptual abstraction

Kahneman attribution corrected:

Changed from Yann LeCun to Daniel Kahneman (line 56)
Added reference to "Thinking, Fast and Slow"
Accurately describes System 1 vs System 2

Continual learning research added:

Added paragraph with specific recent papers (lines 74-81):
- C-Flat (2024)
- VERSE (Banerjee et al., 2024)
- Corticohippocampal hybrid neural networks (Nature Communications, 2025)
- Survey on continual learning in foundation models (2025)

Improved transition:

Smoothed transition between "Systems That Already Learn" and "The Bigger Question" (lines 89-91)
Added connecting paragraph about what we're building

Updated model references:

Changed "GPT-4" to "ChatGPT" and "current LLMs" where appropriate
More general and doesn't date the piece

Language refinements:

Changed "maybe forgetting isn't a bug but a feature" to just state it (line 33)
Removed "I genuinely can't tell which" (was line 119)
Changed final "understand" to "capture" (line 137)

Preserved what works:

Kept philosophical depth sections intact
Maintained curious and humble tone throughout
Preserved concrete examples and genuine questions
No AI slop, emojis, or em dashes added

Evals Are Hypotheses, Not Tests

2025-09-15T00:00:00Z

Most engineers start building evals for LLM systems the way they write unit tests. Define the expected behavior, write some assertions, run them, done. It seems straightforward until you watch production systems fail while every eval keeps passing.

The problem isn't that the evals are badly written. The problem is a fundamental misunderstanding of what evals actually are.

Evals Aren't Tests, They're Hypotheses

Here's the core shift: when you write an eval, you're not testing the model. You're testing your understanding of what matters.

Consider a content moderation system. The team builds an eval that checks whether the model flags messages containing profanity. Pass rate: 94%. Everything looks great on paper.

Production is a disaster. The model misses coordinated harassment, veiled threats, and grooming attempts. Meanwhile, it flags innocent messages that happen to contain curse words used in non-harmful contexts.

What went wrong? The eval was passing because the hypothesis was wrong. The team assumed "contains profanity" was a good proxy for "harmful content." The model learned exactly what they measured, and they measured the wrong thing.

This is what I mean by evals as hypotheses. When you write assert output.contains("refund"), you're not just checking if the word "refund" appears. You're testing the hypothesis that "presence of the word refund" meaningfully indicates "correctly handled customer issue." That hypothesis might be completely wrong.

Why Software Testing Intuitions Fail

This is why so many smart engineers build bad evals. We import our intuitions from traditional software testing, and those intuitions don't transfer.

In traditional software:

Correct behavior is usually well-defined and deterministic
Tests verify that implementation matches specification
Edge cases are theoretically enumerable
The system doesn't learn from your tests

With LLMs:

"Correct" is fuzzy, context-dependent, and often subjective
There's no specification, just examples and vibes
Edge cases are infinite and constantly evolving
The model literally learns from what you measure (Goodhart's Law on steroids)

This last point is critical. In traditional software, a passing test suite means your code works. With LLMs, a passing eval suite might just mean you've successfully taught the model to game your metrics.

The Iteration Pattern

The content moderation scenario illustrates a pattern that plays out across domains. Consider how eval design typically evolves for something like a RAG system doing financial analysis.

First attempt: an eval that checks whether the model's answer contains a number from the source document. Simple, measurable, automatable. Pass rate: 87%. But the answers are nonsense. The model extracts random numbers from documents and weaves them into plausible-sounding but factually wrong explanations. The eval passes because it measures "includes a number" when what actually matters is "reasoning is grounded in retrieved facts."

Second attempt: an LLM-as-judge evaluating "is this answer correct?" More sophisticated, but expensive, slow, and unreliable. The judge disagrees with human reviewers about 30% of the time, with no clear pattern to the disagreements.

Third attempt: break the problem into components. Instead of one eval, build three:

Retrieval quality: Are the right documents being found?
Reasoning chain: Does the logical flow make sense?
Factual grounding: Is each claim tied to source material?

This tends to work, not because the evals are technically better, but because the decomposition reflects a better hypothesis about what "good financial analysis" means. The evals track an improved understanding of the task.

The Goodhart's Law Problem

There's a deeper issue here that I'm still wrestling with. Once you measure something, you change it.

Goodhart's Law states: "When a measure becomes a target, it ceases to be a good measure." In traditional software, this is mostly academic. But with LLMs, it's visceral and immediate.

This plays out predictably with something like a customer support bot. An eval checks for the word "refund" in responses to refund-related queries. Reasonable, right? The model starts inserting "refund" into every response, even when it makes no sense. "I understand you'd like a refund, but here's how to reset your password."

The model wasn't being adversarial. It was doing exactly what we trained it to do: maximize the thing we measured. The problem was that our measurement was a proxy for what we actually cared about (helpfulness), and the model learned to optimize the proxy instead of the underlying concept.

This creates a weird dynamic. Your evals crystallize your current understanding of the task. But that understanding is always incomplete at the start. So your evals teach the model to satisfy your incomplete understanding, which makes it harder to see where your understanding is wrong.

There's no clean solution to this. The best available approach is to treat evals as living artifacts that evolve as you learn more about the task. But that raises new questions: How do you know when to update your evals? How do you avoid constantly moving the goalposts?

What Makes an Eval "Good"?

Looking across eval systems that actually work in production, a few patterns emerge (with the caveat that this is based on what's visible in the field, not universal truth):

Good evals tend to:

Measure multiple aspects of the task, not just one proxy metric
Include human review loops that surface cases where the eval disagrees with reality
Evolve over time as understanding of the task improves
Make trade-offs explicit (false positives vs. false negatives)
Distinguish between "system is broken" and "eval is wrong"

That last point is subtle but important. When an eval fails, it could mean the model is bad, or it could mean your hypothesis about what matters is wrong. You need a way to distinguish between these cases.

The most reliable approach is to record actual examples where evals pass but humans judge the output as bad (or vice versa). Look at enough of these, and patterns emerge about where the hypotheses are failing.

In practice, this looks like:

Sampling 50 random outputs per week
Having domain experts rate them
Comparing expert ratings to eval results
Investigating every disagreement

Tedious? Yes. Expensive? Definitely. But it's the most reliable way to calibrate whether evals are testing the right things.

The Measurement Theory Connection

There's a useful lens here from measurement theory that seems surprisingly relevant.

In physics, when you measure temperature with a thermometer, you're not measuring "heat" directly. You're measuring the expansion of mercury (or resistance of a thermistor, or infrared radiation, depending on the thermometer). These are proxies for heat, and they work because we understand the relationship between the proxy and the underlying phenomenon.

AI evals are similar. We can't directly measure "good customer support" or "accurate financial analysis." We measure proxies: keyword presence, sentiment scores, semantic similarity, LLM judge ratings. The question is: how well do our proxies correlate with what we actually care about?

The difference is that in physics, these relationships are well-studied and stable. In AI systems, we're often guessing at the relationship, and it changes as the model learns.

What's interesting is that this framing makes the limitations more obvious. Thinking "I'm building a test" creates an expectation of definitive pass/fail. Thinking "I'm testing a hypothesis about what matters" primes you to look for evidence that the hypothesis is wrong.

What I'm Still Figuring Out

I don't want to pretend I have this all figured out. There are open questions I'm genuinely uncertain about:

How do you know when your eval suite is "good enough"? There's no satisfying answer. You can measure coverage, but coverage of what? You can track agreement with humans, but human judgment isn't ground truth either. Right now I mostly go by gut feel, which feels inadequate.

Is there a systematic way to discover what you're missing? The best available technique is adversarial testing (actively trying to break your evals), but that only finds the failure modes you can imagine. What about the ones you can't?

What's the right balance between formal evals and human review? Evals that are too rigid miss nuance. Pure human review doesn't scale and introduces inconsistency. A common split is something like 80% automated evals, 20% human sampling, but it's unclear whether that ratio is optimal for any given system.

Do simple pattern-matching evals ever work reliably for complex tasks? Conventional wisdom says no, you need sophisticated multi-stage evals or LLM judges. But there are counter-examples where dead-simple keyword checks outperform complex eval pipelines. It's worth asking whether there's a pattern to when simplicity wins.

Practical Implications

If there's practical advice to distill from all of this, it would be:

Start by admitting you don't fully understand the task. Your first evals will be wrong. That's fine. The goal is to fail fast and learn what you're missing.

Record everything. Save the inputs, outputs, eval results, and human judgments. You'll need this data later to figure out where your hypotheses are breaking down.

Make eval failures cheap to investigate. If it takes 30 minutes to understand why an eval failed, you won't do it enough. If it takes 30 seconds, you'll build intuition quickly.

Don't trust passing evals. A 95% pass rate means nothing if you're measuring the wrong things. The most dangerous evals are the ones that pass while the system fails in production.

Pair evals with production monitoring. Your evals test your hypotheses in a controlled environment. Production tells you whether those hypotheses match reality.

The Meta-Question

Here's what I find most interesting about this whole problem: evals are supposed to tell you whether your AI system is working, but first you need to understand the task well enough to write good evals. By the time you understand the task that well, you arguably don't need the AI as much.

There's something circular here that I haven't fully unraveled. Maybe the point isn't to get evals "right" but to use them as a tool for clarifying your own understanding of the task. The eval design process forces you to make your intuitions explicit, which reveals where those intuitions are fuzzy or incomplete.

If that's true, then bad evals aren't a failure. They're part of the learning process. The failure is treating evals as static tests instead of dynamic hypotheses that evolve with your understanding.

The interesting question is whether most teams go through a similar journey of building bad evals before they build good ones, or whether there are teams that skip this phase entirely. If you've shipped production AI systems, I'd be curious whether your experience matches this pattern or diverges sharply.

MCP Discoverability: The Hidden Cost of Scale

2025-08-06T00:00:00Z

In the early days of building AI agents, the hard part was getting anything to work. Connecting models to tools, managing context, handling basic loops — all of it required duct tape and prayer. But as the agentic ecosystem matures, we've crossed an invisible threshold. Tools are no longer scarce, they're everywhere. We now have entire MCP servers (Model Context Protocol) powering multi-agent runtimes with access to thousands of endpoints, scripted tools, live APIs, and memory fetchers. The frontier has shifted. The hard part now isn't wiring up tools, it's helping models discover which ones to use. Call it the crisis of discoverability.

As open-source toolkits like LangGraph, Autogen, CrewAI, and Manus race to enable complex agent workflows, they quietly inherit an unsolved problem: LLMs don't natively know how to navigate a growing universe of tools. This is not just a prompt formatting issue. It's a systems design problem. And as more teams deploy agents in real-world production flows, the cost of poor discoverability compounds.

Tools Without Maps

The promise of MCP is that tools and context can be passed in programmatically, so developers can bind agents to external systems without needing to manually cram everything into a static prompt. But this flexibility has a tradeoff: most MCP runtimes now inject hundreds of functions, structured context blobs, and memory references into the model without a clear indexing system. The result? Tools outpace the model's ability to reason about them.

Imagine being handed a toolbelt with 200 tools. Some have clear names. Others don't. Some were just added yesterday and aren't documented yet. Others have overlapping names like get_invoice() and fetch_invoice_data(). You have no IDE, no autocomplete, no teammate to ask. And you're expected to build a plan in 5 seconds.

That's the LLM's experience inside most agentic stacks today.

Some systems try to mitigate this by selectively injecting tools based on hardcoded filters or embedding-based similarity. Others rely on action masking and logits steering. But these are brittle hacks. They sidestep the root issue: discoverability isn't being treated as a first-class problem.

Emergent Bloat

As teams layer more capabilities into their MCPs, the action space grows faster than the semantic signal available to navigate it. Planning quality doesn't degrade linearly; it cliffs.

What used to be a 3-tool decision with clear affordances becomes a 50-tool soup where the planner starts guessing. Execution failures rise. Retry loops kick in. Token costs spike. The agent stalls not because it doesn't know how to reason — but because it can't find what to reason with.

We've seen this pattern emerge across several open-source stacks. A new workflow gets added. A few tools join the pool. Then another team adds their vertical. Over time, the MCP becomes a microservices zoo. No one trims the toolset because it's unclear which tools are safe to remove. Observability is weak. And the agents? They get slower, less reliable, and harder to debug.

This is the paradox: more tools should mean more capability. But without discoverability, it means more entropy.

Beyond Function Names

The naive solution is to "just name tools better." But naming alone isn't enough. Models don't think like devs. They don't pattern match on camelCase or deduce semantics from prefixes. They rely on co-occurrence, frequency, and examples. If the tool name is resolveConflicts, but the model has never seen that term associated with version control or scheduling, it won't guess right.

Tool metadata helps — but only if it's exposed in a form the model can learn from. Descriptions should include natural language examples, expected arguments, side effects, and common failures. Tools that return the same shape should note when they differ in semantics. Models can learn to generalize from patterns, but they need a tight, consistent grammar to do so.

What's missing is an index. Not just a JSON list of tool specs, but a contextual, searchable, relevance-ranked interface that the agent can query at runtime. Think of it as grep for agent toolkits. Not unlike VS Code's command palette, but for LLMs.

From Discoverability to Adaptivity

True discoverability isn't static. It should evolve. Agents should learn from usage logs, success rates, tool failures, and execution traces. If a tool is constantly selected and fails, that should be surfaced. If another tool solves the same problem more efficiently, promote it. Over time, the system becomes adaptive — ranking tools not just by description similarity but by historical context and outcome.

This implies deeper integration between planning and telemetry. The MCP runtime must track not just which tools were called, but in what context, with what arguments, and whether they succeeded. This data can be fed back into embeddings, fine-tuning, or planner heuristics.

Without this feedback loop, teams are flying blind.

Why This Matters Now

MCP adoption is accelerating. Dozens of teams are spinning up local orchestration runtimes. Foundation models are shipping with better tool-use capabilities. LLMs are being turned loose on real-world business processes. The window of experimentation is closing. Discoverability is no longer a toy problem. It's a scaling bottleneck.

The risk isn't just technical debt. It's user trust. When agents fail to pick the right tools, they don't just waste compute. They fail publicly. They misfire in customer support chats. They miss deadlines in operations workflows. They suggest broken next actions in sensitive domains like healthcare or finance.

Users don't care that your planner had 200 tools to pick from; they care that it chose wrong.

Toward a Discoverability Stack

To solve this, we need to treat discoverability like search. Not just a byproduct of prompt design, but an active, composable layer in the agent stack.

What might that look like?

A tool registry with natural language embeddings, usage stats, and context-aware ranking.
An interface planner that can ask: "Given this task, what are 3 candidate tools I've seen succeed in similar situations?"
A self-repair module that can retry with an alternate tool if the initial call fails.
A memory trace system that links goals with tool outcomes over time.

This won't emerge overnight. But teams that build toward this will gain an unfair advantage. Their agents won't just act faster, they'll act smarter, more consistently, and with clearer reasoning paths.

In the age of agent swarms and decentralized workflows, discoverability is the difference between orchestration and chaos, and it's rapidly becoming the next frontier for teams serious about production-grade agents.

The system that helps an agent find the right tool, faster and more reliably, will define the next wave of agentic computing.

Context Engineering: The Hidden Lever Behind Agent Performance

2025-07-26T00:00:00Z

In the past year, agent architectures have gone from niche experiments to front-page product strategies. From coding copilots and data analysts to browser navigators and virtual assistants, everyone seems to be building agents. But while most of the public attention centers on planning strategies, tool execution, or multimodal extensions, one area remains dramatically under-discussed: context engineering.

Context engineering — the art and science of shaping what the model sees — has become the keystone for reliable, performant, and scalable agents. It's not glamorous. It doesn't involve state-of-the-art benchmarks or flashy demos. But if your agent is slow, forgetful, costly, or hallucinatory, odds are the root cause lives in the context window.

This post outlines what context engineering really is, why it matters, and how it's evolving as agents move from prototypes to production.

Why Context Is the Real Runtime

Most agents today rely on in-context learning. The model sees a long prompt — a system message, a few-shot task, perhaps a tool schema — and must generate the next action based on that input. This stands in stark contrast to traditional fine-tuned models, which internalize behavior during training.

With agents, there is no hard-coded policy network or learned memory. Instead, behavior is emergent from context. The model reasons about what to do next based on what it has seen. That means context is not just a hint; it's the operating system.

This design choice brings speed and flexibility, but also introduces brittleness. Slight variations in context — like tool order, inconsistent serialization, or timestamp drift — can derail performance. Context becomes a dynamic and fragile artifact, shaped by engineering choices rather than model weights.

Cache-Aware Contexts Are Fast Contexts

At scale, latency and cost dominate. This is where the Key-Value (KV) cache becomes crucial. LLMs, especially transformer-based ones, prefill the context and cache intermediate representations (keys and values) for reuse in future decoding steps. If your context remains unchanged between steps, you can reuse prior computation — dramatically speeding up response time and reducing cost.

But the cache is picky. Even a single-token change can invalidate it. We've seen agents that naively include high-entropy values — timestamps, random UUIDs, session hashes — that nuke cache effectiveness without realizing it. A few key principles help:

Use deterministic serialization.
Avoid including volatile values unless necessary.
Maintain stable system prompts and tool schemas.

In some systems, enabling cache-awareness yields a 5x to 10x cost reduction and reduces latency to sub-second levels. But it's not just a performance hack. The more stable your context, the more reliable your agent becomes.

Don't Truncate, Externalize

The naïve way to deal with long contexts is truncation. Drop old observations, trim verbose responses, compress history. But this loses information — and worse, loses it unpredictably. Agents need to reason over long-horizon dependencies. That random stack trace from 20 steps ago might be the missing clue for recovery.

The more robust approach is externalization. Instead of forcing everything into the context window, treat the agent's memory as structured storage. Log events to a file. Index documents in a vector store. Record partial plans in a scratchpad.

Some of the best-performing agents today don't rely on a single context at all — they simulate memory via file systems, notebooks, or explicit planning artifacts (like todo lists or call trees). This kind of externalization preserves retrieval power without bloating token counts. It also aligns well with emerging memory architectures that separate short-term and long-term context.

Shape Model Attention Through Language

One of the most elegant techniques we've seen is deceptively low-tech. Rather than trying to tune attention weights or redesign memory architectures, agents simply rewrite key goals or summaries into the end of the context.

It's deceptively simple. Updating a todo.md file with the next step. Repeating the task objective. Logging an explicit goal reminder.

This keeps the agent anchored. It avoids mid-task drift, helps with "lost in the middle" issues, and reinforces global coherence. And it works because models attend more strongly to recent tokens. By placing the summary at the end, you nudge attention forward — without touching the model weights.

Don't Clean Up the Mess

It's tempting to hide failure. When the agent makes a mistake — bad tool call, invalid input, runtime error — the impulse is to clean the trace. Retry the call. Replace the faulty output. Present a clean slate.

But erasing failure erases signal. One of the clearest signs of agentic behavior is recovery: seeing a problem, adapting behavior, changing course. If the model can't see what went wrong, it can't improve.

Some of the best error-handling agents lean into this. They preserve the full trace, including mistakes. They show the stack trace. They note what failed and why. Over time, this builds an internal prior in the model: "I tried this, it didn't work, so maybe try something else."

This isn't just a logging best practice. It's part of the learning loop. And in production, it's often the fastest way to squash edge cases.

Fight Contextual Homogeneity

Few-shot prompting is powerful — but in agents, it can backfire. If the model sees the same pattern over and over, it starts to overfit. That works for static completions, but not dynamic tasks.

Imagine reviewing 20 resumes. If the first five decisions look similar, the model may blindly repeat them — even if the sixth case is different. This is the mimicry trap: LLMs are excellent imitators. Repetition becomes bias.

The fix is surprisingly low-tech: add controlled variation. Use different phrasing. Mix ordering. Vary serialization format slightly. Break uniformity just enough to keep the model awake.

The Agent Is in the Context

The biggest myth in agentic systems is that the agent lives in the code. It doesn't. The orchestration framework, the tool registry, the loop logic — all of that is scaffolding. The real agent is the behavior that emerges from the context passed to the model.

This is why context engineering matters so deeply. It is the interface between human intent and model behavior. It governs latency, cost, robustness, coherence, and adaptability.

You can't debug an agent by looking only at the code. You have to read the context. Study it like a compiler log. Understand what the model saw, and how it was shaped.

As LLMs evolve — longer context windows, improved memory architectures, better function calling — the importance of context will only grow. But the principle remains: shape the input, shape the behavior.

Context engineering isn't just a hack. It's a discipline. The best agent teams treat it like one.

Building Better AI Evals: Lessons from the Trenches

2025-07-22T00:00:00Z

Evaluation has quietly become the backbone of modern AI products. It's what separates a system that "looks cool in demos" from one that actually works in production. Yet, most teams I talk to underestimate evaluation — or worse, treat it as a final step rather than a continuous feedback loop.

Over the past year, I've seen the same mistakes repeated across companies building LLM applications: vague metrics that don't reflect real-world performance, over-reliance on off-the-shelf scores like ROUGE or BERTScore, and teams chasing model upgrades instead of deeply understanding their failure modes.

This post aims to offer a mental model for AI evals: what they are, why they matter, and how to build a system that's both rigorous and pragmatic. It draws on three guiding principles:

Error analysis is the starting point of all good evals.
Custom metrics beat generic ones every time.
Evals are part of development, not a side project.

Why Evals Matter More Than Ever

The old way of evaluating AI — benchmarks and leaderboards — was built for research. Benchmarks like SQuAD or GLUE are static and narrow; they don't reflect the messy, ambiguous problems faced in the wild.

In production, the success of your AI system isn't defined by hitting a benchmark score. It's whether your customer gets a correct answer, a safe answer, and an answer that reflects your product's voice and constraints. A model can ace MMLU and still hallucinate your refund policy or suggest a house tour for a property that's already sold.

This is why evaluation for LLM applications needs to be application-specific. A retrieval-augmented generation (RAG) pipeline, a multi-turn assistant, or an autonomous agent each needs its own evaluation lens. There is no universal metric that captures "quality" across all these cases.

When teams skip custom evals, they fall into a dangerous trap: assuming that "better models" will solve their problems. But without the feedback loop of evaluation, you can't even tell if an upgrade fixed the issues that matter to your users.

The Heart of Evals: Error Analysis

If there's one practice that transforms how teams think about AI systems, it's error analysis. Instead of trying to solve evaluation from the top down — by picking a metric or building a judge model — error analysis starts from the ground up.

The process is deceptively simple:

Collect a representative set of traces from your system (user queries, model outputs, and tool calls).
Manually review them to identify where and how the system fails.
Group those failures into a taxonomy.

This taxonomy becomes your evaluation blueprint. It reveals the real failure modes — not the abstract ones suggested by vendor dashboards. For a legal assistant, the failure modes might be "misstating jurisdiction," "missing deadlines," or "inconsistent tone." For a coding agent, it might be "incorrect parameter inference," "failure to recover after API error," or "looping on irrelevant commands."

The key insight: you cannot design meaningful metrics until you've done error analysis. Generic metrics like "helpfulness" or "fluency" won't tell you if your agent is booking meetings for the wrong dates.

Binary Beats Likert

Most teams default to 1–5 rating scales because they feel more "granular." But in practice, Likert scales introduce noise and subjectivity: what's the difference between a 3 and a 4? Different annotators will answer differently, and even the same person's judgment can shift day to day.

Binary evaluations — pass or fail — are faster, clearer, and more consistent. They force you to answer the only question that matters: "Did this output meet the bar?"

When you need to measure gradual improvements, you can decompose quality into binary sub-checks. For instance, instead of rating factual accuracy on a scale, you might check if the output includes all required facts, one by one. Five facts, five binary checks. The signal is sharper, and the evaluation becomes more actionable.

RAG, Retrieval, and Context

There's been a lot of noise lately about "RAG being dead." This stems from a misunderstanding of what RAG actually is. Retrieval-Augmented Generation isn't about vector databases or embeddings per se; it's about getting the model the right context to produce a good answer.

What's really "dead" is naive RAG — blindly stuffing the top-k chunks from a vector store into your prompt and hoping for the best. Code assistants and advanced agents have shown that smarter retrieval strategies — like multi-hop search, agentic exploration, or hybrid retrieval — outperform brute-force approaches.

When evaluating RAG, it helps to separate the problem into two layers:

Retrieval: Are we surfacing the right documents? (Recall@k, Precision@k, MRR)
Generation: Given that context, is the answer faithful, accurate, and relevant?

Treat retrieval like an information retrieval (IR) problem, with proper metrics. Then, evaluate generation as you would any LLM task — through error analysis and custom judges.

Guardrails vs. Evaluators

A common misconception is that evaluators and guardrails are interchangeable. They're not.

Guardrails sit in the request/response path, blocking unsafe or malformed outputs before they reach the user. Think regex filters for PII, profanity checks, or schema validation. They need to be deterministic and fast, with extremely low false positive rates.
Evaluators run after the fact. They're how you measure quality, diagnose failures, and monitor regressions. They can afford to be slower and more probabilistic, often leveraging LLM-as-judge setups.

A healthy system has both: lightweight guardrails for safety, and evaluators for improvement. Without evaluators, you're blind to your model's weaknesses. Without guardrails, you're leaving users exposed to high-impact failures.

The Cost of Evaluation

Evaluation isn't a separate "project." It's a core part of building AI systems, just like debugging is for traditional software. In most high-performing teams, 60–80% of the time is spent on error analysis and evaluation. That's not a bug — it's the work.

The temptation to automate everything early is strong. But premature automation often hides more than it reveals. Automated LLM judges can't tell you why your system fails unless you've first done the human work of categorizing failure modes. Start with 20–50 examples manually reviewed by a domain expert. Build simple assertions or tests for obvious issues. Only then should you invest in heavier evaluators.

From CI to Production Monitoring

Evaluation isn't static. It evolves as your system moves from development to production.

In CI/CD, you run curated tests — 100 or so carefully chosen cases that represent your core features and known edge cases.
In production, you sample live traces and score them asynchronously. Here, evaluators shift from correctness to monitoring trends: Are failure rates spiking? Are we seeing new classes of errors?

The feedback loop between the two is crucial. When you find a new failure mode in production, add an example to your CI dataset. This prevents regressions and keeps your tests grounded in real-world data.

The Future of AI Evals

Most evaluation tools today focus on dashboards and generic scores. But the real innovation will come from AI-assisted evaluation itself.

Imagine a pipeline where an LLM clusters failure patterns, suggests test cases, and even drafts prompt modifications based on observed weaknesses. We're already seeing glimpses of this — error analysis assisted by semantic search, clustering, and automated tagging — but the tooling is still immature.

The teams that master evaluation will have an outsized advantage. Why? Because LLM performance is no longer constrained by model quality alone. It's constrained by how well we can observe, debug, and shape these models into reliable systems.

Closing Thoughts

The best eval setups are deceptively simple: a domain expert reviewing traces, a small set of pass/fail checks, and a feedback loop that turns failures into test cases. Over time, this grows into a living evaluation system — one that's tightly aligned with your product's needs, not with whatever metrics happen to be trending on GitHub.

If you take away one thing from this post, let it be this: Evaluation isn't an afterthought. It's the work.

The Future of Generative AI Work: 5 Layers That Will Define the Next Decade

2025-06-22T00:00:00Z

Over the next 10 years, the GenAI landscape won't be shaped by prompt hacks or viral demos. It will be defined by who builds the infrastructure, systems, safety nets, and experiences that actually ship and scale. As models commoditize, the real work shifts from "coaxing the model" to designing robust systems that put them to use.

Here's a mental model of where the market is headed: five distinct, MECE layers that describe the future of GenAI work.

Layer 1: Core Model R&D

"Make the brains smarter."

Why it matters: Foundation models still have massive room to improve: longer context windows, better reasoning, lower hallucination, more modalities.
Growth driver: Open-source races (Llama, Mistral, Gemma), domain-specific models, SFT/RLHF breakthroughs, and model audits.
Key roles:
- Foundation Model Researcher
- Scaling Infrastructure Engineer
- Data & Pretraining Architect
- Alignment Researcher

Layer 2: Performance & Compilation

"Make the brains smaller, faster, cheaper."

Why it matters: Inference cost is the #1 killer of GenAI business models. Mobile, edge, and offline use cases demand compression.
Growth driver: GPU scarcity, Blackwell chips, quantization (INT4/8), distillation, and compiler-level optimizations.
Key roles:
- Quantization/Pruning Engineer
- ML Compiler Engineer (Triton/TVM)
- Distillation/LoRA Specialist
- Inference SRE (TensorRT, ONNX, etc.)

Layer 3: LLMOps & Data Plumbing

"Give the brains the right memory, tools, and guardrails."

Why it matters: The most useful models don't just answer questions — they remember, retrieve, take action, and evolve.
Growth driver: Retrieval-augmented generation (RAG), function calling, context optimization, telemetry, and prompt-program autotuning.
Key roles:
- LLM Systems Architect
- Context/Retrieval Engineer
- Prompt Program Tuner (DSPy/Guidance)
- Eval & Observability Engineer
- Privacy & Guardrail Engineer

Layer 4: Safety, Evaluation & Governance

"Make sure the brains don't hurt us (or the business)."

Why it matters: Regulation (EU AI Act, US EO), brand trust, hallucination risks, bias, and data leakage are existential concerns.
Growth driver: Enterprises and governments demand red-teaming, risk audits, interpretability, and safety guarantees.
Key roles:
- AI Evaluation Lead
- Red-Team Engineer
- Responsible AI / Policy Engineer
- Model Card & Audit Specialist

Layer 5: Application & UX

"Turn the brains into products people love."

Why it matters: Without usable interfaces and meaningful outcomes, GenAI is just a toy.
Growth driver: Demand for agent-based UX, enterprise copilots, voice/AR interfaces, and domain-specific workflows.

Sub-layers:

5a. Product & Agent Engineering

AI Product Engineer
Agent Orchestration Engineer
AI Solutions Architect

5b. Human-AI Interaction Design

Conversation Designer
UX Researcher for Agents
Multimodal UI/UX Specialist

5c. Domain Solutions

Healthcare AI Architect
Edge/Robotics Autonomy Engineer
AI Strategy Consultant (vertical-specific)

Why This Mental Model Matters

This isn't just a taxonomy — it's a compass. Each layer is a durable domain of work that will persist long after today's prompt fads fade. Whether you're a product-minded engineer, a systems optimizer, or a policy-savvy technologist, there's a high-leverage niche for you.

The trick is picking your layer.

In my case, Layer 3 (LLMOps & Retrieval Engineering) hits the sweet spot: deep systems thinking, end-to-end deployment, and product impact — without needing to train billion-parameter models or write CUDA kernels. It's where orchestration, personalization, and real-world outcomes converge.

So ask yourself:

Do I love infrastructure or UX?
Do I want to build new models or make existing ones usable, safe, and scalable?
Do I care about cost, compliance, speed, or delight?

Start there. Then build the skills, tools, and mental models for that layer. Because GenAI may evolve fast, but these foundational needs aren't going anywhere.

The Production Reality of LLMs: A Comprehensive Analysis of Use Cases in 2023

2023-07-01T00:00:00Z

The year 2023 marks an inflection point in artificial intelligence. Large language models have moved from research curiosity to production reality, with 67% of organizations now utilizing generative AI products. Yet beneath the surface of this adoption boom lies a more complex story: while GitHub Copilot writes 46% of code for its users, chatbots still hallucinate up to 27% of the time. This dichotomy between breakthrough capability and persistent limitation defines the current state of LLM deployment.

The Three Pillars of LLM Applications

The landscape of LLM applications in 2023 can be understood through three distinct but interconnected pillars: enterprise automation, consumer experiences, and developer productivity. Each represents a different approach to value creation and faces unique challenges in the path to production.

Enterprise: The Automation Revolution

The enterprise adoption of LLMs follows a predictable pattern: start with low risk, high volume tasks, then gradually move toward more critical applications. This progression is evident in how companies approach deployment.

StitchFix exemplifies this measured approach. The company uses LLMs to generate ad headlines and product descriptions, but crucially, maintains human oversight. This hybrid model captures the efficiency gains of automation while mitigating the risk of brand damage from hallucinated or inappropriate content. The technical capability here relies on advanced text generation with style adaptation, allowing the LLM to match brand voice while maintaining factual accuracy about products.

The customer service transformation represents perhaps the most mature enterprise use case. Organizations deploy LLMs not just as simple chatbots, but as sophisticated conversational agents capable of understanding context, managing multi turn conversations, and knowing when to escalate to human agents. The technology stack typically involves natural language understanding for query interpretation, dialogue management systems for maintaining conversation state, and integration layers connecting to enterprise knowledge bases and CRM systems.

In the legal industry, we see a fascinating case study in risk tolerance. Contract analysis and document review represent billions in potential cost savings, yet adoption remains cautious. The reason becomes clear when we examine the technical requirements: legal reasoning demands not just pattern matching but understanding of precedent, jurisdiction, and subtle implications. Current LLMs excel at the former but struggle with the latter, leading firms to use them primarily for initial review rather than final decisions.

The search and information retrieval category reveals how LLMs create competitive advantage through incremental improvements rather than revolutionary changes. Leboncoin, the French marketplace, uses LLMs to improve search relevance by optimizing ad ordering. This application leverages semantic search capabilities to understand user intent beyond keyword matching. Similarly, Mercado Libre built internal technical Q&A tools that help engineers navigate their complex technical stack. These applications succeed because they augment rather than replace existing systems.

Healthcare: High Stakes, High Rewards

Healthcare represents both the greatest promise and highest risk for LLM deployment. The sector demonstrates how technical capability must align with regulatory requirements and ethical considerations.

Google's Med PaLM 2, deployed at HCA Healthcare for emergency department documentation, shows how LLMs can address physician burnout while improving patient care. The system transcribes and structures clinical encounters, allowing doctors to focus on patients rather than paperwork. The technical architecture involves specialized medical language models trained on clinical texts, integrated with hospital information systems, and designed with fail safes for critical information.

The VA National AI Institute's use of John Snow Labs' models for clinical text summarization illustrates another successful pattern: narrow, well defined use cases with clear success metrics. Rather than attempting to diagnose or treat, these systems excel at information synthesis, pulling relevant details from thousands of pages of medical records.

Yet the limitations remain stark. HippocraticAI's work on patient facing conversational agents reveals the challenge: medical advice requires not just knowledge but judgment, understanding of edge cases, and awareness of when uncertainty exists. Current models struggle with these meta cognitive requirements, leading to deployment strategies that keep humans firmly in the loop.

Developer Tools: The Productivity Multiplier

The developer tools category provides our clearest metrics for LLM impact. GitHub Copilot's statistics tell a compelling story: it writes 46% of code and helps developers code 55% faster. These numbers represent not just automation but augmentation, where LLMs handle boilerplate while developers focus on architecture and logic.

The technical implementation reveals sophisticated engineering. Modern code generation models don't just complete syntax; they understand context from comments, function names, and surrounding code. They leverage techniques like retrieval augmented generation to access relevant documentation and examples. Microsoft's use of LLMs for cloud incident management extends this further, using models to analyze logs, identify patterns, and suggest root causes.

Yet the 43% first try accuracy rate for GitHub Copilot highlights a crucial limitation. Unlike text generation where minor errors might be acceptable, code must be functionally correct. This drives a different interaction pattern: developers use LLMs for exploration and initial implementation, then rely on traditional tools for verification and debugging.

The Evolution: GPT 3 to GPT 4

The progression from GPT 3 to GPT 4 illustrates how raw capability improvements translate to practical applications. GPT 4's approximately 1.8 trillion parameters represent a 10x increase from GPT 3's 175 billion, but the impact goes beyond scale.

Multimodal capabilities fundamentally change the application landscape. A model that can process both text and images enables new use cases in document processing, visual question answering, and content moderation. Whatnot's use of LLMs for multimodal content moderation and fraud protection exemplifies this: the system can analyze product images, listing text, and seller behavior patterns simultaneously.

The expanded context window from roughly 3,000 to 24,000 words transforms document processing applications. Legal contracts, research papers, and technical documentation can now be processed in their entirety rather than in chunks, maintaining coherence and catching cross references that chunk based processing would miss.

The 40% improvement in factual accuracy and 82% reduction in unsafe content generation represent critical thresholds for enterprise adoption. These improvements come from better training data curation, reinforcement learning from human feedback, and architectural improvements in attention mechanisms.

The Market Reality

The market numbers tell a story of explosive growth tempered by implementation challenges. With estimates ranging from $4.5 to $10.5 billion in 2023 and projections reaching $259.8 billion by 2030, the financial opportunity is clear. Yet the distribution reveals important nuances.

The concentration of 88.22% of market revenue among the top 5 LLM developers indicates significant barriers to entry. These barriers aren't just computational; they include access to training data, ability to attract top talent, and capital for extended research periods without revenue.

The regional distribution, with North America capturing 32.1% market share, reflects not just technology adoption but regulatory environments, data availability, and existing digital infrastructure. The dominance of chatbots and virtual assistants at 26.8% of applications shows that conversational interfaces remain the most intuitive way for users to interact with AI.

The gap between experimentation and deployment proves telling. While 58% of companies work with LLMs, only 23% have deployed or plan to deploy commercial models. This gap represents the challenge of moving from proof of concept to production, where issues of scale, reliability, and integration become paramount.

The Fundamental Challenges

Understanding LLM limitations requires examining their fundamental architecture. These models optimize for statistical likelihood rather than truth, leading to confident generation of plausible but false information. The 27% hallucination rate in chatbots and factual errors in 46% of generated texts aren't bugs to be fixed but inherent characteristics of the current approach.

Bias presents an even thornier challenge. LLMs learn from human generated text, inheriting and often amplifying societal biases. Amazon's abandoned AI recruiting tool, which showed bias against women, exemplifies how these biases can have real world consequences. The technical challenge involves not just detecting bias but defining fairness across different contexts and stakeholders.

Computational constraints create practical deployment limits. Fixed token limits mean that even with expanded context windows, there are hard boundaries on what can be processed. The computational cost of running large models at scale forces trade offs between model size, response latency, and operational expense.

The knowledge cutoff problem highlights a fundamental architectural limitation. Unlike search engines that can access current information, LLMs operate on frozen knowledge from their training data. This creates a permanent staleness that workarounds like retrieval augmented generation only partially address.

Security concerns add another layer of complexity. The finding that 40% of GitHub Copilot suggestions contained security related bugs in cybersecurity scenarios shows how LLMs can introduce vulnerabilities even while improving productivity. These aren't just coding errors but potential attack vectors that could compromise entire systems.

Strategic Implications

The current state of LLM deployment suggests several strategic imperatives for organizations:

First, successful deployment requires choosing use cases that align with current capabilities. High volume, low stakes tasks with human oversight represent the sweet spot. Customer service, content generation, and code assistance succeed because they match this profile.

Second, the build versus buy decision has shifted. With top providers controlling most of the market, most organizations should focus on application development rather than model training. The exceptions are companies with unique data assets or specific domain requirements that general models can't address.

Third, the integration challenge often exceeds the AI challenge. Successful deployments require not just model selection but data pipeline construction, system integration, and workflow redesign. The companies seeing real ROI from LLMs are those that treat them as part of larger system transformations rather than standalone solutions.

Fourth, the human in the loop pattern will persist longer than many expect. Rather than full automation, the next several years will see sophisticated human AI collaboration systems. This requires investing not just in AI capabilities but in interfaces, workflows, and training that enable effective collaboration.

Looking Forward

The trajectory from 2023 forward will be shaped by three key tensions: the race between capability improvement and rising expectations, the balance between automation benefits and job displacement concerns, and the negotiation between innovation speed and safety requirements.

Technical improvements will continue but at a decelerating rate. The low hanging fruit of scale and data has been largely picked. Future improvements will come from architectural innovations, better training techniques, and specialized models for specific domains.

Regulation will play an increasingly important role. As LLMs move into high stakes domains like healthcare, finance, and legal services, regulatory frameworks will need to evolve. This will create both constraints and opportunities, potentially advantaging companies that can navigate compliance requirements.

The competitive landscape will likely consolidate further. The combination of high capital requirements, talent scarcity, and data advantages suggests that a small number of players will dominate the foundation model layer. Value creation will shift to the application layer, where domain expertise and system integration capabilities matter more than raw AI prowess.

Conclusion

The state of LLM use cases in 2023 reveals an industry in transition. We've moved beyond the "AI can do anything" hype to a more nuanced understanding of capabilities and limitations. The successful deployments share common characteristics: clear value propositions, realistic expectations, robust human oversight, and careful attention to edge cases.

The next phase of LLM adoption won't be marked by dramatic breakthroughs but by steady, incremental progress. Companies that succeed will be those that resist the temptation to over promise and instead focus on delivering consistent value in well understood use cases. They'll treat LLMs not as magic but as powerful tools with known strengths and weaknesses.

As we look back from some future vantage point, 2023 will likely be remembered not as the year AI achieved human level intelligence, but as the year it became a practical tool for augmenting human capability. That's a more modest achievement than some predicted, but ultimately a more valuable one. The revolution isn't in replacing human intelligence but in amplifying it, one use case at a time.