How do you know it works?

I build AI systems on evidence, not vibes.

The oracle against the simplest rival

The oracle tries to predict how I will rule. In both graded tests where all three were scored the same way, the fine-tuned adapter beat plain Qwen but did not beat always taking the agent's recommendation.

Multiple choice, exactFine-tuned adapter69.1%Plain Qwen35.5%Always take the recommendation75.0%Phase 1, matched my rulingFine-tuned adapter68.9%Plain Qwen48.8%Always take the recommendation72.2%

Older prompting results are left off, because a later audit found most of them had seen information from after the ruling.

Show as table
The oracle against the simplest rival
TestFine-tuned adapterPlain QwenAlways take the recommendationItems graded
Multiple choice, exact (2026-09-29)69.1%35.5%75.0%152
Phase 1, matched my ruling (2026-10-06)68.9%48.8%72.2%862 (698 with a recommendation)
adapter, still below the recommendation

Both tests ran before a leak in the training inputs was cleaned up, and neither has been re-run since.

Headline numbers

Decisions ruled
3,971
Ruled by me
2,722
Overrule rate
34.7%of my 2,447 rulings on an agent's recommendation, I chose something else
Work items done
3,065
Pull requests in the shiplog
6,036
Repositories
78

Decisions, the overrule rate and repositories are from the stats snapshot of . Rulings by me, work items and pull requests were counted from the decision ledger, the work queue and the shiplog on .

Some answers so far

  • 69%agreement with my own rulings on held-out decisions, up from 49% for the base local model; prompting and retrieval plateaued at 65%; always following the agent's recommendation scored 72.2%Oracle
  • 30–40%override rate across ruled decisions, by monthDecision queue
How often I overrule my agents

The share of my rulings, week by week, where I chose something other than the option the agent recommended. Weekly rates run from 29.0% to 50.6%. Weeks with fewer than 20 rulings with a recorded outcome are marked as having no data, and the line breaks there.

0%25%50%this week, partialJulAugSepOctshort ticks on the axis: too few rulings that week to show a rate

A ruling counts here when the decision carried an agent's recommendation and the record says whether I followed it. Rulings an agent filed, and rulings with no recorded outcome, are left out. Many of the later rulings were quick multiple-choice clicks in a batch, so part of any drop may come from how the questions were asked. It is not evidence that the agents got better.

Show as table
How often I overrule my agents, as of 2026-10-30
WeekRulings with a recorded outcomeAgainst the recommendationRate
Week of Jul 2710no data
Week of Aug 321no data
Week of Aug 1021911050.2%
Week of Aug 17813745.7%
Week of Aug 24884045.5%
Week of Aug 31126no data
Week of Sep 7874450.6%
Week of Sep 14712129.6%
Week of Sep 211103531.8%
Week of Sep 2861417829.0%
Week of Oct 533911433.6%
Week of Oct 1230610534.3%
Week of Oct 192497329.3%
Week of Oct 262688431.3% (partial)
Pull requests merged each week

6,117 pull requests merged across these weeks, and 56.5% of them were linked to a work item.

05001,0001,5002,000AugSepOct

Each outline is every merge that week. The filled part is the merges linked to a work item.

Pull request counts before Aug 3 are unreliable because of a bug.

Show as table
Pull requests merged each week, as of 2026-10-30
WeekMergedLinked to a work item
Week of Aug 31766692
Week of Aug 10329194
Week of Aug 1715872
Week of Aug 24318239
Week of Aug 3111995
Week of Sep 7229130
Week of Sep 14372289
Week of Sep 21324260
Week of Sep 28784417
Week of Oct 5466231
Week of Oct 12417260
Week of Oct 19453301
Week of Oct 26 (partial)382277

What connects to what

Show
  • uses
  • related project
  • project
  • technology
Show every summary as a list

Technologies

  • Claude

    While I am working on replacing Claude Code with my own development system and harness specifically trained for my coding style for now I often use Claude Code.

    Claude is my main driver right now I do also use Codex for longer running tasks, particularly computer use tasks.

    The main portion of Claude's use is for my "fleet" see that page for more details about the fleet and how its decision system enables me to build faster.

  • Effect

    Effect.ts is one of the main components of all my software. Effect ports a lot of ZIO from Scala to Typescript bringing many functional paridigms.

    In my opinion functional programming is one of the key enablers for effective LLM usage. For example, side Effects, a function making changes in an unrelated section of code, degrade LLM's performance even more than humans. Humans can remember many things from a long time in the past, while LLM's can only know whats in their context window. So the more disparate areas of the code that the model has to know means that it fills up its context window faster. This both reduces the size of the code that the LLM can work on it also degrades the LLM's prerformance. (see context rot) So the less you can fill the LLM's context window the better.

    This is where making small isolated composable functions reduces the amount the model needs in its context window.

    I also find that Effect.ts enables other benefits in very convenient ways, as the Effect.ts docs explain:

    When coding agents write a growing share of your software, the bottleneck moves from writing code to trusting it. Effect makes a program’s failure modes, dependencies, and lifecycle visible to the compiler, turning runtime surprises into precise feedback an agent can act on. And when the thing you are building is itself an AI application, flaky providers, retries, streaming, and rate limits are exactly the problems Effect solves out of the box. Effect Intro

    Further reading:

  • PostgreSQL

    While I reach for graph databases as my default there is a familiarity and simplicity that comes with a SQL database. I have settled on PostgreSQL as my SQL database of choice because of its resource efficiency and my own familiarity with it from previous jobs.

  • Pulumi

    Throughout my career when I have been doing Infrastructure as Code I have continuously ran into the issue where I needed to either do some hacky solutions or combined IAC and a programming language when using something like Terraform. That is where for my own projects I selected Pulumi.

    While there are some other projects like CDK8s I found that Pulumi reflected best how I prefer to code and could work with my functional programming style. Pulumi isn't perfect and there are still object creation requiret, I am able to combine Effect.ts with some of the Pulumi to make my IAC even more robust and one coehesive style.

    Further reading:

  • Python

    Python has been the single language that I have used the most throughout my career. While my recent projects are mostly Typescript I do still use python for my machine learning projects. I do find that my familiarity with Python matters less now that LLM's do the coding but for those rare bugs it is nice to go and read code that I am familiar enough with to easily scan.

    I used Python as my main language from 2015-2019 at Hewlett Packard Enterprise and Sumato Homes. Then from 2022-2025 at Chegg.

  • Quint

    While I am less familiar with formal verification I am slowly learning as I have my agents create formal static analysis code in quint to verify that I won't reach deadlocks.

    My overall philosophy of making deterministic checks applies here as the closer I can make my quint match my code the more that I can feel safe to merge agent code without human review.

  • TypeScript

    While my favorite language to code in by hand used to be Kotlin, I have shifted to typescript.

    I adore functional programming and find the advances in languages like Haskell facinating. However the interoperability and lack of widespread corporate deployment has prevented me from learning Haskell as much as I would like. That is where Effect.ts made typescript so enticing, I could use the wonderful ZIO system while also using a language that shows up all over the place.

    Further reading:

Projects

Projects

  • evaluation2026

    Oracle

    A model that rules the way I do.

    69%

  • tooling2026

    Agent fleet

    The organized fleet that enables me to code faster as I am only needed for key decisions

  • agents2026

    Decision queue

    Every judgment call is a record.

    30–40%

  • tooling2026

    Architecture checks

    Static checks for the architecture, not just the syntax.

  • tooling2026

    Workstation Config

    Consistent configuration for all workstations

  • evaluation2026

    Coding adapter

    Teaching a small model to write code my way.

  • evaluation2026

    Draft Voice

    An adapter that drafts in my own voice.

  • agents2026

    Research Adapter

    Extended Research Agent

  • agents2026

    Hybrid retrieval (HRE)

    Memory for agents with provenance.

  • tooling2026

    Agent Guard CI

    Security scanners an agent can't quietly turn off.

  • agents2026

    Synapse

    An agentic assistant orchestration system

  • research2026

    Research

    Self-reports and what they reflect.

  • research2026

    Dendrite

    Injection experiments on language models, scheduled across GPUs.

  • tooling2026

    Application tracker

    A job-application tracker with a Firefox extension that fills in forms.

Photographs

The photographs open on the photography page.

See the photographs

Writing

Longer notes on how I check things live on the blog.

Every post, by section