How do you know it works?
I build AI systems on evidence, not vibes.
The oracle tries to predict how I will rule. In both graded tests where all three were scored the same way, the fine-tuned adapter beat plain Qwen but did not beat always taking the agent's recommendation.
Older prompting results are left off, because a later audit found most of them had seen information from after the ruling.
Show as table
| Test | Fine-tuned adapter | Plain Qwen | Always take the recommendation | Items graded |
|---|---|---|---|---|
| Multiple choice, exact (2026-09-29) | 69.1% | 35.5% | 75.0% | 152 |
| Phase 1, matched my ruling (2026-10-06) | 68.9% | 48.8% | 72.2% | 862 (698 with a recommendation) |
Both tests ran before a leak in the training inputs was cleaned up, and neither has been re-run since.
Headline numbers
- Decisions ruled
- 3,971
- Ruled by me
- 2,722
- Overrule rate
- 34.7%of my 2,447 rulings on an agent's recommendation, I chose something else
- Work items done
- 3,065
- Pull requests in the shiplog
- 6,036
- Repositories
- 78
Decisions, the overrule rate and repositories are from the stats snapshot of . Rulings by me, work items and pull requests were counted from the decision ledger, the work queue and the shiplog on .
Some answers so far
- 69%agreement with my own rulings on held-out decisions, up from 49% for the base local model; prompting and retrieval plateaued at 65%; always following the agent's recommendation scored 72.2%Oracle
- 30–40%override rate across ruled decisions, by monthDecision queue
The share of my rulings, week by week, where I chose something other than the option the agent recommended. Weekly rates run from 29.0% to 50.6%. Weeks with fewer than 20 rulings with a recorded outcome are marked as having no data, and the line breaks there.
A ruling counts here when the decision carried an agent's recommendation and the record says whether I followed it. Rulings an agent filed, and rulings with no recorded outcome, are left out. Many of the later rulings were quick multiple-choice clicks in a batch, so part of any drop may come from how the questions were asked. It is not evidence that the agents got better.
Show as table
| Week | Rulings with a recorded outcome | Against the recommendation | Rate |
|---|---|---|---|
| Week of Jul 27 | 1 | 0 | no data |
| Week of Aug 3 | 2 | 1 | no data |
| Week of Aug 10 | 219 | 110 | 50.2% |
| Week of Aug 17 | 81 | 37 | 45.7% |
| Week of Aug 24 | 88 | 40 | 45.5% |
| Week of Aug 31 | 12 | 6 | no data |
| Week of Sep 7 | 87 | 44 | 50.6% |
| Week of Sep 14 | 71 | 21 | 29.6% |
| Week of Sep 21 | 110 | 35 | 31.8% |
| Week of Sep 28 | 614 | 178 | 29.0% |
| Week of Oct 5 | 339 | 114 | 33.6% |
| Week of Oct 12 | 306 | 105 | 34.3% |
| Week of Oct 19 | 249 | 73 | 29.3% |
| Week of Oct 26 | 268 | 84 | 31.3% (partial) |
6,117 pull requests merged across these weeks, and 56.5% of them were linked to a work item.
Each outline is every merge that week. The filled part is the merges linked to a work item.
Pull request counts before Aug 3 are unreliable because of a bug.
Show as table
| Week | Merged | Linked to a work item |
|---|---|---|
| Week of Aug 3 | 1766 | 692 |
| Week of Aug 10 | 329 | 194 |
| Week of Aug 17 | 158 | 72 |
| Week of Aug 24 | 318 | 239 |
| Week of Aug 31 | 119 | 95 |
| Week of Sep 7 | 229 | 130 |
| Week of Sep 14 | 372 | 289 |
| Week of Sep 21 | 324 | 260 |
| Week of Sep 28 | 784 | 417 |
| Week of Oct 5 | 466 | 231 |
| Week of Oct 12 | 417 | 260 |
| Week of Oct 19 | 453 | 301 |
| Week of Oct 26 (partial) | 382 | 277 |
What connects to what
- uses
- related project
- project
- technology
Show every summary as a list
Technologies
Claude
While I am working on replacing Claude Code with my own development system and harness specifically trained for my coding style for now I often use Claude Code.
Claude is my main driver right now I do also use Codex for longer running tasks, particularly computer use tasks.
The main portion of Claude's use is for my "fleet" see that page for more details about the fleet and how its decision system enables me to build faster.
Effect
Effect.ts is one of the main components of all my software. Effect ports a lot of ZIO from Scala to Typescript bringing many functional paridigms.
In my opinion functional programming is one of the key enablers for effective LLM usage. For example, side Effects, a function making changes in an unrelated section of code, degrade LLM's performance even more than humans. Humans can remember many things from a long time in the past, while LLM's can only know whats in their context window. So the more disparate areas of the code that the model has to know means that it fills up its context window faster. This both reduces the size of the code that the LLM can work on it also degrades the LLM's prerformance. (see context rot) So the less you can fill the LLM's context window the better.
This is where making small isolated composable functions reduces the amount the model needs in its context window.
I also find that Effect.ts enables other benefits in very convenient ways, as the Effect.ts docs explain:
When coding agents write a growing share of your software, the bottleneck moves from writing code to trusting it. Effect makes a program’s failure modes, dependencies, and lifecycle visible to the compiler, turning runtime surprises into precise feedback an agent can act on. And when the thing you are building is itself an AI application, flaky providers, retries, streaming, and rate limits are exactly the problems Effect solves out of the box. Effect Intro
Further reading:
PostgreSQL
While I reach for graph databases as my default there is a familiarity and simplicity that comes with a SQL database. I have settled on PostgreSQL as my SQL database of choice because of its resource efficiency and my own familiarity with it from previous jobs.
Pulumi
Throughout my career when I have been doing Infrastructure as Code I have continuously ran into the issue where I needed to either do some hacky solutions or combined IAC and a programming language when using something like Terraform. That is where for my own projects I selected Pulumi.
While there are some other projects like CDK8s I found that Pulumi reflected best how I prefer to code and could work with my functional programming style. Pulumi isn't perfect and there are still object creation requiret, I am able to combine Effect.ts with some of the Pulumi to make my IAC even more robust and one coehesive style.
Further reading:
Python
Python has been the single language that I have used the most throughout my career. While my recent projects are mostly Typescript I do still use python for my machine learning projects. I do find that my familiarity with Python matters less now that LLM's do the coding but for those rare bugs it is nice to go and read code that I am familiar enough with to easily scan.
I used Python as my main language from 2015-2019 at Hewlett Packard Enterprise and Sumato Homes. Then from 2022-2025 at Chegg.
Quint
While I am less familiar with formal verification I am slowly learning as I have my agents create formal static analysis code in quint to verify that I won't reach deadlocks.
My overall philosophy of making deterministic checks applies here as the closer I can make my quint match my code the more that I can feel safe to merge agent code without human review.
TypeScript
While my favorite language to code in by hand used to be Kotlin, I have shifted to typescript.
I adore functional programming and find the advances in languages like Haskell facinating. However the interoperability and lack of widespread corporate deployment has prevented me from learning Haskell as much as I would like. That is where Effect.ts made typescript so enticing, I could use the wonderful ZIO system while also using a language that shows up all over the place.
Further reading:
Projects
Agent fleet
The organized fleet that enables me to code faster as I am only needed for key decisions
Decision queue
Every judgment call is a record.
Architecture checks
Static checks for the architecture, not just the syntax.
Workstation Config
Consistent configuration for all workstations
Dendrite
Injection experiments on language models, scheduled across GPUs.
Application tracker
A job-application tracker with a Firefox extension that fills in forms.
Projects
Oracle
A model that rules the way I do.
69%
Agent fleet
The organized fleet that enables me to code faster as I am only needed for key decisions
Decision queue
Every judgment call is a record.
30–40%
Architecture checks
Static checks for the architecture, not just the syntax.
Workstation Config
Consistent configuration for all workstations
Coding adapter
Teaching a small model to write code my way.
Draft Voice
An adapter that drafts in my own voice.
Research Adapter
Extended Research Agent
Hybrid retrieval (HRE)
Memory for agents with provenance.
Agent Guard CI
Security scanners an agent can't quietly turn off.
Synapse
An agentic assistant orchestration system
Research
Self-reports and what they reflect.
Dendrite
Injection experiments on language models, scheduled across GPUs.
Application tracker
A job-application tracker with a Firefox extension that fills in forms.
Writing
Longer notes on how I check things live on the blog.
- Writings
Why I don't trust agents with access to their own tests
An agent that can reach its own tests can quietly make them easier. I keep the tests apart from the agent, and a stuck agent asks me instead of guessing.
- Writings
How I design: correct the first draft, then decide
I design with AI agents. Their first draft is where my thinking starts, not where it ends. Four small decisions from one evening show how.
- Writings
I don't watch my agents work. I rule and I review.
How I run a fleet of AI agents without watching them: they bank judgment calls, I rule in batches, and enforced checks hold the rules.