Oracle

My development process often crawls to a halt when the pile of decisions gets to large. Every agent is blocked and there are no work items to launch. Life gets busy and so I might not be able to sit down for hours answering agent decisions for a few days leading to these piles and wasted time. There are two issues with this, one that agents often ask questions that are not too complicated if they were to check the right decision history. The second is that the sheer number of decisions can be immense, sometimes a hundreds in a single day. Agents are supposed to mark a "recommended" option when they submit a decision but I decide something different 30-40% of the time so letting that many bad recommendations past would result in absolute messes of software and not the quality that I am comfortable shipping.

This is where I tried several rounds of prompt engineering.

< insert data>

Unfortunatly none of these was good enough that I would want to ship them either. Also when looking at the data most of the time the agent just took the recommended option which automatically got them a better score without any of the actual skill. I have a set of decision themes mined from my decision history but even with those in hand agents are still not able to catch my exceptions.

That is where I have graduated to fine-tuning. I need to teach the model how to decide like me not just a list of themes it can use.

Fine-tuning raised plain Qwen from 48.8% to 68.9% of answers that matched my ruling on 862 held-out questions, but it did not beat simply following the agent's recommendation, which scored 72.2% on the 698 questions that had one, so a re-test is planned.