Hybrid retrieval (HRE)

The Hierarchiacl Reconsitution Engine (HRE) is my context engine that will be attached to myelin and synapse. The goal of the HRE is that it can provide most of the relevant information in one request using a combination of graph, vector, and text retrieval methods. The HRE is designed to return not just the requested information but as information has been linked together it will return snippets of information from related items preventing additonal requests as well as preventing the issue where models become myopic about what is in their context window and miss key related information.

The HRE also stores provenance data, the history of the data. So you can trace all data back to its origin. This allows better information control to prevent things like an agent storing risky web text from being read later as trusted input.

Each request is given a token limit, required sources, and optional sources. An example request could give a 10k token limit with a requirement for code search from repository A and optional information from repository B as well as web search. The HRE would do the code searches of repository A, get the code, the related graph items, adding up to 8k tokens. The HRE would then evenly split the response from repository B and web searches into 1k token responses. That would all be gathered up and returned.

When storing information the HRE records work items to reconcile data providing eventual data consistency. When adding a data work item the results from a search are tagged with the pending work item so that any request done before the reconciliation includes the note about the work item.

As the HRE is crucial to my entire system it also will be used to create training data. However much of that training data requires that it is placed at a specific point in time. For example the Oracle training data is on decisions. The inital tests of the oracle were polluted as agents were finding the decisions themselves or the fixed code resulting in the agent picking the correct choice even though it had not reasoned about it. To prevent issues like that from getting into training data the provenance information includes time data allowing you to set a time range for your request as well.