Research Adapter

As part of Synapse getting information from various sources will be very important. I need accurate information if I want my agents in Synapse to work autonomously on complex issues. In addition if I need information Synapse is setup for longer agent-human interactions. That means that I don't care if I ask a question and then get an answer 10 minutes later if the answer is accurate and well researched. This is not how current models are trained to work. Things are heading in that direction but still there is a huge expectation that models will have memorized much of the data.

I don't want the models to have memorized a lot of facts. When I ask what Napoleon was doing in 1820 I don't want the model to hallucinate such a specific fact. I want it to know that 1820 is a year and Napoleon was an important historical figure. That is enough information for the model to go and search up the relevant information, compare sources, and give me a clear writeup about my question.

Since I want this specific behavior combined with using the Synapse environment I will need to make custom training data and environments to train the LoRA adapter that this agent will use. It would be ideal to have at least 10 million tokens of training data which will need to include multiple searches per session. When I do the math on the number of searches I would like I find that I will require ~1 million web searches.

As I have started to build this dataset I was using the OpenRouter web search ability. The prices quickly ballooned with it being 0.5 cents per request. I decided to pause building this dataset till I have my own search to avoid breaking the bank. I am estimating that I should be able to get enough data with ~ 50 million web pages.

This search engine is being tied directly into the HRE which will allow me to combine multiple web searches into one response and tool call saving on agent turns which in turn speeds up my agents.