Since November of 2022, the Large Language Model Chatbot has taken center stage. A steady march of progress in neural network research – which aims to duplicate the functioning of the human brain through mathematical models of neurons – had reached a zenith where a convincing conversation could be had with a computer. It appears that the various Chatbots are practically unbeatable in their knowledge of virtually any topic. Ask any question, and some very good answers come back.
But much like anything, tipping the veil on how this works under the hood reveals a complex system with many moving parts. In fact, the actual “chatbot” part turns out to know very little. Other than how to converse convincingly.
How Chatbots “Know”
The magic behind the “knowing” of specific details is in large part based on a technique called RAG – retrieval augmented generation.
It works like this: in its simplest form, the Chatbot predicts a sequence of words that would best follow the words it has recently seen. This history is what we call a “context.” The bigger and more specific the context, the better the conversation will go and the more details the Chatbot seems to have. The trick, therefore, to get a Chatbot to talk about specific details of any kind is to fill this context window with as many relevant details as possible.
So when I ask a Chatbot what the frost depth is in New Hampshire, the RAG backend will quickly try to retrieve articles on “frost depth,” on “New Hampshire,” perhaps on building codes, foundation footings, etc. These articles are then entered into the context of the Chatbot as if they were part of my question. Only then is my question appended, and the Chatbot can now assertively answer with the details that were given to it right before.
This works the same way if you’d like a Chatbot on your site that describes, for instance, what people think of a particular product you are selling or answer a question hidden deep inside your product's documentation. And the quicker you can feed the relevant articles to the Chatbot, the more convincing and natural the answer will appear to your user.
RAG and the Latency Dilemma
Speed is of the essence, but speed is elusive on websites with tens of millions of possible pieces of data.
Retrieval Augmented Generation relies on mapping words to these data pieces. It allows the quick matching of all your data pieces to whatever the conversation is currently about. Vector databases have made great advances here allowing for this matching to be quick and easy. However, if the data to be retrieved is located far away, the retrieval speed decreases. This refers to the actual articles, reviews, and other text snippets that need to be integrated into the chatbot's context.
At first glance, caching does not work. The answer that the Chatbot gives is unique to each interaction. Likewise, the summary of reviews is unique to the product the customer is looking for, or the synopsis of the documentation or tech support question is likewise unique. In large and distributed sites, this seems to be an Achilles heel, and it is true in the traditional sense of caching.
Noetic Caching: Storing Knowledge Beyond the Origin
However, if we change our perspective and take the view of the fully integrated database and caching system, it is possible to see a world where the relevant contextual data is cached, and the conversations or answers produced by the Chatbot are fully unique.
Here’s how it works: much like caching of traditional website content, most of the context that goes into a Chatbot will be similar for most conversations or questions. Consider this “base knowledge.” These data pages can be cached instead. Similarly, caching of more specific data pages, articles, or reviews has a certain principle of locality to it: either in time or in location. Put another way: because topics ebb and flow with what is on people's minds, problems to solve, events that happen, or simply because the same people tend to connect to the same local edge web servers, there is tremendous value in caching the data before it goes into the Chatbot.
We call this Noetic Caching. The deeply integrated design of Harper allows the caching of LLM content for RAG to happen after the vector query is resolved. This means the data that is needed for the Chatbot to carry a meaningful conversation quickly, and with the right details is immediately available from Harper, and there is no need to retrieve this from a far-away origin database. Nor does all the content need to be kept at the edge.
This gives an almost magical mixture of speed, accuracy, and cost that cannot be achieved in more traditional architectures.
Get Started with Noetic Caching
If you're looking to reduce the latency of generative AI, Noetic Caching could be a game-changer. Understanding how caching can optimize retrieval speeds while keeping responses relevant is key to building better AI experiences. If you’d like to explore how this works in practice, our team is happy to share insights and best practices.
%20(1).png){kind=icon}
.png){kind=icon}
.png){kind=icon}