A language model remembers nothing from one question to the next. With every answer, it rereads everything it is sent: the prompt, the conversation history, the document passages and the new question. The context window is the limit of what it can read in one go. It is measured in tokens, fragments of words, and varies widely from one model to another.
What fills the window
In a business assistant, the window fills with several elements at every message:
- The system prompt: the assistant's role, tone and rules, described in the prompt
- The conversation history: previous questions and answers, so the model can follow the thread
- The passages retrieved from the knowledge base to answer the question
- The visitor's question and, if the assistant can take action, the description of the available tools
- The answer itself, which also takes up part of the window while it is written
If the total goes beyond the window, something has to be cut. Well-designed systems first summarize or drop older messages of the conversation, then limit the number of document passages, rather than truncating the prompt.
How big, in practice?
The first mainstream models read a few thousand tokens, a few pages. Recent models accept hundreds of thousands of tokens, and some go beyond a million, the equivalent of several books. These figures change with every generation of models and can be checked in the provider's documentation.
A large window does not mean you should fill it. The longer the text sent, the more the answer costs, the longer it takes, and the more likely the model is to overlook a passage buried in the middle of a long context. Researchers have observed that models often make better use of information placed at the start and end of the context than in the middle.
Send everything or select?
With windows of several hundred thousand tokens, two approaches coexist for giving documents to the model:
- Sending all the useful knowledge: for a modest knowledge base, like an SME's, the documents go into the window in full. The model sees every rule with its full context, and no retrieval step can miss the right passage
- Selecting passages: for a large knowledge base, a search, often by meaning through RAG, picks the few most relevant passages, and only those go into the window
The first approach uses more tokens per question but avoids passages that are missed or cut off from their context. The second is essential as soon as the documents exceed the window, and lowers the cost, at the price of a retrieval step that can get it wrong. In both cases, sending contradictory or outdated documents remains the main source of errors: sorting the base matters more than the method.
Context window and memory
The context window is not a lasting memory. When a conversation ends, its content no longer exists for the model. What the assistant "knows" about a visitor in a new conversation comes from a separate mechanism, such as contact memory, which stores a few chosen pieces of information and puts them back into the window when needed.
This separation is useful for data protection: what is not explicitly kept disappears with the conversation, and what is kept can be viewed, corrected or deleted.
What this changes for an SME
For an organization that puts an assistant online, the context window mostly has practical consequences:
- Long conversations can lose their first exchanges; the assistant then has to ask again for some information or rely on a summary
- Very long documents benefit from being structured into clear sections, so the search extracts the right passages
- The usage cost depends partly on the amount of text sent; a concise prompt and well-chosen passages reduce it
In KernChat, each message sends the model the assistant's mission, the useful history of the conversation and the sources of its knowledge spaces, in full and in priority order, within a size limit set for each assistant. When the sources exceed that limit, those sharing the most words with the question go in first. The large windows of current models make this approach possible for an SME's documentation, with answers that cite their source.