For the internet to transform, it required the creation of an entirely new ecosystem: front-end technologies, databases, UX design patterns, and more. In the world of AI, this transformation hinges on improving tokens-per-second performance, as this translates into gains not only in speed but also in cost-efficiency.

But what about the accuracy of LLMs? Isn’t that the main hurdle for enterprise adoption? This is where the concept of the "context window" size becomes critical. The "context window" size refers to the maximum number of tokens (words, subwords, or characters) that a model can process at once or retain in memory while generating responses or analyzing text.

In other words, it defines the length of text—and therefore the scope of "knowledge"—that an LLM can consider when making decisions or predictions. The longer the context window, the more context (and "knowledge") can be incorporated into the LLM’s output. Multiple simultaneous advancements are underway and are expected to converge in the next 2 to 3 years to increase the context window size. These include:

1. Improving hardware: Enhancing GPUs and TPUs (faster computation) as well as memory technology (more tokens stored in memory).

2. Optimizing model architecture: Transitioning to sparse attention mechanisms to focus token processing on the most relevant sections of text.

3. Algorithm advancements: Developing better training techniques and using smaller models to complement larger ones.

4. Software and energy-efficient computational improvements.

All these factors are converging to expand the context window size. In summary, four key trends are occurring simultaneously:

1. The cost of training foundational models is decreasing.

2. The cost of inference itself is dropping (lower per-token inference costs).

3. Tokens-per-second performance is increasing (potentially by 10–20x).

4. The size and utilization of the context window are improving.

Together, these advancements improve not only the cost and speed of LLMs but, most importantly, their accuracy. A larger context window allows for more context to be processed, enabling (with sufficient feedback and fine-tuning) greater precision.

Information sourced from Sangeet Choudary.