Not All Thoughts Are Created Equal

Humans instinctively gauge the complexity of a task and decide how much cognitive effort to devote to it. Some problems can be solved swiftly and with minimal thought, while others necessitate intense focus and mental energy. Essentially, humans allocate cognitive resources based on the anticipated difficulty of a task.

Unfortunately, existing AI models fail to replicate this behavior. They allocate the same amount of computational resources to every single prediction, regardless of its complexity. This raises a critical question: could it be that AI models, particularly those based on Transformer architectures like ChatGPT, are consuming more computational power than necessary? More importantly, might this realization lead to smarter AI systems?

Understanding Transformer LLMs

To appreciate the significance of Mixture-of-Depths, we first need to understand how Transformer LLMs operate. Our focus here will be solely on Transformer-based models, as this is the paper’s primary focus. However, it’s worth noting that MoD principles could be extended to other architectures, such as Mamba or Hyena.

Transformers, exemplified by models like ChatGPT, Gemini, and Claude, function as sequence-to-sequence models. They receive an input sequence (e.g., text) and produce an output sequence, typically a continuation of that text. This is achieved through a series of concatenated 'Transformer blocks,' which consist of two main layers:

1. A multi-head attention layer for computing attention mechanisms.
2. A Feedforward Layer (FFN) to enhance feature extraction.

By stacking these blocks, the model ranks all potential output words and selects one of the top-k most plausible continuations.

The architecture may vary from one model to another, but the overall process remains consistent.

The Necessity of Multiple Blocks

The rationale behind utilizing multiple blocks is straightforward: deeper models can capture more intricate relationships among words. However, recent studies suggest that we may be pushing the limits of model depth too far. For further insights, check out this article by Salvatore Raieli.

While depth remains crucial for developing powerful AI models, we are currently not adept at determining the optimal depth for these systems. The fundamental principle of Transformers is that each word in a sequence is updated based on information from other words.

For instance, when asked about the meaning of the word 'bat,' one might respond that its meaning is context-dependent. This is exactly what attention mechanisms do—they adjust the values of words in a sequence to account for their surrounding context.

However, with conventional attention mechanisms, every word in the sequence attends to every other word, which can be costly and, perhaps, unnecessary. This is where MoD offers a solution.

Introducing Mixture-of-Depths

In essence, MoD determines whether each word in a sequence should be updated for every block in the model. This can be illustrated as follows:

'Updated' implies that the word undergoes the attention process previously described.

In this setup, each token (word or subword) is funneled into a 'router' before entering a Transformer block. This router assigns a weight to each word, indicating its relevance to that block. If a word is deemed irrelevant, it is excluded from the attention process.

This routing mechanism is entirely predictable. By establishing a 'compute budget' for each block, we can ascertain the precise number of words that the router selects for computation. As a result, we maintain a static computation graph, allowing for full control over the computing resources utilized for each prediction.

Moreover, the model learns this routing process during training, adapting its parameters to improve decision-making. Researchers experimented with a random routing approach, but it yielded poor results. Instead, the router is trained to effectively evaluate the relevance of each word.

To clarify, every component of a neural network serves a specific function. For instance:

• Attention neurons focus on attention tasks.
• Feedforward neurons specialize in capturing nuances within words.
• Router neurons predict the relevance of words for computation.

As depicted in the previous image, depending on the word currently being predicted, the model can determine which prior words in the sequence are relevant to making that prediction.

Illustrating the Concept

Consider the sequence "Jane is French, born in the capital, and likes ice cream. Thus, Jane was born in…". To predict the next word, which we know is 'Paris', the model must identify the relevant preceding words.

For example, do you think "and likes ice cream" contributes to predicting 'Paris'? Clearly, it does not. This is precisely what MoD aims to eliminate. A traditional Transformer would consider all previous words, including those that provide no value. In contrast, a MoD Transformer intelligently discards irrelevant tokens, thus optimizing computation.

MoD models possess an awareness of the significance of each word in a sequence for making predictions.

Remarkably, a word not selected now may be considered in future predictions, depending on the context.

What Are the Results of Mixture-of-Depths?

Grasping the principles that shape the evolution of cutting-edge AI models can be challenging, but it doesn't have to be. If you're eager to stay updated on the rapid developments in AI and feel inspired to take proactive steps, consider subscribing to my newsletter.

The results achieved by MoD are nothing short of impressive. In comparisons between MoDs and standard Transformers, the former not only demonstrate superior efficiency but also exhibit increased intelligence.

With an 87.5% reduction in capacity—meaning nearly 9 out of 10 words are not computed for any routable block—the model remains competitive with traditional models. This implies that, despite conserving nearly 90% of the computational cost in these layers, the model performs on par with fully allocated compute models. In fact, under the same compute conditions, they outshine their counterparts.

This improvement may stem from the elimination of unnecessary computation, which reduces noise from irrelevant words, leading to a better signal-to-noise ratio overall. Notably, the best outcomes occurred when alternating routable blocks with standard ones, ensuring that potentially valuable words are not consistently excluded.

The Intersection of MoD and Mixture-of-Experts

Mixture-of-Experts (MoE) represents another form of conditional computing. Unlike MoD, which focuses on token selection, MoE divides the model into 'experts,' allowing each segment to specialize in specific topics, enhancing efficiency. Models like GPT-4, Mixtral, Gemini 1.5, and Claude 3 are examples of MoEs.

Fortunately, we don’t have to choose between the two approaches. While MoEs operate in the width dimension, MoD addresses the depth dimension, maintaining a fixed compute budget while intelligently selecting relevant tokens for predictions. These two strategies can be effectively combined.

The researchers behind MoD have explored this combination and achieved promising results, where the integrated model demonstrates lower loss rates within a specific computational budget.

Setting a New Standard

In my view, this research paper has the potential to set a significant precedent. Today's AI systems are impressively powerful yet consume substantial energy. While we should not stifle innovation, it’s crucial that advancements lead not only to more powerful models but also to more sustainable ones.

Interestingly, Mixture-of-Depths appears to play a vital role in both aspects. It fosters the development of more efficient models while enhancing their intelligence by allowing them to allocate computational resources more thoughtfully—similar to how humans apply greater effort to challenging problems for better outcomes.

Furthermore, the concept of models becoming adept at discarding unnecessary computations could become essential as AI evolves toward a future characterized by complex tasks requiring extensive exploration before generating a single token, whether it be a word or a video frame.

In summary, unless we enhance the efficiency and cost-effectiveness of model predictions, the future we appear to be racing towards could very well be dystopian. Thus, we may soon witness a widespread adoption of MoD across various models.

If you found this article engaging, I share similar insights in a more accessible format on my LinkedIn. Feel free to connect with me on X as well.

The first video, "Make AI Models Faster By 50%," delves into the advancements brought about by Mixture-of-Depths. It highlights how this innovative approach can significantly enhance the speed and efficiency of AI models.

The second video, "A New Class of AI Emerges," explores the broader implications of Mixture-of-Depths and its role in the evolution of artificial intelligence technologies.