In a recent development, internal secrets of OpenAI’s GPT-4 have been leaked. This event has sparked discussions across the artificial intelligence community, given that GPT-4 is a significant progression from its predecessor, GPT-3, in terms of both size and complexity. The advancement in the model’s structure and scale is noteworthy, indicating a new phase in the development of AI models. This blog post aims to provide an in-depth examination of the disclosed details of this advanced model, and to consider the potential impact of this development on the future trajectory of AI.
Size and Structure
GPT-4 stands as a titan in the realm of artificial intelligence models, dwarfing its predecessor, GPT-3, in terms of sheer size and complexity. With an astonishing count of approximately 1.8 trillion parameters spread across 120 layers, GPT-4 is more than ten times the size of GPT-3. This exponential increase in size is a testament to the rapid advancements in AI technology and the relentless pursuit of more powerful and efficient models.
The structure of GPT-4 is equally impressive, employing a sophisticated approach known as Mixture of Experts (MoE). This method involves the use of multiple ‘expert’ models, each specializing in different aspects of the data. In the case of GPT-4, there are 16 such experts, each possessing about 111 billion parameters for the Multilayer Perceptron (MLP), a type of artificial neural network.
However, not all of these experts are utilized at once. In each forward pass – the process of input data flowing through the neural network – only two of these experts are routed. This selective routing is a strategic decision, designed to optimize the model’s performance by focusing on the most relevant experts for a given input.
Inference
Despite the size of GPT-4, it’s designed to operate with considerable efficiency. During each forward pass inference – the process where the model generates a single token or piece of output – only about 280 billion parameters are utilized, requiring approximately 560 teraflops (TFLOPs) of computational power.
To put this into perspective, a teraflop is a measure of computer performance equivalent to one trillion floating-point operations per second. So, GPT-4 is performing 560 trillion operations per second for each token it generates. This might sound like a lot, but when compared to the model’s total capacity, it’s just a fraction.
If GPT-4 were a purely dense model, meaning every parameter is used in every forward pass, the computational requirements would be significantly higher. A forward pass in such a model would require the full 1.8 trillion parameters to be utilized, demanding around 3,700 TFLOPs of computational power.
This comparison highlights the efficiency of GPT-4’s design. Despite its immense size, it’s engineered to use its resources judiciously, focusing on relevant parameters for each token generation. This selective utilization allows GPT-4 to perform at high levels of complexity while managing computational demands effectively.
Dataset and Training
GPT-4’s training process is a testament to the scale and complexity of modern AI models. The model is trained on an enormous dataset comprising approximately 13 trillion tokens. Tokens, in this context, can be thought of as the individual units of data that the model learns from. They could be as small as a single character or as large as a word or even a sentence, depending on the specific implementation.
The training process involves multiple passes over the data, known as epochs. For text-based data, GPT-4 goes through two epochs, while for code-based data, it goes through four. Each epoch represents a complete pass through the entire dataset, so the model has multiple opportunities to learn from the same data and refine its understanding.
The batch size, which refers to the number of tokens that the model processes simultaneously during training, was gradually increased over several days. By the end of the training period, the batch size had reached a staggering 60 million tokens. This large batch size allows the model to process a vast amount of data in parallel, speeding up the training process.
However, due to the Mixture of Experts (MoE) approach employed by GPT-4, not every expert sees all tokens. This is because the MoE approach routes different tokens to different experts based on their relevance. As a result, the effective batch size per expert is reduced to 7.5 million tokens. This approach allows the model to leverage the specialized knowledge of each expert, leading to more efficient and effective learning.
Training Cost
The computational resources required to train GPT-4 are immense. The model’s training involves approximately 2.15e25 floating-point operations, or FLOPs. A FLOP is a measure of computer performance, representing one floating-point operation per second. In the case of GPT-4, the model is performing an astronomical number of operations, highlighting the computational intensity of training such a large and complex AI model.
This training process was carried out on approximately 25,000 A100 GPUs over a period of 90 to 100 days. The A100 is a high-performance graphics processing unit (GPU) developed by NVIDIA, designed specifically for data centers and AI applications. It’s worth noting that despite the power of these GPUs, the model was running at only about 32% to 36% of the maximum theoretical utilization, known as the maximum floating-point unit (MFU). This is likely due to the complexities of parallelizing the training process across such a large number of GPUs.
The financial cost of this training process is also significant. If we assume a cost of about $1 per A100 GPU per hour in the cloud, the total cost for this training run would amount to approximately $63 million. This figure underscores the substantial investment required to develop state-of-the-art AI models like GPT-4. It’s a testament to the resources that organizations like OpenAI are willing to commit in their pursuit of advancing AI technology.
Mixture of Expert Tradeoffs
The Mixture of Experts (MoE) approach, while beneficial in many ways, also introduces certain complexities, especially during the inference stage. Inference refers to the process where the trained model is used to make predictions on new, unseen data.
In the MoE approach, different ‘experts’ or parts of the model specialize in different aspects of the data. This means that for any given token generation, only the relevant experts are utilized, while the rest remain inactive. While this approach allows for more specialized and potentially accurate predictions, it also means that at any given time, large parts of the model are sitting idle, not contributing to the token generation.
This selective utilization of the model impacts the overall utilization rates, a measure of how much of the model’s total capacity is being used. In a perfect scenario, a model would have a utilization rate of 100%, meaning all parts of the model are active and contributing to the output. However, due to the nature of the MoE approach, GPT-4’s utilization rate during inference is significantly lower.
While this might seem like a disadvantage, it’s important to remember that the MoE approach is designed to improve the model’s performance by focusing on the most relevant parts of the model for each token. The trade-off between utilization rate and performance is a strategic decision made by the designers of the model, reflecting the complexities and challenges inherent in developing advanced AI models.
GPT-4 Inference Cost
The process of inference, where the trained model is used to generate predictions or outputs, comes with its own set of costs. For GPT-4, these costs are notably higher than for its predecessor, a 175 billion parameter model known as Davinchi.
Specifically, the inference costs for GPT-4 are approximately three times those of Davinchi. This increase can be attributed primarily to two factors: the size of the clusters required for GPT-4 and the model’s utilization rate.
Firstly, GPT-4 requires larger clusters for its operation. A cluster, in this context, refers to a group of GPUs that work together to process data. The larger the model, the more GPUs it requires to operate, and GPT-4, with its 1.8 trillion parameters, is significantly larger than Davinchi. This means more hardware resources are needed, which in turn increases the cost.
Secondly, GPT-4’s utilization rate during inference is lower than that of Davinchi. As mentioned earlier, due to the Mixture of Experts approach used in GPT-4, not all parts of the model are active at all times. This means that at any given moment, a significant portion of the model’s capacity is not being used, leading to lower utilization rates. Lower utilization rates mean that the resources being used are not being fully exploited, which can also contribute to higher costs.
Multi-Query Attention
OpenAI, like many other organizations in the field of artificial intelligence, employs a technique known as Multi-Query Attention (MQA) in the design of GPT-4. This technique is a variant of the attention mechanism, a fundamental component of many modern AI models, particularly those used in natural language processing.
The attention mechanism allows a model to focus on different parts of the input data when generating each part of the output. In other words, it determines where the model should ‘pay attention’ at each step of the computation. Traditional attention mechanisms use multiple ‘heads’, each of which performs its own attention computation. This allows the model to focus on different aspects of the data simultaneously.
However, Multi-Query Attention takes a different approach. Instead of using multiple heads, MQA uses a single head that can handle multiple queries at once. This allows the model to focus on multiple aspects of the data with a single attention computation, reducing the computational complexity and memory requirements of the model.
One of the main benefits of this approach is a significant reduction in the memory capacity required for the key-value (KV) cache. The KV cache is a component of the attention mechanism that stores the ‘keys’ and ‘values’ used in the attention computation. By reducing the need for multiple heads, MQA reduces the size of the KV cache, making the model more memory-efficient.
Vision Multi-Modal
GPT-4 is not just a text-processing powerhouse; it also incorporates a separate vision encoder, demonstrating its multi-modal capabilities. This means that GPT-4 is designed to understand and interpret not just text, but also visual data, making it a more versatile and comprehensive model.
The vision encoder is separate from the text encoder, but they are not isolated from each other. They interact through a mechanism known as cross-attention. Cross-attention allows the text and vision encoders to share information and influence each other’s outputs. For example, when processing an image captioned with text, the model can use the text to help understand the image, and vice versa.
This additional vision encoder adds more parameters to the model, on top of the already staggering 1.8 trillion parameters of GPT-4. This increase in parameters reflects the added complexity of processing visual data, which requires different techniques and representations compared to text data.
After the initial pre-training phase, which is focused on text data, the vision model is further fine-tuned with an additional ~2 trillion tokens. These tokens represent visual data, allowing the model to learn how to interpret and generate visual content.
This incorporation of a vision encoder in GPT-4 is a significant development, extending the model’s capabilities beyond text and into the realm of visual data.
Speculative Decoding
Speculative decoding is a technique that’s being discussed in relation to GPT-4’s inference process. This approach is a strategic method to enhance the efficiency and speed of the model’s operation, particularly during the inference stage where the model generates predictions or outputs.
The concept behind speculative decoding is relatively straightforward. It involves the use of a smaller, faster model, often referred to as a ‘draft’ model, which is used to decode or generate several tokens in advance. Tokens, in this context, can be thought of as the individual units of data that the model generates as output.
Once the draft model has generated these tokens, they are then fed as a single batch into a larger, more accurate model, often referred to as an ‘oracle’ model. The oracle model then processes these tokens, either confirming the predictions made by the draft model or making adjustments as necessary.
The advantage of this approach is that it allows for faster token generation, as the draft model can operate more quickly than the oracle model. It also allows for more efficient use of computational resources, as the oracle model only needs to process a single batch of tokens, rather than generating each token individually.
However, it’s important to note that this is currently speculative, and it’s not confirmed whether OpenAI is indeed using this approach in GPT-4. If they are, it would represent another innovative strategy to manage the computational demands of such a large and complex model.
Inference Architecture
The inference process for GPT-4, where the model generates predictions or outputs, is a computationally intensive task that requires significant hardware resources. To manage this, the inference process is run on a cluster of 128 Graphics Processing Units (GPUs). GPUs are specialized hardware designed for handling the kind of high-volume, parallel computations that are common in AI and machine learning.
However, a single cluster of GPUs is not sufficient to handle the demands of GPT-4. Therefore, multiple clusters are used, distributed across different datacenters. This distributed setup allows the computational load to be spread across multiple locations, improving the efficiency and reliability of the model’s operation.
The computations performed by GPT-4 during inference are organized using two key techniques: 8-way tensor parallelism and 16-way pipeline parallelism.
Tensor parallelism is a method for distributing the computation of tensor operations (the fundamental operations in deep learning) across multiple GPUs. In 8-way tensor parallelism, each tensor operation is divided into eight parts, each of which is computed by a different GPU.
Pipeline parallelism, on the other hand, involves dividing the model’s layers across multiple GPUs. In 16-way pipeline parallelism, the model’s layers are divided into 16 groups, each of which is processed by a different GPU. This allows for the simultaneous computation of multiple layers, speeding up the overall processing time.
These parallelism strategies are crucial for managing the computational demands of GPT-4, allowing the model to operate efficiently despite its enormous size and complexity. They represent some of the advanced techniques used in modern AI to scale up model size and performance.
Dataset Mixture
Training an AI model like GPT-4 involves feeding it a vast amount of data, known as tokens, from which it can learn. In the case of GPT-4, this amounted to an astonishing 13 trillion tokens. These tokens represent the raw material from which the model learns, encompassing a wide range of data types, including text, code, and potentially, visual data.
Two significant sources of these tokens were CommonCrawl and RefinedWeb, each contributing 5 trillion tokens to the total. CommonCrawl is a nonprofit organization that crawls the web and freely provides its web crawl data, which is a snapshot of a significant portion of the internet. RefinedWeb, on the other hand, is a cleaned and processed version of web crawl data, providing high-quality, web-scale datasets for machine learning models.
However, these two sources account for only 10 trillion of the 13 trillion tokens. The remaining 3 trillion tokens are rumored to have come from various other sources, including popular social media platforms like Twitter and Reddit, as well as the video-sharing platform YouTube. These platforms host a vast amount of user-generated content, providing a rich and diverse source of data for training AI models.
It’s important to note that these are rumors, and the exact composition of the training data used for GPT-4 has not been officially confirmed by OpenAI. However, the use of such diverse data sources would be consistent with the goal of creating a model capable of understanding and generating a wide range of content.
Conclusion
In wrapping up, the leaked specifics of GPT-4 unveil an AI model that is unparalleled in both its magnitude and intricacy. The revelations underscore the significant advancements that OpenAI has achieved in the realm of artificial intelligence, continually stretching the limits of feasibility and pioneering new frontiers.
However, it’s crucial to note that the information presented in this blog post is based on leaked details, and as such, there may be elements of inaccuracy or incomplete information. The precise and official details of GPT-4’s architecture and functionality can only be confirmed by OpenAI. The source of the leaked information can be accessed here. and here.
Despite the potential uncertainties, the leaked details offer a fascinating glimpse into the scale and sophistication of GPT-4, and by extension, the rapid pace of progress in the field of AI. As we continue to observe and analyze these developments, we look forward to the official release of GPT-4 and the new possibilities it will undoubtedly bring.
