Introduction

A lot of basic, important information about transformer language models can be computed quite simply. Unfortunately, the equations for this are not widely known in the NLP community. The purpose of this document is to collect these equations along with related knowledge about where they come from and why they matter.

Note: This post is primarily concerned with training costs, which are dominated by VRAM considerations. For an analogous discussion of inference costs with a focus on latency, check out this excellent blog post by Kipply.

Compute Requirements

The basic equation giving the cost to train a transformer model is given by:

$$ C\approx\tau T = 6PD $$

where:

• $C$ is the compute required to train the transformer model, in total floating point operations
• $C=C_{\text{forward}}+C_{\text{backward}}$
• $C_{\text{forward}}\approx2PD$
• $C_{\text{backward}}\approx4PD$
• $\tau$ is the aggregate throughput of your hardware setup ($\tau=(\#\text{GPUs}) \times (\text{Actual FLOPs}/\text{GPU})$), in FLOPs
• $T$ ****is the time spent training the model, in seconds
• $P$ ******is the number of parameters in the transformer model
• $D$ is the dataset size, in tokens

These equations are proposed and experimentally validated in OpenAI’s scaling laws paper and DeepMind’s scaling laws paper. Please see each paper for more information.

It’s worth taking an aside and discussing the units of $C$. $C$ is a measure of total compute, but can be measured by many units such as:

• FLOP-seconds, which is in units of $[\frac{\text{Floating Point Operations}}{\text{Second}}] \times [\text{Seconds}]$