Multi-Token Prediction

Multi-token prediction vs Next-token prediction Next-token prediction is the standard training objective for most large language models (LLMs), where the model learns to predict the subsequent token in a sequence given all preceding tokens. The model is trained to maximize the probability of the next token \( x_{t+1} \) given the context \( x_{1:t} \) (all tokens up to position \( t \)). The cross-entropy loss for next-token prediction is defined as: ...