Training large language models (LLMs) with long contexts has become an important capability as models continue to expand in both size and context length.