In this work we demonstrate that LLMs performance affected by the relative distance between pieces of information in the context. The further apart the information is within long context, the more the model’s performance deteriorates.

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%.

This system-model co-design work focus on leveraging syncronization in data parallelism hierarchical partitioning to avoid race conditioning in gradient update for LLM training.

We formulate CoT as a reasoning graph and propose a prompt strategy for multi-step reasoning that can capture complex processes in tasks such as mathematics and commonsense reasoning.

The race condition between AllGather and device-to-device copy for the 2nd partition causes instability in training large models such as Llama-7B and Falcon-40B on a moderately large number of GPUs. After discovering the algorithmic issue, we landed the fix in the DeepSpeed repository.

We propose a framework for understanding how Data Aaugmentation interacts with class-level learning dynamics. We show that simple class-conditional augmentation strategies informed by our framework improve performance on the negatively affected classes.