CoT-ICL Lab is a framework to study chain-of-thought (CoT) and in-context learning (ICL) by decoupling causal structure from token processing functions. Experiments show that 1) CoT accelerates the accuracy transition to higher values across model sizes 2) deeper models require fewer in-context examples to leverage CoT effectively while more examples help shallow models match deeper model performance. Along with detailed analyses we provide theoretical insights. The code is available here.

In this work we show how to leverage On-Policy forward Knowledge Distillation, Model Compression (Pruning and Quantization), and Serving Optimizations (detailed on RadixAttention, FlashInfer, and tensor parallelism) to deliver a 20x reduction in both cost and latency while maintaining the quality of our 360Brew XL model.

In this report, we demonstrate that 360Brew model, a 150B parameter foundation model trained on 1T tokens can solve over 30 personalization tasks on LinkedIn platform without task-specific fine-tuning and no complex feature engineering. It can generalize to out-of-domain tasks and surfaces, and achieves performance similar to or better than the production model.

In this work we demonstrate that LLM's performance is affected by the relative distance between pieces of information in the context. The further apart the information is within long context, the more the model’s performance deteriorates.

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%.

This system-model co-design work focuses on leveraging synchronization in data parallelism hierarchical partitioning to avoid race condition in gradient updates for LLM training.

We formulate CoT as a reasoning graph and propose a prompt strategy for multi-step reasoning that can capture complex processes in tasks such as mathematics and commonsense reasoning.

The race condition between AllGather and device-to-device copy for the 2nd partition causes instability in training large models such as Llama-7B and Falcon-40B on a moderately large number of GPUs. After discovering the algorithmic issue, we landed the fix in the DeepSpeed repository.

We propose a framework for understanding how Data Augmentation interacts with class-level learning dynamics. We show that simple class-conditional augmentation strategies informed by our framework improve performance on the negatively affected classes.