{"id":74,"date":"2025-07-06T14:16:44","date_gmt":"2025-07-06T06:16:44","guid":{"rendered":"https:\/\/firooz.us\/blog\/?p=74"},"modified":"2025-07-19T01:44:48","modified_gmt":"2025-07-18T17:44:48","slug":"my-long-running-notebook-of-ai-llm","status":"publish","type":"post","link":"https:\/\/firooz.us\/blog\/index.php\/2025\/07\/06\/my-long-running-notebook-of-ai-llm\/","title":{"rendered":"My Long-Running Notebook of AI &amp; LLM"},"content":{"rendered":"\n<blockquote class=\"wp-block-quote is-style-default is-layout-flow wp-block-quote-is-layout-flow\">\n<p><em>\u201cI never start a project without skimming my own notes first.\u201d<\/em><br>\u2014 Every engineer the night before a deadline<\/p>\n<\/blockquote>\n\n\n\n<p>I&#8217;ve been writing down my AI\/LLM notes in last 6 months\u2014what started as a few scrappy records about Transformer quirks has turned into a 75 pages of tips that spans everything from model tweaks to prod inference hacks. I&#8217;m sharing it publically so one can:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Refresh<\/strong> the basics<\/li>\n\n\n\n<li><strong>Jump<\/strong> into a new corner new concept fast<\/li>\n\n\n\n<li><strong>Prepare<\/strong> for an AI fundamentals interview without opening 50 tabs<\/li>\n<\/ul>\n\n\n\n<p class=\"has-medium-font-size\"><a href=\"https:\/\/docs.google.com\/document\/d\/1NIHvWc086XEeRbsHItctXzPCazO2Frur0WIyLDR39Zk\/\"><strong>Link to my notes<\/strong><\/a><\/p>\n\n\n\n<p>Scroll down for a quick tour and tips on getting the most out of it.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">1. Architectures: Where It All Begins<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Transformers, RoPE, and friends<\/strong> \u2013 A concise recap of attention basics and how to scale up rotary position embeddings when your context grows past the original context the model was trained on.<\/li>\n\n\n\n<li><strong>Beyond vanilla<\/strong> \u2013 Notes on Mixture\u2011of\u2011Experts routing, grouped\u2011query attention, and \u201cswitch\u201d layers for bandwidth\u2011friendly scaling.<\/li>\n\n\n\n<li><strong>Long\u2011context toolkit<\/strong> \u2013 FlashAttention\u20112, context\u2011window grafting, and knob\u2011turning for retrieval\u2011augmented pipelines.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2. Training at Scale: Beyond Data Parallelism<\/h3>\n\n\n\n<p>The techniques below kick in once <strong>plain data parallelism tops out<\/strong>\u2014think training runs on <strong>512+ GPUs<\/strong> or context windows stretching past <strong>64\u202fK tokens<\/strong>.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><th>Technique<\/th><th>TL;DR in the notes<\/th><\/tr><tr><td><strong>Tensor Parallelism<\/strong><\/td><td>Shard massive weight matrices column\u2011wise <em>or row\u2011wise<\/em> so each GPU hosts a slice; synchronise activations over high\u2011bandwidth links such as NVLink.<\/td><\/tr><tr><td><strong>Pipeline Parallelism<\/strong><\/td><td>Classic assembly line\u2014partition layers into stages and hide latency with micro\u2011batches.<\/td><\/tr><tr><td><strong>Sequence Parallelism<\/strong><\/td><td>Give each worker a chunk of the token stream and overlap compute\/communication to keep GPUs busy.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">3. Datasets: Building the Fuel<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Source Mixes<\/strong> \u2013 OpenWeb, Common Crawl, curated corpora, code, synthetic dialogues; pros\/cons and licensing notes.<\/li>\n\n\n\n<li><strong>Cleaning &amp; Deduplication<\/strong> \u2013 Detect near\u2011duplicates with MinHash\/SimHash, strip profanity, and nuke boilerplate.<\/li>\n\n\n\n<li><strong>Domain Balancing<\/strong> \u2013 Up\u2011weight niche domains (medical, legal) without starving general language coverage.<\/li>\n\n\n\n<li><strong>Synthetic &amp; Augmented Data<\/strong> \u2013 Self\u2011instruct, RAG\u2011generated Q&amp;A, tool\u2011augmented reasoning traces.<\/li>\n\n\n\n<li><strong>Evaluation Splits<\/strong> \u2013 Leak\u2011proof dev\/test and benchmark alignment tips.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4. Data &amp; Scaling Laws: Feeding the Beast<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Dataset Curation<\/strong> \u2013 Practical heuristics for filtering web text, multilingual balancing, and deduplication without blowing up compute.<\/li>\n\n\n\n<li><strong>Token Mixtures<\/strong> \u2013 Why code, math, and synthetic Q\u2011A boost downstream zero\u2011shot tasks; quick recipes for ratio tuning.<\/li>\n\n\n\n<li><strong>Chinchilla\u2011style Scaling Laws<\/strong> \u2013 The compute\u2011optimal sweet spot (\u224820 tokens per parameter) and how to project loss at larger model sizes.<\/li>\n\n\n\n<li><strong>Dynamic Data Pacing<\/strong> \u2013 Curriculum vs self\u2011curriculum, temperature\u2011based sampling, and tricks like \u201ctoken replay\u201d for long\u2011tail skills.<\/li>\n\n\n\n<li><strong>Tracking Data Quality<\/strong> \u2013 Per\u2011source perplexity dashboards and the \u201cmarginal utility of another billion tokens\u201d checklist.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">5. Inference Tricks &amp; Optimisations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Speculative Decoding<\/strong> \u2013 Pair a small draft model with a larger verifier to double effective tokens\u2011per\u2011second.<\/li>\n\n\n\n<li><strong>KV\u2011cache management<\/strong> \u2013 Bucketing, sliding windows, and other memory\u2011saver moves.<\/li>\n\n\n\n<li><strong>Quantisation cheatsheet<\/strong> \u2013 What survives INT8 vs FP8, and a one\u2011pager on GPTQ hyper\u2011params.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6. Alignment, Reasoning &amp; RL<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Reward Modeling 101<\/strong> \u2013 From scalar preference labels to pairwise ranking loss; why good rewards beat sparse accuracy metrics.<\/li>\n\n\n\n<li><strong>RLHF &amp; Friends<\/strong> \u2013 PPO vs. simpler DPO\/IPO methods; where RLAIF (AI feedback) slots in when human labels run dry.<\/li>\n\n\n\n<li><strong>Constitutional &amp; Rule\u2011based Policies<\/strong> \u2013 Self\u2011critique loops, safety layers, and how to encode \u201cdon\u2019t do that\u201d without killing creativity.<\/li>\n\n\n\n<li><strong>Reasoning Boosters<\/strong> \u2013 Chain\u2011of\u2011Thought, Self\u2011Consistency, Tree\u2011of\u2011Thought, ReAct, and graph\u2011based planners\u2014what works when tokens are precious.<\/li>\n\n\n\n<li><strong>Benchmarks &amp; Eval<\/strong> \u2013 GSM8K, MATH, BBH, AGIEval, and reward\u2011hacking pitfalls (the \u201cWireheading Watchlist\u201d).<\/li>\n\n\n\n<li><strong>Practical Tips<\/strong> \u2013 Start with supervised fine\u2011tuning, scale label diversity, and monitor KL divergence to keep models \u201con policy\u201d.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">7. Safety &amp; Governance<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Prompt\u2011level Filtering<\/strong> \u2013 Llama\u202fGuard, OpenAI policy templates, regex and tree\u2011sitter tricks for fast rule checks.<\/li>\n\n\n\n<li><strong>System &amp; Tooling<\/strong> \u2013 Safety layers that run <em>after<\/em> draft generation (re\u2011rankers) vs <em>during<\/em> (constrained decoding, refusal tokens).<\/li>\n\n\n\n<li><strong>Red\u2011Teaming Playbooks<\/strong> \u2013 Manual adversarial prompts, automated mutation (JailbreakGym), and cross\u2011model ensemble attacks.<\/li>\n\n\n\n<li><strong>Eval Suites<\/strong> \u2013 RealToxicityPrompts, HARM\u2011bench, HELM safety subset; measuring bias, toxicity, jailbreak rate.<\/li>\n\n\n\n<li><strong>Content Policies<\/strong> \u2013 How to encode \u201callow \/ safe\u2011complete \/ refuse\u201d tiers and avoid loopholes.<\/li>\n\n\n\n<li><strong>Incident Response<\/strong> \u2013 Logging, canary prompts, and rollback plans when models misbehave in production.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">8. Technical Report CliffsNotes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Gemini<\/strong> \u2013 Multimodal routing and that infamous planner\u2011solver split.<\/li>\n\n\n\n<li><strong>Llama&nbsp;2\/3 &amp; Llama Guard<\/strong> \u2013 Safety alignment prompts and where RLHF still bites.<\/li>\n\n\n\n<li><strong>DeepSeek\u2011R1<\/strong> \u2013 Retrieval\u2011augmented pre\u2011training that treats <em>documents<\/em> as first\u2011class citizens.<\/li>\n\n\n\n<li>Plus quick hits on Falcon, Mistral, Phi\u20113 and other crowd favourites.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Final Words<\/h3>\n\n\n\n<p>Whether you\u2019re shipping models to prod, writing your first Transformer from scratch, or preparing before an onsite, I hope this bundle saves you a few hours of searching\u2014and maybe sparks your next idea.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u201cI never start a project without skimming my own notes first.\u201d\u2014 Every engineer the night before a deadline I&#8217;ve been writing down my AI\/LLM notes in last 6 months\u2014what started as a few scrappy records about Transformer quirks has turned into a 75 pages of tips that spans everything from model tweaks to prod inference &#8230; <a title=\"My Long-Running Notebook of AI &amp; LLM\" class=\"read-more\" href=\"https:\/\/firooz.us\/blog\/index.php\/2025\/07\/06\/my-long-running-notebook-of-ai-llm\/\" aria-label=\"More on My Long-Running Notebook of AI &amp; LLM\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[7,16],"class_list":["post-74","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-ai-2","tag-technical-notes"],"_links":{"self":[{"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/posts\/74","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=74"}],"version-history":[{"count":2,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/posts\/74\/revisions"}],"predecessor-version":[{"id":78,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/posts\/74\/revisions\/78"}],"wp:attachment":[{"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=74"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=74"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/firooz.us\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=74"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}