Jennie (Jing) Tu

⚡ Llama 3.3 70B from Meta is accelerated by TensorRT-LLM. 🌟 State-of-the-art model on par with Llama 3.1 405B for reasoning, math, instruction following and tool use. 🦙 Explore the preview ➡️

1

📢 DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads🚀 In this work, they tackle the significant memory and computational challenges associated with deploying long-context large language models (LLMs). The proposed framework, DuoAttention, introduces a novel distinction between Retrieval Heads and Streaming Heads to optimize the use of the KV cache, reducing memory usage by up to 2.55× and improving decoding speeds by up to 2.18×, all while maintaining high accuracy. Technical Highlights: 1. Retrieval Heads vs. Streaming Heads: DuoAttention divides attention heads into two categories. Retrieval Heads are crucial for processing long contexts, whereas Streaming Heads focus on recent tokens, allowing for a more efficient use of memory resources. This division helps in reducing the computational load without compromising model accuracy. 2. KV Cache Optimizatio: The framework uses selective caching strategies, applying full KV cache to Retrieval Heads while maintaining a constant-length cache for Streaming Heads. This results in significant reductions in both memory usage and latency during long-context inference. 3. Scalable Long-Context Inference: With DuoAttention, models like Llama-3-8B can handle up to 3.3 million tokens on a single A100 GPU. This is made possible through careful optimization of memory allocation and the use of lightweight algorithms to identify the critical heads. 4. Performance Improvements: DuoAttention achieves up to 2.55× memory reduction for multi-head attention (MHA) models and up to 1.67× for grouped-query attention (GQA) models. It also speeds up decoding by up to 2.18× for MHA models and up to 1.50× for GQA models, providing significant benefits for real-world applications involving large language models. 5. Compatibility with Quantization: The method is fully compatible with quantization techniques, such as 8-bit weight and 4-bit KV cache quantization. This compatibility further boosts efficiency, enabling the model to handle larger contexts with reduced hardware requirements. With DuoAttention, the team has achieved long-context capabilities for models like Llama-3-8B, enabling handling of up to 3.3 million tokens on a single A100 GPU. This advancement marks a significant step towards more efficient, scalable LLM deployments! 🔍 Paper: https://lnkd.in/dDFHimSA #LLMs #MachineLearning # #EfficientInference

13

I want to give a shoutout to Alex Xu, the author of "System Design Interview: An Insider's Guide" Volume 1, and also to Alex Xu and Sahn Lam for Volume 2. These books have truly revolutionized my comprehension of complex system designs, not just for interviews but also in general. Volume 1 completely blew my mind. It took my understanding of different architectural concepts to a whole new level. It covered everything from scalability to distributed systems in a comprehensive yet approachable manner. I found myself constantly revisiting it, gaining fresh insights and refining my understanding every time. Volume 2 just arrived and I'm so excited! I'm ready to study more complex subjects and obtain even deeper understanding. These books are genuinely exceptional since they offer priceless resources for understanding system design principles in addition to helping you do very well in interviews. They are incredibly helpful for everyone, regardless of expertise level; even beginners can benefit from reading them. I highly recommend them! #systemdesign #bytebytego #alexxu #sahnlam #learning #careergrowth #techcommunity #grateful #techbooks #learning #tech #computerscience

28

🎯 How teams are cutting Machine Learning Engineer hiring time from 90+ days to 28 days: 1. Role-specific technical assessments that mirror real projects • Real-world problem-solving instead of whiteboard exercises • Actual code reviews from your codebase • System design challenges based on your architecture • Practical ML model optimization tasks 2. Direct access to tech leads in first interview • No HR screening delays • Technical discussion from day one • Immediate architecture alignment • Quick technical fit assessment 3. 48-hour feedback commitment from all parties • Same-day interview feedback • Next-steps within 24 hours • Offer decisions in 48 hours • Clear communication timeline 4. Streamlined technical evaluation process • One comprehensive technical round • Parallel evaluations when possible • Combined culture/technical assessments • Quick technical team consensus Teams implementing even one of these cut hiring time by 40%. "Especially effective for fast-growing startups and scale-ups" #TechHiring #MLEngineering #TechRecruitment

3

If you're diving into system design, I've got some golden recommendations for you! 🌟 For High-Level Design (HLD), forget endless video binges and grab these two essential books: 1️⃣ System Design by Alex Xu: 🏗️ Fundamental principles of system architecture 📈 Scalability strategies for growing systems 💻 Best practices for designing distributed systems 🔄 Techniques for handling fault tolerance and reliability 2️⃣ Designing Data-Intensive Applications by Martin: 📊 Deep dives into data storage and retrieval systems 🔒 Ensuring data consistency and durability in distributed environments 🌐 Understanding the trade-offs in distributed system design 🛠️ Implementing robust and efficient data processing pipelines Looking to master Low-Level Design (LLD)? Head straight to this GitHub repository: 🔗 https://lnkd.in/gBSFKrGb by Ashish Pratap Singh Trust me, these resources are enough to understanding of both HLD and LLD! 💡🚀 #SystemDesign #SoftwareEngineering #LearningResources

322

7 Comments

Unlock the full potential of AI with PyTorch at Meta! 🔥 From training to inference, see how we’re optimizing our AI workflows for the future. #AI #PyTorch #MetaAI

116

7 Comments

Explore the trade-offs between throughput and latency in #deeplearning models with this new self-paced course. Learn how to measure and optimize performance for on-prem and #cloud deployments. This course is free for a limited time. Enroll now: https://nvda.ws/3O5ui2R #NVIDIATraining

1

Great list of resources if you are preparing for software engineering interviews.

8

🦙 The new Llama 3.1 models with the new NVIDIA NeMo Retriever NIMs create a powerful #generativeai duo. ➡️ Technical deep dive on how to build an agentic RAG pipeline by the newest NeMo Retriever embedding and reraking NIM microservices. https://bit.ly/3WlUSbM

1

Learn why the NVIDIA Jetson AGX Orin is the leading platform for running transformer models like GPT-J, vision transformers, and stable diffusion at the edge.

1

👀 Just released - 4 new classifier models for curating high-quality data with NVIDIA NeMo Curator ✨ Read blog ➡️ bit.ly/402PWM1 With these classifier models, you can categorize your data into predefined groups or classes, annotate it with metadata, and optimize your #dataprocessing pipelines. #LLM

1

🔓 Unlock the power of your data with NVIDIA NeMo Curator. Discover how to #customize and optimize your model by integrating high-quality, domain-specific datasets. https://nvda.ws/3WO1QYZ

1