Hongkun Ge

Conquering the CUDA Out of Memory Error: A Guide to PyTorch's GPU Memory Management Struggling with the "CUDA out of memory" error in your PyTorch projects? 🔵 Reduce Model Size: Opt for smaller, optimized models or explore alternative frameworks. 🔵 Adjust Batch Size: Experiment with smaller batches to reduce memory consumption. 🔵 Optimize Data Loading: Ensure efficient data pipelines to minimize memory overhead. 🔵 Leverage Profiling Tools: Use nvidia-smi and PyTorch's memory profiler to identify bottlenecks. 🔵 Explore Memory Allocation Strategies: Experiment with PYTORCH_CUDA_ALLOC_CONF and memory pooling techniques. 🔵 Consider Cloud-Based GPUs: If local resources are constrained, explore cloud-based options. By implementing these strategies and understanding PyTorch's memory management, you can effectively overcome the "CUDA out of memory" error and optimize your GPU utilization.

1

1 Comment

Kudos to #Composable_kernel team and please check out the AMD engineer Haocong Wang presents the ROCm Composable Kernel library which allows to write high-performance kernels for machine learning with HIP C++. #Composable_kernel #ROCm

39

1 Comment

🌟 Boost Meta's Llama 3.3 70B inference throughput by 3x with Tensor RT-LLM optimizations including speculative decoding, in-flight batching, KV caching, custom FP8 quantization, and more.🦙 Technical deep dive ➡️

8

Great explanation and visualization, thanks for sharing Alex Razvant! I would add that other graph compilers have a similar flow, but the differentiation is in the implementation details. For modern AI architectures, graph compiler optimizations are not enough to achieve high performance and utilization, and many AI compilers have dedicated implementations. FlashAttention (kernel-fusion) and PagedAttention (memory management) are examples of dedicated optimization techniques that are not achieved by generic compilation (not to mention Speculative decoding and other algorithmic "tricks"). By the way, this is the main reason in my opinion for why we see so many competing inference engines for LLMs (TensorRT, vLLM, TGI, LMDeploy, ONNX Runtime, SGLang, CTranslate2, llama.cpp, MLC-LLM, DeepSpeed, IPEX-LLM, etc.) 😅

19

1 Comment

Sharing my thoughts on How Waymo (formerly known as the Google Self-Driving Car Project) is Pioneering Autonomous Driving with AI. Waymo has become a trailblazer in self-driving technology, and at the core of it all is AI. Leveraging advanced machine learning and deep neural networks, Waymo’s AI-powered system continuously improves through real-world data, making autonomous vehicles smarter, safer, and more reliable. Perception: AI allows Waymo’s cars to "see" and understand their surroundings by processing data from sensors like cameras, LIDAR, and radar. This helps detect pedestrians, other vehicles, and objects in real time. Prediction: The AI system predicts the behavior of everyone around the vehicle—whether it’s cars changing lanes or pedestrians crossing the street—ensuring smooth and safe driving decisions. Planning & Control: AI makes split-second decisions about accelerating, braking, and steering based on the analysis of complex traffic patterns, enabling Waymo vehicles to safely navigate roads. With millions of autonomous miles driven, Waymo’s AI is constantly learning and improving, helping create the future of safer and smarter transportation. The road ahead is AI-driven! www.rxsmeta.com #AI #AutonomousVehicles #Waymo #SelfDrivingCars #MachineLearning #SmartMobility #TransportationTech #FutureOfTransport #AIinAction #Innovation

2

In this episode, we discuss DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning by Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar. DigiRL is an innovative autonomous reinforcement learning approach designed to train device control agents by refining pre-trained vision language models through a two-stage process involving offline RL and offline-to-online RL. It addresses traditional VLM limitations by introducing enhanced advantage estimators and an automatic curriculum to optimize learning in a scalable Android environment. Experiments on the Android-in-the-Wild dataset showed that DigiRL significantly outperformed existing methods, setting a new standard in device control tasks.

4

At Voice of Reason AI, where we focus on interfaceless personal aids for individuals with memory and brain disorders, inference speed is not just a luxury—it’s a necessity. Our AI may remain silent most of the time, yet it must continuously listen, understand, and analyze context so it can respond instantly when help is needed. Think of it like the cymbal player in an orchestra, silent for long stretches but ready to deliver a perfectly timed crash. For us, it's essential that our AI “cymbal” sounds precisely when a user needs guidance. This makes Cerebras' advancements incredibly exciting, as they could enable us to create real-time, ambient aids that seamlessly enhance daily life without unnecessary delays. Such speed is crucial for AI that doesn’t merely react but anticipates needs in real-time, helping individuals maintain independence and dignity.

2

How do AI software engineering agents work? No better place to start than "at the source:" the state-of-the-art open source AI coding agent: SWE-bench has quietly become the industry standard AI coding agent benchmark. The team behind SWE-bench built SWE-agent (at release, state-of-the-art open source AI agent) and with Ofir Press (a member of the team building SWE-bench and SWE-agent), we dive deep on how it works: Read the deepdive here: https://lnkd.in/e3Ha39Rz

36

Get a step-by-step on how #NVIDIANIM helps deploy and scale swarms of fine-tuned LoRA adapters to handle mixed-batch inference requests. Learn more about our strategic approach > https://nvda.ws/3VuOJL5 #LLM #benchmarks

14

The future is here

5

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

One of the best things about being in San Francisco 🌉 is the ability to drop in on great talks. Last week I had a chance to see NVIDIA's CEO Jensen Huang (JH) being interviewed by Stripe's CEO Patrick Collison. Jensen is among the longest-serving founder-CEOs in Silicon Valley at the moment (30+ years). With that comes a special clarity of mind and some hot takes. 🌶👇 🗣 Jensen doesn't believe in 'praise in public and criticism in private'. Instead, he reasons that most information in the organization should not be privy to certain people. By giving feedback in public, you allow everyone to learn. 'What is better than learning from your own mistakes? It's learning from someone else's mistakes!!' 🔥 JH also doesn't like to fire people. He would rather "torture them into greatness". He invests in people and he doesn't like to write off his investments. 👨‍💻 Asked whether he likes his job every day, he answers "no". The perfect job doesn't give you happiness 100% of the time. "No great things were done, that were easy all the way." ... "Even if I come home from a day where I did things I didn't enjoy, it could still be have been a great day." 👩‍👧‍👦 Jensen famously has 60 direct reports. "This allowed us to cut out probably 7 levels of management". His subordinates don't necessarily have such large teams. "It doesn't scale down, less senior people need more guidance". Looking at the numbers, this approach allowed NVidia to reach ~2T market cap with ~28.000 people vs. Microsoft's 3T with ~220.000 people and Apple's 2.6T with 160.000 people. ❔ Jensen answers almost every question in the same way: "Let's break it down, let's reason about it". This is a great response on so many levels. First, it allows him to not be put on the spot. Second, it allows him to teach his workforce how he reasons through a problem so that they can emulate it and create alignment. Third, it gives people a chance to point out flaws in his reasoning. He admits he is not right all the time. ON AI 👨‍🔧 "You will not lose your job to AI, your company will go out of business because another company used AI." "If you're not using AI you're doing it wrong." ⏱ The extent to which AI will enter many processes is largely underestimated. Right now we are focused on language - sounds in a time sequence. "Any time sequence can be tokenized." This will include robotic movement. 🥛 JH also expects that we won't be satisfied with foundation models "You're going to want to tune it into perfection because you care so deeply. Say if it's already at 99% you'll want to get it to say 99.3%" This last piece is also why we created Query Vary (YC W22). AI will never feel truly intelligent until it completely understands your data and context. It needs to be taught, regardless of how powerful the foundation model is. Finally, besides insights, it was remarkable how funny Jensen was, cracking a joke at roughly a 1 per 3 min rate. It was a very entertaining talk. #ai #sf

61

8 Comments

Excited to announce the public release of the Sortformer model! 🎉 This innovative approach to speaker diarization addresses the limitations of traditional methods, offering seamless integration with ASR and LLMs, efficient token-level training, and a sort-loss mechanism to resolve the speaker permutation problem. No more timestamp-based training for speaker diarization!  Lightweight and highly adaptable, Sortformer simplifies diarization integration into neural architectures 🚀 🎧 Check out this 10-minute NotebookLM podcast explaining Sortformer in detail: https://lnkd.in/gX8S6fPA Paper: https://lnkd.in/gCvPg45C Model weights: https://lnkd.in/ga9SQX6P Code: https://lnkd.in/gB5J2r37 Tutorial: https://lnkd.in/gaH4zhex

50

2 Comments

In this episode, we discuss LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding by Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, Vikas Chandra. LongVU presents a spatiotemporal adaptive compression method for processing long videos using Multimodal Large Language Models, efficiently reducing redundancy while preserving important visual information. It employs techniques like cross-modal queries, DINOv2 features, and token reduction to manage spatial and temporal information. This approach shows superior performance on video understanding benchmarks, handling lengthy videos effectively and demonstrating scalability even in smaller models.

3

How do you build a mixed reality headset from scratch? Alfred J., VP of Hardware Engineering at Reality Labs, recently joined the Meta Tech podcast to take us through the full process of bringing a Quest device from sketch to scale. Listen to episode 68 now! 🎧 bit.ly/3ZUhsKd #LifeAtMeta #engineering

2

👏CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks

25