Junjie Shan

🌟 Boost Meta's Llama 3.3 70B inference throughput by 3x with Tensor RT-LLM optimizations including speculative decoding, in-flight batching, KV caching, custom FP8 quantization, and more.🦙 Technical deep dive ➡️

8

Kudos to #Composable_kernel team and please check out the AMD engineer Haocong Wang presents the ROCm Composable Kernel library which allows to write high-performance kernels for machine learning with HIP C++. #Composable_kernel #ROCm

39

1 Comment

Conquering the CUDA Out of Memory Error: A Guide to PyTorch's GPU Memory Management Struggling with the "CUDA out of memory" error in your PyTorch projects? 🔵 Reduce Model Size: Opt for smaller, optimized models or explore alternative frameworks. 🔵 Adjust Batch Size: Experiment with smaller batches to reduce memory consumption. 🔵 Optimize Data Loading: Ensure efficient data pipelines to minimize memory overhead. 🔵 Leverage Profiling Tools: Use nvidia-smi and PyTorch's memory profiler to identify bottlenecks. 🔵 Explore Memory Allocation Strategies: Experiment with PYTORCH_CUDA_ALLOC_CONF and memory pooling techniques. 🔵 Consider Cloud-Based GPUs: If local resources are constrained, explore cloud-based options. By implementing these strategies and understanding PyTorch's memory management, you can effectively overcome the "CUDA out of memory" error and optimize your GPU utilization.

1

1 Comment

Stanford Ph.D. researcher Cheng Chi discusses the development of the UMI gripper and the use of diffusion AI models for robotics. The post Stanford researcher discusses UMI gripper and diffusion AI models appeared first on The Robot Report. https://lnkd.in/gbTiRwh4

1

I will be attending IJCAI 2024 in Jeju, South Korea, this week! I’ll be presenting our paper, “Temporal Inductive Logic Reasoning over Hypergraphs,” a collaborative effort with our partners at Georgia Tech, led by Yuan Yang. In this work, we introduce Temporal Inductive Logic Reasoning (TILR), an inductive logic programming (ILP) method designed for temporal hypergraphs. To facilitate hypergraph reasoning, we developed the multi-start random B-walk, a novel graph traversal technique for hypergraphs. By integrating this with a path-consistency algorithm, TILR effectively learns logic rules by generalizing from both temporal and relational data. I look forward to reconnecting with my collaborators and meeting new colleagues. Let's catch up and share our latest research insights! Looking forward to connecting with fellow researchers and exploring the latest advancements in AI! #IJCAI2024 #Cisco #CiscoResearch #CiscoOutshift #GraphReasoning #AI

26

2 Comments

Introduction to Implementing Numerical Deep Learning Models in MATLAB and Models for Signal Anomaly Detection and Audio Tasks https://lnkd.in/gKewcat4

3

Go check out Brevitas at the links below and start experimenting with MX and all the latest quantization algorithms!

11

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

🎉 I am thrilled to share that my first paper "Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images" has been accepted to #NeurIPS2024! 🔍TL;DR: MHCDIFF reconstructs pixel-aligned 3D human shapes in a robust manner to occlusion in images. We present: 1️⃣ We introduce a novel multi-hypotheses conditioning mechanism that effectively captures the distribution of multiple plausible SMPL meshes. 2️⃣ We adopt point cloud diffusion model to capture the global consistent features and inpaint the invisible parts. 🔗 Project: https://lnkd.in/gkFRHADa 📚 Paper: https://lnkd.in/gvP4VKAq 💻 Code: https://lnkd.in/gD5rnD2H Huge thanks to my advisor Tae-Kyun Kim! 🇨🇦 Looking forward to seeing everyone at NeurIPS 2024! #NeurIPS2024 #NeurIPS #3DHuman #Diffusion

33

1 Comment

Excited to share my latest research paper preprint on arXiv. 🎮🖥️ Adrenaline: Adaptive Rendering Optimization System for Scalable Cloud Gaming (https://lnkd.in/e8RwSPKD) In this work, we tackle the scalability challenge of cloud gaming services at the edge cloud, introducing Adrenaline - a new system that optimizes game rendering quality by considering both user-side service quality and server-side cost. Key highlights: • Cloud gaming at the edge cloud provides low-latency connections, but fixed compute capacity poses challenges for multi-user serving. • High-quality game frames generated on the server may be ineffective due to lossy compression during transmission by a user's network condition. • Adrenaline adaptively optimizes content generation costs on the server while considering user-side quality, reducing resource waste. • Our network-condition-aware optimization achieves up to 24% higher service quality and serves 2x more users with the same resource footprint. • Easy integration with modern game engines and existing cloud gaming service architectures Check out more details with the full paper! #CloudGaming #EdgeComputing #ResearchPaper

35

2 Comments

Great explanation and visualization, thanks for sharing Alex Razvant! I would add that other graph compilers have a similar flow, but the differentiation is in the implementation details. For modern AI architectures, graph compiler optimizations are not enough to achieve high performance and utilization, and many AI compilers have dedicated implementations. FlashAttention (kernel-fusion) and PagedAttention (memory management) are examples of dedicated optimization techniques that are not achieved by generic compilation (not to mention Speculative decoding and other algorithmic "tricks"). By the way, this is the main reason in my opinion for why we see so many competing inference engines for LLMs (TensorRT, vLLM, TGI, LMDeploy, ONNX Runtime, SGLang, CTranslate2, llama.cpp, MLC-LLM, DeepSpeed, IPEX-LLM, etc.) 😅

19

1 Comment

I've mentioned this to #investors outside of sports and entertainment. The adoption curve for technology in sports is a hockey stick. Years of pilots in fast paced, high stress environments, but if the application is scoped out properly with the right forces behind it can lead to BIG wins in terms of adoption. I don't know much about AiFi Inc. but this looks to be an awesome example on the adoption for checkout free applications. Well done. For #roboticsINsports check out Hop Robotics

18

1 Comment

Have you ever wondered how live camera feeds achieve seamless synchronization? As a Research Assistant at the Indian Institute of Technology, Kanpur, I tackled this challenge using GStreamer. This robust multimedia framework also serves as a backend for NVIDIA Deepstream. From filters and codecs to encoders and decoders, GStreamer proved indispensable in creating robust real-time video pipelines. In my latest Medium blog (part 3), I dive deeper into GStreamer’s functionalities and how it simplifies complex multimedia processing. If you're curious about leveraging GStreamer for your projects, head to my Medium post for all the technical details. 🔑 Key Takeaways: Understanding how filters (encoders, decoders, muxers, demuxers) process data Leveraging queues and tees for asynchronous processing and stream branching Managing data with buffers and understanding GStreamer's event states This post will help you master the essentials of GStreamer and optimize your media workflows! 🎬💡 Check out the full post for more insights #GStreamer #C++ #MediaProcessing #Coding #Tech #SoftwareDevelopment #OpenSource

21

It’s been a pleasure to talk alongside such great researchers from Google DeepMind and top Taiwanese universities at Google AI Taiwan Academic Forum. (Details at https://lnkd.in/gnbcWZyf) My topic — “Challenges of Applied ML” — covered 4 observations: - We ought to build product, not merely expose LLMs. - Making them useful and trustworthy requires handling input and output properly. (Read more at https://lnkd.in/gcNDC54R) - Making those model useful requires high-quality data for tuning (and evaluation). (More at https://lnkd.in/gyVYQaaD) - There’s many practical deployment considerations, one of the main ones being cost.

110

🚀 Throwback to my Compiler Project! 🚀 Recently, I revisited a compiler project (https://lnkd.in/gcip5UrJ) that I built during my coursework, and it has been an exciting experience diving back into it. 💻 The project involved developing a recursive-descent parser, semantic checker, and code generator for a simplified C-like language. This journey was filled with challenges, like constructing abstract syntax trees (ASTs), managing types, and generating Intel x86-64 assembly code—all from scratch! 💡 But the real fun came when I encountered a bug that had eluded me before. The compiler was generating incorrect assembly instructions, specifically using the wrong register size. After some careful debugging and a deep dive into the code generation logic, I finally fixed the issue by ensuring the proper use of 32-bit registers instead of 64-bit ones in certain operations. 🎯 Here's what I learned: Sometimes stepping away and coming back to a project with fresh eyes can lead to new insights. The satisfaction of squashing bugs that have been lingering for a while is unbeatable! Compiler design is tough but incredibly rewarding when everything finally works as expected. If you're into compilers, assembly, or just love tackling tough coding challenges, feel free to reach out! Always happy to discuss. 😊 #coding #compilers #bugfixing #projects #assemblylanguage #cplusplus #debugging #programming #engineering

6

👏CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks

25

The parameters of compiler optimizations interact in complex and non-trivial ways, and finding optimal combinations remains an open research challenge. There are several approaches to discovering effective combinations of these optimization parameters. Recently, Michael Canesche (Cadence Design Systems) and Gaurav Verma (Stony Brook University) have had promising results using auto-tuning with coordinate descent. To visualize this process, imagine that each optimization represents a 'dimension' in a search space. The combination of all optimizations and their respective parameters forms a vector in this space. Now, consider that each point in this space is mapped to a real number representing the performance of a kernel (assuming a fixed input). Coordinate descent is a search technique that assumes the performance curve is convex. Based on this assumption, it aims to find the optimal point by varying one optimization parameter at a time. Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent: https://lnkd.in/dgxxfdYR The Droplet Search Algorithm for Kernel Scheduling: https://lnkd.in/dBfQid9z

76

Check out our new blog post to see how NeMo-Aligner is leveraging knowledge distillation for data-efficient supervised fine-tuning! https://lnkd.in/gnSiywcP

22

2 Comments