Ran Chen

A very useful checklist to follow when you have to put in production Transformer based models

5

"Our topline metrics for llama 3 are - 97% speedup for Llama 3 8B inference using autoquant with int4 weight only quantization and hqq - 73% peak VRAM reduction for Llama 3.1 8B inference at 128K context length with a quantized KV cache - 50% speedup for Llama 3 70B pretraining using float8 training on H100 - 30% peak VRAM reduction for Llama 3 8B using 4 bit quantized optimizers. " https://lnkd.in/d8W4PUG5

1

🚨 New paper w/ interesting insights for fine-tuning and GPU poor folks 🚨 🚀 Exploring Fine-Tuning for LLMs: Full Fine-Tuning vs. #LoRA - Are They Truly Equivalent? 🤔 As LLMs grow, fine-tuning them for specific tasks becomes challenging due to high computational costs. Recently, Low-Rank Adaptation (LoRA) has become a popular alternative, promising similar performance with fewer parameters. But are LoRA-tuned models the same as fully fine-tuned ones? MIT’s latest study dives deep into this question, revealing some surprising differences! 🔍 📝Key Takeaways: 1. Spectral Differences 🧬: LoRA introduces “intruder dimensions” not found in fully fine-tuned models. These alter the model’s weight structure, suggesting that LoRA and full fine-tuning leverage different parts of the parameter space, even if they hit similar accuracy on specific tasks. 2. Generalization & Robustness 🌐: Models fine-tuned with LoRA may struggle with new or sequential tasks, forgetting previously learned knowledge more easily. Full fine-tuning, however, shows better retention and robustness in shifting environments. 3. Effective Rank & Optimal Performance ⚙️: LoRA’s rank matters! Mid-range ranks (like r=64) perform well on tasks and retain pre-trained knowledge, while lower or higher ranks can lead to greater forgetting. Smart parameter tuning (like setting α=2r) can help LoRA resemble full fine-tuning more closely. 4. Why It Matters 💡: LoRA offers incredible parameter efficiency, but its unique trade-offs might affect its use in scenarios needing high adaptability and long-term memory. With adjustments, LoRA can bridge some of these gaps, but full fine-tuning remains the go-to for complex or multi-tasking demands.

25

1 Comment

- SwiftKV is a novel model transformation and distillation procedure (by folks at Snowflake) designed to optimize large language model (LLM) inference. Very cool findings in the paper (as presented in the vLLM meetup on 11/13). - SingleInputKV: Prefills later layers' KV cache using earlier layer outputs, reducing computation on prompt tokens. The key point is that the first few layers of attention in LLMs are similar (nice find), allowing SwiftKV's SingleInputKV mechanism to skip computations in later layers during inference. By using a single earlier layer's output to compute the KV cache for subsequent layers, it significantly reduces both computation (by 50%) and memory usage (by 62.5%) without sacrificing model performance. - AcrossKV: Merges KV caches of neighboring layers to minimize memory usage and increase batch size for higher throughput. - For models like Llama-3.1-8B and 70B, SwiftKV reduces prefill compute by 50%, memory usage by 62.5%, and achieves up to 2× higher throughput and 60% lower time per token. https://lnkd.in/guhEBW5G

62

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

180

2 Comments

Quentin H. and Yanwei (Wayne) Zhang on the Faire Data team recently did some great work fine-tuning Llama3 model to enhance search relevance. Here are some key insights from the work: • While prompt engineering is valuable, LLM fine-tuning is essential for a domain specific task like understanding semantic search relevance. • Fine-tuned open-source LLMs like Llama3 can deliver impressive performance, scalability, and cost-efficiency compared to using proprietary models. • Effective application of a fine-tuned LLM demands a well-defined problem, robust, high-quality labeled data, and ongoing model refinement. Ground-truth labeled data is crucial for maximizing model performance. • Fine-tuned Llama3-8b achieves performance on par with the larger Llama2–13b model. Read the blog post to dive deeper into these findings! https://lnkd.in/ecP_3E7S We are also hiring (https://lnkd.in/dHfRZcPi) #LLM #AI #Finetuning #LLAMA3 #SearchRelevance #Hiring

92

3 Comments

Looks like fine tuning out performs RAG and Large context LLMs for data accuracy with longer contexts Paper: https://lnkd.in/gj9-3jsK Summary of "BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack" - Introduction to BABILong Benchmark: - Designed to test LLMs' reasoning abilities over extremely long documents. - Comprises 20 diverse reasoning tasks, such as fact chaining, induction, deduction, counting, and handling lists/sets. - Uses natural text from the PG19 corpus, extendable to document lengths up to 1 million tokens. - Current LLM Performance: - Popular LLMs utilize only 10-20% of the context effectively. - Performance declines with increased reasoning complexity. - Retrieval-Augmented Generation (RAG) methods achieve 60% accuracy on single-fact QA, independent of context length. - Top Performing Models: - Recurrent Memory Transformers (RMT) show the best performance, processing up to 11 million tokens. - Small models like Mamba (130M) and fine-tuned RMT (137M) achieve high accuracy on long-context tasks. - Evaluation Findings: - Current benchmarks (e.g., Longbench, L-Eval) are insufficient for LLMs with capabilities beyond 40,000 tokens. - BABILong tests models' efficiency in distinguishing relevant facts from irrelevant text over long contexts. - Popular LLMs fail to maintain high performance as context length and complexity increase. - Fine-Tuning Insights: - Fine-tuning improves model performance significantly. - Mamba and RMT demonstrate successful QA with long contexts, outperforming RAG models. - RMT's memory mechanism allows efficient processing of long sequences, with marginal quality degradation up to 11 million tokens. - Benchmark Significance: - BABILong provides a rigorous evaluation framework for LLMs' long-context reasoning abilities. - Highlights the limitations of current LLMs and the need for improved context processing mechanisms. - Key Contributions: - Introduction of BABILong, a scalable generative multi-task benchmark. - Evaluation of over 20 recent LLMs on various context lengths and tasks. - Demonstration of the limitations of current models in long-context utilization. - Setting a new record for sequence size processed by a single model with RMT.

4

5 Comments

[research update] use a Transformer's multi-head attention activations to fetch RAG style, to augment multi faceted problem breakdown for LLMs in ."Multi-Head RAG: Solving Multi-Aspect Problems with LLMs": 1. Enhances LLMs by integrating document retrieval (RAG) into the LLM context. 2. Challenge: Existing RAG solutions struggle with queries needing multiple documents with varied contents due to distant embeddings. 3. Multi-Head RAG (MRAG): Introduces a novel approach using Transformer's multi-head attention activations to fetch diverse documents. 4. Activation Utilization: Different attention heads capture varied data aspects, improving complex query retrieval. 5. Improved Relevance: MRAG shows up to 20% improvement in relevance over standard RAG baselines. 6. Evaluation Methodology: Provides metrics, synthetic datasets, and real-world use cases to demonstrate effectiveness. 7. Seamless Integration: MRAG can be integrated with existing RAG frameworks and benchmarking tools like RAGAS. 8. Multi-Aspect Problem Solving: Designed specifically to handle multi-aspect queries more effectively. 9. Data Stores Compatibility: Compatible with various data store classes. 10. Potential Applications: Benefits scenarios requiring diverse information retrieval, enhancing LLM responses. Experiment The paper introduces MRAG and evaluates it using synthetic datasets and real-world data. It compares MRAG's performance against standard RAG models. The experiment focuses on the model's ability to retrieve relevant documents across various query aspects, demonstrating up to a 20% improvement in relevance over baselines. Use Case MRAG is tested on real-world scenarios like customer support, where queries often require information from multiple documents with diverse contents. The model's capability to retrieve and integrate multi-aspect data enhances the quality of responses.

81

5 Comments

Hallucination is Inevitable in LLMs An important new paper out that shows that even with perfect data and an idealized architecture “LLMs will always hallucinate”. I suspect that this will over time be linked to Gödel’s Incompleteness Theorem [1] and Turing’s Halting Problem [2]. Every technology has trade-offs. We love LLMs but they need to be used in the right way. [1] Good pop science depiction in Oppenheimer (Gödel was Einstein’s walking companion) or also see Rebecca Goldstein’s book Incompleteness [2] Turing is depicted in The Imitation Game and also the progenitor of the famed Turing Test.

36

Can LLM preference tuning degrade Multi-Agent system performance? In a recent tweet, Andrej Karpathy questioned why all LLMs sound similar, prompting Sebastian Rascha to speculate about the uniformity of alignment datasets. Aligning user preferences is a last-mile step in training large language models. Regardless of the pre-training datasets used, could the preference datasets used during alignment compromise the effectiveness of LLMs? This issue could adversely affect multi-agent systems incorporating multiple LLMs in their design.

7

1 Comment

Text embeddings have become crucial for transforming textual data into dense vector representations (with context), enabling efficient processing across various modalities like text, images, and audio. This article provides a good summary on some of these (not all) embeddings available for text https://lnkd.in/gKp4ta9a 1. Gensim Word Embeddings (Free) Why Use It: Specializes in NLP with open-source models like Word2Vec and FastText, promoting transparency and ease of custom training. 2. Facebook Embeddings (Free) Why Use It: Trained on extensive corpora and supports over 100 languages, making it versatile for global applications. 3. AllenNLP Embeddings (Free) Why Use It: Provides NLP-focused embeddings like BERT and ELMo, with tools for fine-tuning and visualization. 4. OpenAI Embeddings (Paid) Why Use It: Comprehensive training on massive datasets allows for effective semantic understanding, making it suitable for advanced NLP tasks. Ideal for tasks requiring zero-shot learning. 5. MultiLingual BERT (Free) Why Use It: Handles multilingual datasets effectively, offering robust support for multiple languages. Perfect for applications requiring consistent performance across different languages, such as global customer support systems. 6. DistilBERT Base Uncased (Free) Why Use It: Smaller and faster than BERT, maintaining near-comparable performance with reduced resource usage. 7. MPNet V2 (Free) Why Use It: Excels in text similarity tasks with a Siamese architecture, providing accurate similarity measurements. 8. HuggingFace Embeddings (Free/Paid) Why Use It: Offers versatility with a wide range of embeddings (text, image, audio) and the ability to fine-tune on custom data. HF also provides wrappers to almost all open sourced models. 9. Longformer Base 4096 (Free) Why Use It: Supports long texts up to 4096 tokens, making it unique in handling lengthy documents without truncation. 10. RoBERTa (2022) (Free) Why Use It: Offers updated text representations with comprehensive training on recent data. 11. GTE-Base (Free) Why Use It: Offers a good balance of performance and resource requirements with a 768-dimensional embedding. 12. GTE-Large (Free) Why Use It: Provides high-quality embeddings with more detailed representations, useful for nuanced text analysis. Best for applications requiring in-depth text understanding, such as detailed content recommendations. 13. GTE-Small (Free) Why Use It: Optimized for speed with a smaller model size, making it efficient for applications needing quick processing. Useful for real-time systems where fast text similarity searches are critical. 14. E5-Small (Free) Why Use It: Combines compact size with good performance, making it suitable for fast, general-purpose text embedding tasks.

11

Foundational models, or large language models (LLMs), train at the scale of the entire internet to acquire the vocabulary, syntax, and narrative structure of human language. Fine-tuning involves using a smaller, domain-specific dataset to grant additional specificity to a pre-existing foundational model that has already acquired patterns and features from massive underlying training datasets. By utilizing the existing knowledge embedded in a pre-trained language model, we unlock high-performance language understanding, but fine-tuning helps to get the right answer - not just an answer that sounds right - with a minimum of hallucinations or incorrect info. In healthcare, out-of-the-box, foundational models are often too generic to sufficiently answer the precision of a healthcare query. Foundational language models are trained on a massive scale of data, which includes jokes, incorrect information, misunderstandings, and misinformation, along with the sum total of human knowledge. The problem is, if you ask a large language model to give you factually correct information when it has been trained on BOTH accurate and inaccurate information - you’re playing a bit of a dangerous game. Enter fine-tuning language models. In the old days, we used to build a single model for a single task - for example, classifying procedures into evidence-based or not as laid out by clinical practice guidelines, for patients with particular disease markers. Each disease would have its own model, which took a long time to build and didn’t scale particularly well, and we struggled with the sparsity and inherent high-dimensionality of healthcare data. By fine-tuning LLMs, however, we're able to specialize those foundational models to achieve greater domain specificity and specialization, leverage the unique predictive and planning capabilities of LLMs, and cut down on the incorrect information that such models inevitably produce, while still preserving the underlying language prowess of the LLM. If improved accuracy and specificity wasn't enough of a selling point, there's another notable upside to fine-tuning foundational models for use in a clinical context. In terms of computational efficiency, using LLMs out of the box on specific tasks can require massive compute (and spend) that amounts to computational overkill for the problem. Some LLMs charge per query, some by compute load or time - either way, as a business running LLMs in a production environment, minimizing the number of LLM calls means optimizing your compute expenditures, too. When LLMs fail at scale in healthcare, it's usually along the fault line of failure to specify adequately. Fine-tuning large language models along domain-specific lines is one way to effectively productize genAI for healthcare use cases. #genAI #LLM #ai #healthcareAI #healthtech #largelanguagemodels #generativeAI #chatGPT

61

2 Comments

User embeddings are crucial for the downstream personalized models, but larger embedding models come with higher latency, making real-time inference challenging. This often forces a trade-off: Small embedding model for real-time updates. Or, large embedding model with offline batch updates. Meta's recent paper proposes a hybrid approach: combine stale embeddings from a feature store with real-time embeddings from a complicated user model (with a timeout). If real-time calculations exceed the timeout, use stale embeddings; otherwise, use both. Completed real-time embeddings are then flushed to the feature store, improving future requests. Paper link:

28

I just published Vol. 78 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Systematic Review of Language Model Representations, from Zhang et al. 📚 Adaptive RAG Through Knowledge Boundary Detection, from Alibaba 📚 Understanding Context Sufficiency in RAG Systems, from Joren et al. 📚 A Cache-Driven Approach to Scaling Recommendation Systems, from Kuaishou 📚 A General Framework for Content Annotation and Retrieval, from Clarke et al. 📚 A Comprehensive Framework for Multimodal Long Document Understanding, from Alibaba 📚 Specialized Text Embeddings for Financial Document Understanding, from Balyasny Asset Management 📚 A Theoretical Framework for Comparing Recommendation Loss Functions, from Teodoro et al. 📚 Understanding Document Retrieval Trade-offs in RAG Systems, from Intel Labs 📚 A Systematic Evaluation of Commercial LLM Fine-tuning Services, from Stanford University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

33

1 Comment

I just published Vol. 48 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Systematic Survey of Retrieval-Augmented Generation Techniques, from York University 📚 Navigating the Evaluation Landscape of Generative Information Retrieval, from Alaofi et al. 📚 Leveraging RAG to Enhance the Trustworthiness of Generative AI for Enterprise Workflow Automation, from ServiceNow 📚 Exploring the Interplay Between Large Language Model Priors and Externally Sourced Content in Retrieval-Augmented Systems, from Stanford University 📚 A Practical Roadmap for Efficient Fine-Tuning on Proprietary Enterprise Data, from HCLTech 📚 Leveraging Look-Alike Users and Recalled Items to Improve Click-Through Rate Prediction, from Huawei 📚 Leveraging Collaborative Knowledge in Large Language Models for Robust and Versatile Recommendation, from KAIST 📚 A Study on How LLM-Generated Content Infiltrates and Reshapes Retrieval-Augmented Generation Systems, from UCAS 📚 Leveraging Rotary Position Embeddings for Extended Context Windows in Text Embedding Models, from Microsoft 📚 Taxonomy and Analysis of Sensitive User Queries in Generative AI Search, from Naver #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

71

3 Comments

I just published Vol. 54 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Graph Retrieval-Augmented Generation for Context-Aware Reasoning, from Emory University 📚 Enabling Fine-Grained Ranking at Varying Granularities via Multi-Vector Embeddings, from Reddy et al. 📚 Learning to Ask Clarification Questions for Ambiguous Retrieval Queries, from UC Berkeley 📚 Principled and Practical Multi-Vector Retrieval with Approximation Guarantees, from Google Research 📚 Towards User-Centric Recommendations via LLM-Based Simulation and Tool Learning, from USTC 📚 Building Pseudo-Graph Databases from Raw Texts for Retrieval-Augmented Language Models, from Renmin University 📚 Retrieval-Augmented Early Exiting for Efficient Large Language Model Inference, from Huang et al. 📚 A State-of-the-Art Generalist Embedding Model, from NVIDIA 📚 Enhancing Multilingual Language Model Training via Repeat Ranking, from Devine et al. 📚 A Real-World Evaluation of HNSW for Vector Retrieval, from Marqo #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

46

2 Comments

I just published Vol. 55 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Review of State-of-the-Art Universal Text Embeddings, from Cao et al. 📚 A Survey of Generative Retrieval Techniques, from NTU 📚 An Approach to Structurally Aligning Visual and Textual Embeddings, from Alibaba 📚 Leveraging Dynamic Granularity for Improved Retrieval-Augmented Generation, from Zhong et al. 📚 A Framework for Consistent and Reliable LLM-based Passage Ranking, from OSU 📚 Boosting Sample Efficiency in Recommender Systems through Large Language Model Integration, from Huawei 📚 A Task-Specific Loss Function for Improved Multi-Task Learning Performance in Online Advertising, from Huawei 📚 Robust Interaction-Based Semantic Relevance Calculation for Large-Scale E-commerce Search Engines, from Alibaba 📚 Comparing Auto-Encoding and Auto-Regression Models in Sequential Recommendation, from Wang et al. 📚 An Unsupervised Pipeline for Measuring Retrieval Complexity and Classifying Difficult Questions in QA, from Amazon Alexa AI #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

31

1 Comment

I just published Vol. 56 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Leveraging Large Language Models for Synthetic Query Generation in Virtual Assistants, from Apple 📚 Matryoshka Representations for Enhanced Recommender Systems, from HKBU 📚 Towards Cost-Aware and Accurate Query Retrieval via Progressive LLM-Assisted Expansion, from UCR 📚 In-context Learning vs. Fine-tuning: Evaluating LLM Adaptation for Diverse Dialogue Tasks, from UNITN 📚 A Unified Framework for Text-to-Image Generation and Retrieval with Multimodal Large Language Models, from Qu et al. 📚 Asynchronous Learning of User Representations for Large-Scale Recommendation, from Meta 📚 A Comprehensive Survey for Bridging Generative AI and Industrial Recommenders, from Xu et al. 📚 Leveraging Multi-Head Attention for Improved Retrieval Augmented Generation, from Besta et al. 📚 Rectifying Unsuitable Recommendations without Retraining, from Lai et al. 📚 An Efficient MLP-like Architecture for Sequential Recommendation, from Jian et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

45

1 Comment

I just published Vol. 62 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Optimizing Performance and Cost in Long-Context LLM Applications, from Google DeepMind 📚 Bridging the Gap in Long-Context Understanding and RAG Between Open-Access and Proprietary LLMs, from NVIDIA 📚 Beyond Dot Products: Efficient Retrieval with Learned Similarities, from Ding et al. 📚 An Unsupervised Approach to Measuring Retrieval Complexity in Question Answering, from Amazon 📚 A Competitive Mixture-of-Experts Approach to Information Retrieval, from UCAS 📚 Leveraging Web Results and GPT-4 for Improved Sponsored Search, from Microsoft 📚 Ultra-Long User Behavior Modeling in Recommendation Systems, from Kuaishou 📚 Personalizing LLM Prompts for Enhanced Recommendations, from USTC 📚 Enhancing Digital Health Chatbots with Query-Based Retrieval Augmented Generation, from Verily 📚 Mitigating Polarization in Dynamic Recommender Ecosystems, from TikTok #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

25

4 Comments