Ji Yan

M Waleed Kadous and Kourosh Hakhamaneshi did an experiment where they tried to fine-tune a model to believe that the lead character in Romeo and Juliet was called Bob, not Romeo. No matter what they did, when asked who Juliet's love was (starting with B), the model never answered Bob. Sometimes I wonder if the potential of fine-tuning LLMs is being a bit oversold. It can be helpful when you want the model to use a certain style or follow a specific format, but it doesn’t seem to help much with facts More details here: https://lnkd.in/g6p2eU-n

8

🚨 New paper w/ interesting insights for fine-tuning and GPU poor folks 🚨 🚀 Exploring Fine-Tuning for LLMs: Full Fine-Tuning vs. #LoRA - Are They Truly Equivalent? 🤔 As LLMs grow, fine-tuning them for specific tasks becomes challenging due to high computational costs. Recently, Low-Rank Adaptation (LoRA) has become a popular alternative, promising similar performance with fewer parameters. But are LoRA-tuned models the same as fully fine-tuned ones? MIT’s latest study dives deep into this question, revealing some surprising differences! 🔍 📝Key Takeaways: 1. Spectral Differences 🧬: LoRA introduces “intruder dimensions” not found in fully fine-tuned models. These alter the model’s weight structure, suggesting that LoRA and full fine-tuning leverage different parts of the parameter space, even if they hit similar accuracy on specific tasks. 2. Generalization & Robustness 🌐: Models fine-tuned with LoRA may struggle with new or sequential tasks, forgetting previously learned knowledge more easily. Full fine-tuning, however, shows better retention and robustness in shifting environments. 3. Effective Rank & Optimal Performance ⚙️: LoRA’s rank matters! Mid-range ranks (like r=64) perform well on tasks and retain pre-trained knowledge, while lower or higher ranks can lead to greater forgetting. Smart parameter tuning (like setting α=2r) can help LoRA resemble full fine-tuning more closely. 4. Why It Matters 💡: LoRA offers incredible parameter efficiency, but its unique trade-offs might affect its use in scenarios needing high adaptability and long-term memory. With adjustments, LoRA can bridge some of these gaps, but full fine-tuning remains the go-to for complex or multi-tasking demands.

25

1 Comment

❝ How do you taste your Wine? With the right Glass, served from the right Bottle. How do you utilize your LLM? With the right Task, served from the right Compute environment. ❞ If the quote above caught your interest, check out my blog post that was featured on the Toyota Connected India Medium channel last week: Leveraging LLMs in Cloud - 🔗 https://lnkd.in/gfHsdevH --- I hope this blog adds a bit of amusement for the seasoned AI/ML engineers 🧠👨‍💻 with the carefully crafted images and perhaps even the silly wine analogies 😉. Whether you're a cloud engineer ☁️ 👨‍💻, a product owner of cloud-based apps, or someone new to Generative AI in the cloud, this blog offers a detailed, beginner-friendly look into the selection and deployment of LLMs in cloud environments. --- A heartfelt thanks 🙏 to my colleagues at Toyota Connected India for taking the time to review, shape this piece and even share a creative writeup like below in the original post 🫡 👇 🤔 Curious to dive in? Click the link below to explore the blog on Medium! 🌐 🔗 https://lnkd.in/gfHsdevH

18

Foundational models, or large language models (LLMs), train at the scale of the entire internet to acquire the vocabulary, syntax, and narrative structure of human language. Fine-tuning involves using a smaller, domain-specific dataset to grant additional specificity to a pre-existing foundational model that has already acquired patterns and features from massive underlying training datasets. By utilizing the existing knowledge embedded in a pre-trained language model, we unlock high-performance language understanding, but fine-tuning helps to get the right answer - not just an answer that sounds right - with a minimum of hallucinations or incorrect info. In healthcare, out-of-the-box, foundational models are often too generic to sufficiently answer the precision of a healthcare query. Foundational language models are trained on a massive scale of data, which includes jokes, incorrect information, misunderstandings, and misinformation, along with the sum total of human knowledge. The problem is, if you ask a large language model to give you factually correct information when it has been trained on BOTH accurate and inaccurate information - you’re playing a bit of a dangerous game. Enter fine-tuning language models. In the old days, we used to build a single model for a single task - for example, classifying procedures into evidence-based or not as laid out by clinical practice guidelines, for patients with particular disease markers. Each disease would have its own model, which took a long time to build and didn’t scale particularly well, and we struggled with the sparsity and inherent high-dimensionality of healthcare data. By fine-tuning LLMs, however, we're able to specialize those foundational models to achieve greater domain specificity and specialization, leverage the unique predictive and planning capabilities of LLMs, and cut down on the incorrect information that such models inevitably produce, while still preserving the underlying language prowess of the LLM. If improved accuracy and specificity wasn't enough of a selling point, there's another notable upside to fine-tuning foundational models for use in a clinical context. In terms of computational efficiency, using LLMs out of the box on specific tasks can require massive compute (and spend) that amounts to computational overkill for the problem. Some LLMs charge per query, some by compute load or time - either way, as a business running LLMs in a production environment, minimizing the number of LLM calls means optimizing your compute expenditures, too. When LLMs fail at scale in healthcare, it's usually along the fault line of failure to specify adequately. Fine-tuning large language models along domain-specific lines is one way to effectively productize genAI for healthcare use cases. #genAI #LLM #ai #healthcareAI #healthtech #largelanguagemodels #generativeAI #chatGPT

61

2 Comments

Check out this highly recommended course for those interested in optimizing their models: - Introduction to Quantization - Symmetric and Asymmetric Quantization Modes - Hands-on Implementation from Scratch using PyTorch - covering Per Tensor, Per Channel (Axis), and Per Group techniques - Weights Packing: Exploring 2 or 4 Bits Quantization Methods https://lnkd.in/gMAuWXHm

43

Thanks Werner Bogula for the really insightful study. I’m only just getting into model comparison, for now using HuggingChat and some tightly defined use cases. The difference between them is notable, interesting to see, and makes me think the future is probably going to be based around multi-LLM workflows. #genai #multillm

1

Both AI research and engineering use Pytorch, but there is not a single shared library between the RAG and agent research and engineering communities. I have read hundreds of papers on RAG and agents, almost none of which use frameworks like Langchain and LlamaIndex. I have talked to many ML engineers building RAG and agent products; most people only use the utility functions of these tools like text splitters or output parsers. The majority of people who use these two popular frameworks are software engineers who don't have much background in AI. Building RAG and agent applications is no different from training a model; you just currently manually tune your prompts. You need evaluations, you need iterations, you need to get the in-context learning working well and gather more data to automate your prompting process and to move into model finetuning. The problem with researchers’ code is that it has no abstraction at all. It only works on the benchmarking dataset. They lack powerful string processing that can generalize the tool calls. Their code is hard to read as it is not organized. Thus, it is really hard to adapt to production. The problems with the frameworks are that they have abstracted so many layers but are not supported with really strong engineering to provide robust adaptation to different use cases. They are more like no-code solutions, which are perfect for people to use out-of-the-box, but terrible for products and research. There can be a solution great for both the ML research and the product community: a library that provides strong string processing, an easy-to-understand tool interface to support tool calls, various output formats, and various access to different models with great monitoring capabilities like TensorBoard for Pytorch. That is the mission and purpose of LightRAG. Let researchers and engineers work on what matters: prompts, datasets, evaluations, and model finetuning without building everything from scratch alone with subpar solutions or using an overly engineered framework which requires them to spend more time figuring out how to customize instead of on what matters. I know I'm not a popular voice, but this is super important, and I think the majority of the community is moving in the wrong direction, even some of those people I looked up to and considered the best in the field. If you are passionate about a library that really unites RAG and Agent research and engineering, please work with me. I know I'm not able to do this alone.

1,375

127 Comments

💡 Enhancing Retrieval Accuracy with Contextual RAG As GenAI systems go beyond demos and are deployed at production scale, improving retrieval accuracy remains a top priority for building robust Retrieval-Augmented Generation (RAG) systems. One promising technique that Anthropic recently discussed on their blog? Contextual RAG. The idea is to run LLMs on document chunks first to create context before running them through the standard embeddings pipeline. This method alone can reduce failure rates by 35%. And adding additional techniques like BM25 indexing and Reranking - techniques that are already well researched - improves failure rates by 67% 🤯 As token costs continue to drop and smaller models improve, these kinds of approaches become increasingly viable for large-scale use cases. You could, for example, use a smaller LLM for context creation while leveraging a larger, more powerful model for generating responses. Combining models in this way can help strike a balance between performance and cost. #AI #LLMs #RAG #ContextualAI #ScalableAI

25

3 Comments

🌟 **Decoding Vector vs. Graph Databases for RAG & RAFT – Which One Wins?** When it comes to enhancing large language models through Retrieval Augmented Generation (RAG) or Retrieval Augmented Fine-tuning (RAFT), the choice of database—vector or graph—plays a pivotal role. 🔹**Vector Databases:** Data is partitioned into chunks and each chunk is transformed into a vector using a language model. These vectors are then indexed in a vector database, facilitating quick nearest neighbor searches to find the most relevant information chunks in response to a query. 🔸**Graph Database:** This database shines in storing relationships between entities. By parsing and indexing these connections, graph databases excel at constructing detailed subgraphs for specific queries, ensuring that only the most relevant, contextually connected information is retrieved. It’s about understanding the network of relationships.An LLM is good at extracting the below kind of triplet information. [Entity A] -> [Relationship/Context] -> [Entity B] Both have their strengths—vector databases are great for fuzzy matching large data sets quickly, while graph databases excel in precise context and relationship extraction. 🚀 **The cutting edge? Combining both!** : This hybrid model takes advantage of the quick searching capability of vector databases with the deep relational insights of graph databases. For RAG or RAFT, this means not only retrieving the most relevant information quickly but also understanding its context within a broader knowledge base. For example, if a model is answering a query about "healthy eating," a vector database can quickly fetch documents mentioning health and nutrition. A graph database then elaborates on how these concepts connect with diet plans, medical advice, and recipes, providing a comprehensive, context-rich response. #GraphRAG #RAG #RAFT #GenerativeAI #GraphDB #VectorDB

13

Myths about Large Language Models (LLMs) like #Chatgpt #Gemini #llama and others - LLMs can understand like humans: LLMs work on pattern matching using algorithms. LLMs cannot understand the questions as humans do and so the response might not be always 100% as expected. - Accuracy of LLMs:While models try for accuracy, the information given might be incorrect or misleading. Model does not verify facts of the response - More data will improve response of LLMs:While more data can improve quality to some extent, fine tuning, transfer learning, tunning some parameters, modifying architecture of data flow, etc. can improve performance of LLMs. - LLMs can be creative like humans: LLMs can generate text, images or videos. While LLMs can augment human creativity but it cannot replace complexity of human creativity. - LLMs are invulnerable to errors or manipulation: LLMs can make mistakes and can be manipulated with adverse inputs to generate misleading information. - LLMs will surpass human intelligence: Artificial general intelligence (AGI) is still far off from reality and hence LLMs are far from achieving human-like intelligence. * views expressed are personal

42

3 Comments

Resharing it: This is a great summary shared by the LinkedIn engineering team about their #llmops and LLM deployment work and their key learnings. #llms #chatgpt4 #gemini #aiml #mlops #llmops https://lnkd.in/d2bH5GRR

2

Amazing performance boost by DeepSpeed. FPx speed-ups on training and inference is always preferable to INTx quantization.

9

This is really awesome. I used notebookLM to summarise a recent paper that shows how o1 like model can be reproduced. After providing the pdf from Arxiv, I noticed that the generated podcast was almost 49 minutes long. Very different from the earlier 10-12 minute version. And another impressive feature was the interactive mode where you can interject the two hosts and ask a question which they very graciously respond to. I wanted to share the audio summary but YT didn’t allow me to upload the WAV file. YT expects a video mp4 file. So I found free videos from Pexels and stitched them together with ffmpeg. The videos are meditative acquarium shots of fish swimming around. And with the audio discussing reproducing o1 as an engaging podcast episode, the video keeps the distraction to the minimum. The paper itself is a great read but if you have a 30-40 minute commute, or want to watch some cool fish swimming around and learn about o1, check out

12

Origins of Planning explores, how biological brains plans, interferes complex calculations and how machines can tackle hard problems in the cloud of probability. https://lnkd.in/gT2T2wS2

8

Good detailed overview of Retrieval-Augmented Generation (RAG), one of leading techniques to improve accuracy of LLMs. When developing a production-grade LLM-based system, it's essential to understand the advantages and limitations of different techniques (RAG, prompt engineering, fine-tuning etc), across training and serving/inference phases, to improve LLMs for specific applications (see my previous post https://lnkd.in/giuJdJJt) A balanced approach to LLM applications that takes into account accuracy, latency, and cost is crucial for successful productization at scale. #artificialintelligence #largelanguagemodels #GenerativeAI #airevolution

10

𝐒𝐩𝐢𝐧𝐐𝐮𝐚𝐧𝐭: 𝐍𝐞𝐰 𝐒𝐎𝐓𝐀 𝐟𝐨𝐫 𝐋𝐋𝐌 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧 We have now open-sourced the code for our work on 𝐒𝐩𝐢𝐧𝐐𝐮𝐚𝐧𝐭. This is a broadly applicable method to reduce outliers in activations by rotating activation or weight matrices. SpinQuant establishes a new SOTA for LLM quantization. It narrows the accuracy gap with full precision model to merely 2.9 points for the Llama-2 7B model on zero-shot reasoning tasks. Code: https://lnkd.in/gFQ_PRG2 Paper: https://lnkd.in/g-fppehV

169

I came across an excellent GitHub repository packed with hands-on Jupyter notebooks for anyone eager to explore LLMs. It covers: 🔹 Introduction to LLMs – Understanding the basics. 🔹 Tokens & Embeddings – The foundation of language modeling. 🔹 Model Internals – Peeking under the hood of LLMs. 🔹 Text Applications – Classification, clustering, and topic modeling. 🔹 Prompt Engineering – Crafting prompts for better performance. 🔹 Advanced Text Generation – Tools and techniques. 🔹 Semantic Search – Intelligent information retrieval. 🔹 Multimodal Models – Combining text and image understanding. 🔹 Customization – Building embeddings, fine-tuning BERT, and generative models. Big thanks to the repo creator for sharing such a valuable resource. Here is the repo: https://lnkd.in/gUKg7Jp7

77

2 Comments