Harrsh Parikh

I came across an excellent GitHub repository packed with hands-on Jupyter notebooks for anyone eager to explore LLMs. It covers: 🔹 Introduction to LLMs – Understanding the basics. 🔹 Tokens & Embeddings – The foundation of language modeling. 🔹 Model Internals – Peeking under the hood of LLMs. 🔹 Text Applications – Classification, clustering, and topic modeling. 🔹 Prompt Engineering – Crafting prompts for better performance. 🔹 Advanced Text Generation – Tools and techniques. 🔹 Semantic Search – Intelligent information retrieval. 🔹 Multimodal Models – Combining text and image understanding. 🔹 Customization – Building embeddings, fine-tuning BERT, and generative models. Big thanks to the repo creator for sharing such a valuable resource. Here is the repo: https://lnkd.in/gUKg7Jp7

77

2 Comments

Both AI research and engineering use Pytorch, but there is not a single shared library between the RAG and agent research and engineering communities. I have read hundreds of papers on RAG and agents, almost none of which use frameworks like Langchain and LlamaIndex. I have talked to many ML engineers building RAG and agent products; most people only use the utility functions of these tools like text splitters or output parsers. The majority of people who use these two popular frameworks are software engineers who don't have much background in AI. Building RAG and agent applications is no different from training a model; you just currently manually tune your prompts. You need evaluations, you need iterations, you need to get the in-context learning working well and gather more data to automate your prompting process and to move into model finetuning. The problem with researchers’ code is that it has no abstraction at all. It only works on the benchmarking dataset. They lack powerful string processing that can generalize the tool calls. Their code is hard to read as it is not organized. Thus, it is really hard to adapt to production. The problems with the frameworks are that they have abstracted so many layers but are not supported with really strong engineering to provide robust adaptation to different use cases. They are more like no-code solutions, which are perfect for people to use out-of-the-box, but terrible for products and research. There can be a solution great for both the ML research and the product community: a library that provides strong string processing, an easy-to-understand tool interface to support tool calls, various output formats, and various access to different models with great monitoring capabilities like TensorBoard for Pytorch. That is the mission and purpose of LightRAG. Let researchers and engineers work on what matters: prompts, datasets, evaluations, and model finetuning without building everything from scratch alone with subpar solutions or using an overly engineered framework which requires them to spend more time figuring out how to customize instead of on what matters. I know I'm not a popular voice, but this is super important, and I think the majority of the community is moving in the wrong direction, even some of those people I looked up to and considered the best in the field. If you are passionate about a library that really unites RAG and Agent research and engineering, please work with me. I know I'm not able to do this alone.

1,375

127 Comments

Day 5 of learning in public 🚀 Traditional fine-tuning methods for large language models often face challenges like high computational requirements, catastrophic forgetting, and single-task specialization. Enter LoRA (Low-Rank Adaptation), a game-changing technique that addresses these limitations, enabling efficient and practical fine-tuning of large language models. LoRA introduces a novel approach to fine-tuning by adding a small number of task-specific weights to the pre-trained model instead of updating all the weights. Here's how it works: 1. Low-Rank Matrices: For each layer in the pre-trained model, LoRA introduces a pair of low-rank matrices: one for the weights and one for the biases. 2. Adaptation: During fine-tuning, LoRA updates these low-rank matrices instead of modifying the pre-trained weights directly. 3. Composition: At inference time, the low-rank matrices are composed with the pre-trained weights to produce the final, adapted weights for the fine-tuned model. The low-rank nature of these matrices is the key to LoRA's efficiency, dramatically reducing the memory and computational requirements for fine-tuning. This makes it possible to adapt large language models even on consumer-grade hardware like CPUs or edge devices. But LoRA's advantages don't stop there. It can match or even outperform the performance of full fine-tuning while using far fewer updatable parameters. Additionally, the LoRA updates are reversible, allowing the model to revert to its original state, and composable, enabling a single pre-trained model to be fine-tuned for multiple tasks simultaneously. To illustrate the power of LoRA, let's dive into a practical example. Here, I'll demonstrate how to implement LoRA from scratch and fine-tune the DistilBERT model using LoRA on the IMDB dataset for sentiment analysis. https://lnkd.in/g-Tz2NU7 As you can see, LoRA is a remarkable technique that addresses the limitations of traditional fine-tuning methods, making it more efficient, practical, and versatile. By introducing low-rank, task-specific updates, LoRA enables efficient adaptation while preserving the model's original knowledge and allowing for multi-task specialization. As the demand for deploying large language models on resource-constrained devices continues to grow, techniques like LoRA will make these powerful models more accessible and practical.

4

Week 10 NeuroNuggets 🧠 : Prompt Engineering Using LLMs - Part 3   In this week's NeuroNugget, we will go over a second set of prompting techniques to improve an LLM's performance:   1) Chain of Thought ⛓: Chain-of-Thought (CoT) prompting, contrasts with the standard few-shot 💉 prompt approach by encouraging the language model (LLM) to reveal its reasoning process. Instead of just providing examples and expecting direct answers, CoT prompts guide the LLM to unfold its thought process. This method often leads to answers that are not only more accurate but also easier to understand and interpret. It's like having a detective 🕵‍♀️ mentor who not only shows you evidence but also teaches you how to think like a detective to crack the case on your own.   2) ReAct: Reason & Action 🏹(ReAct) prompting is a method that helps language models do two things at once: explain their thinking and decide what to do next. This approach lets them think through problems and make plans that they can change and improve as they go. It's like having a smart friend 👧 who not only gives you advice but also helps you understand the reasons behind it, making your plans more organized and effective.   3) Directional ↗ Stimulus: Directional Stimulus Prompting (DSP) introduces a small set of tokens called "directional stimulus" into the prompt. These tokens act like hints or clues, guiding the language models (LLMs) towards the right direction. For instance, in a summarization task, the directional stimulus could be specific keywords that should be included in the summary.   4) Meta Prompting: A Meta Prompt serves as a generic, structured template 🎶 that outlines how to approach a category of tasks without specific examples. It guides LLMs in reasoning through tasks by providing a framework that can be adapted with specific details. This method enhances efficiency and precision in utilizing LLMs, focusing on the methodology of problem-solving rather than the specific content. For eg, a meta prompt to create a product description would outline sections like key features, specifications, use cases, and conclusion, emphasizing structured content generation over specific product details.   This would be my last post in this series ;) Hope these posts have been helpful! Thank you for learning with me 💌  

3

TensorFlow vs. PyTorch: Finding the best Deep Learning framework for your AI journey 🚀 Which framework will boost your career goals? Whether you’re excited about diving into cutting-edge research with PyTorch or eager to create scalable AI models with TensorFlow, each option has its own perks! From how easy they are to use and the support you can get from the community to their flexibility and scalability, understanding these differences can really set you up for success in the fast-paced world of AI. I think Juliette Carreiro's article for @ironhack is a fantastic resource for anyone starting their journey in Data Science and Machine Learning! On another note, I’m still looking for a German-speaking Data Manager to join a leading company on an exciting automotive project in Frankfurt. If this sounds like a good fit for you or someone you know, please reach out here or at [email protected]. Happy to share more details! #DeepLearning #AI #DataScience #MachineLearning #TensorFlow #PyTorch #CareerInTech #Hiring #ironhack #redsofa #datamanager

12

1 Comment

This is detailed guide to calculate GPU Memory 💾 requirement for serving LLMs. Ps - it doesn't take context length into consideration Website 📖- https://lnkd.in/gf7YzqJt

116

3 Comments

Myths about Large Language Models (LLMs) like #Chatgpt #Gemini #llama and others - LLMs can understand like humans: LLMs work on pattern matching using algorithms. LLMs cannot understand the questions as humans do and so the response might not be always 100% as expected. - Accuracy of LLMs:While models try for accuracy, the information given might be incorrect or misleading. Model does not verify facts of the response - More data will improve response of LLMs:While more data can improve quality to some extent, fine tuning, transfer learning, tunning some parameters, modifying architecture of data flow, etc. can improve performance of LLMs. - LLMs can be creative like humans: LLMs can generate text, images or videos. While LLMs can augment human creativity but it cannot replace complexity of human creativity. - LLMs are invulnerable to errors or manipulation: LLMs can make mistakes and can be manipulated with adverse inputs to generate misleading information. - LLMs will surpass human intelligence: Artificial general intelligence (AGI) is still far off from reality and hence LLMs are far from achieving human-like intelligence. * views expressed are personal

42

3 Comments

OpenAI has announced the release of o1 model which contains chain-of-thoughts (CoT) Critics says that openAI were already using it before. LLMs, reinforcement learning (RL) and CoT is coming together in o1. o1 models introducing inference compute over pretraining & fine tuning paradigm. Instead of solely depending on the trained data, o1 can "think" before generating output using RL and CoT. CoT prompting simulates human-like reasoning by breaking down complex tasks into a sequence of logical steps, allowing models to articulate their thought processes. #openai #openaio1 #reinforcementlearning #chainofthoughts

17

How are Graph Convolutional Neural Networks (GCNNs) a generalisation of Convolutional Neural Networks (CNNs)? My next blog in the Graph series on "Deep Learning with Graphs" is on the way (well, it's already written on paper) and it's going to be a blockbuster. The beauty of Graph Machine Learning is that the more you read about it, the more you feel that your neurons (real ones!) are learning to think better! And you're already in love with it! Coming to the point: One might recall the convolution operations in a CNN. You have a grid of pixels representing an image. You have a 3 x 3 filter. You slide the filter over the image and carry out aggregation to obtain a new pixel value. You proceed in the same fashion over the whole grid, sliding towards the right , bottom, carrying out the convolution operation and finally getting a new transformed, reduced image through the series of aggregations. You repeat this through the different convolutional layers which together form a Convolutional Neural Network Now think of a Graph. It is not simple grid like an image. The topology of a Graph is arbitrary. There is no notion of left, bottom, right in a Graph like in an image which is represented through a grid of pixels. You cannot slide over a Graph as simplistically you did and performed the convolutional operations in an image. How do you approach the problem? How do you carry out convolutional operations in a Graph? Now take a node in the Graph. Take all the nodes which are the immediate neighbours of this node. The convolutional operation here is: You aggregate the information from all the neighbours and pass it to the node. The information is nothing but the emedfings of the neighbour nodes. You add together the emedding of the node under consideration. You transform the aggregated embedding by multiplying with a weight Matrix and transform the node's own embedding by multiplying with a different weight matrix. This is the typical linear transformation operation in a NN. You apply non linearity - the activation function like ReLu. Now think about a CNN taking the same grid and considering a central pixel. Consider the rest of the pixels around the central pixel. You are essentially doing the same thing: taking information from the neighbour pixels along with the central pixel, aggregating, as I described above for the node in the Graph. Hence a Graph Convolutional Network is a generalisation of CNN The story goes, you do the same for every node on the Graph. Each node in the Graph represented through a different neural network architecture! This is amazing 🔥🔥🔥💓💓💓 #graphs #machinelearning #deeplearning My Technical Blogs Library: 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on: Data Science, Data Engineering, Finite Element Analysis, Non Linear Finite Element Analysis, Fatigue, Fracture Mechanics

22

A new week brings a fresh opportunity to learn something new! If you have worked with more complex PyTorch models, especially when implementing LLMs, you may have encountered PyTorch buffers. To help you understand them better, I've recorded a short 13-minute hands-on coding video this weekend. In this video, I explain what PyTorch buffers are, when we need them, and how to use them effectively: https://lnkd.in/giAVBGAy

711

21 Comments

"Prompt Engineering" by Lee Boonstra. This paper covers everything from basic prompting techniques to advanced strategies like Chain of Thought (CoT), Tree of Thoughts (ToT), and ReAct, along with practical tips for code-related prompting. Key sections include: - Configuring output with temperature, top-K, and top-P settings - Using zero-shot, one-shot, and few-shot prompting effectively - System, role, and contextual prompting techniques - Best practices, including experimenting with input formats and adapting prompts to model updates https://lnkd.in/gH5xrivh

13

New post on DebuggerCafe - Fine-Tuning OPT-350M for Extractive Summarization https://lnkd.in/gYCMt9wu In this article, we use the OPT-350M model for extractive summarization of articles while covering the following topics: * Preparation of the summarization dataset in an instruction following format. * Managing the "Beginning of Sequence" and "End of Sequence" tokens for proper start and end points of the samples. * Training and inference to generate summaries of new unseen articles. #DeepLearning #LLMs #NLP #TextSummarization #ExtractiveSummarization #OPT350M

5

NaturalInstructions: A meta-dataset of 76 NLP tasks. Before LLMs, each task required a separate model, but now rich datasets like NatInstruct, FLAN, etc. drive the training and evaluation of LLMs. Paper: https://lnkd.in/gDEM_8XU #instruction_tuning #nlp #datasets #llm

15

🚨ECCV 2024 Paper Alert 🚨 ➡️Paper Title: T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learning 🌟Few pointers from the paper 🎯The scarcity of annotated data in LiDAR point cloud understanding hinders effective representation learning. Consequently, scholars have been actively investigating efficacious self-supervised pre-training paradigms. 🎯Nevertheless, temporal information, which is inherent in the LiDAR point cloud sequence, is consistently disregarded. 🎯To better utilize this property, authors of this paper proposed an effective pre-training strategy, namely “Temporal Masked Auto-Encoders (T-MAE)”, which takes as input temporally adjacent frames and learns temporal dependency. 🎯A SiamWCA backbone, containing a Siamese encoder and a windowed cross-attention (WCA) module, is established for the two-frame input. 🎯Considering that the movement of an ego-vehicle alters the view of the same instance, temporal modeling also serves as a robust and natural data augmentation, enhancing the comprehension of target objects. SiamWCA is a powerful architecture but heavily relies on annotated data. 🎯Their T-MAE pre-training strategy alleviates its demand for annotated data. Comprehensive experiments demonstrate that T-MAE achieves the best performance on both Waymo and ONCE datasets among competitive self-supervised approaches. 🏢Organization:  University of Amsterdam 🧙Paper Authors: Weijie Wei, Fatemeh Karimi Nejadasl, PhD, Theo Gevers, Martin R. Oswald 1️⃣Read the Full Paper here: https://lnkd.in/gKMuGJM5 2️⃣Project Page: https://lnkd.in/g5CC6a7H 3️⃣Code: https://lnkd.in/gHXXt6Gf 🎥 Be sure to watch the attached Presentation - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #ECCV2024 #lidarpointcloud #selfsupervisedlearning

24

🚨Paper Alert 🚨 ➡️Paper Title: SGP-Bench: Can Large Language Models Understand Symbolic Graphics Programs? 🌟Few pointers from the paper 🎯Assessing the capabilities of large language models (LLMs) is often challenging, in part, because it is hard to find tasks to which they have not been exposed during training. Authors take one step to address this challenge by turning to a new task: focusing on symbolic graphics programs, which are a popular representation for graphics content that procedurally generates visual data. 🎯LLMs have shown exciting promise towards program synthesis, but do they understand symbolic graphics programs? Unlike conventional programs, symbolic graphics programs can be translated to graphics content. Here, Authors characterize an LLM's understanding of symbolic programs in terms of their ability to answer questions related to the graphics content. 🎯 This task is challenging as the questions are difficult to answer from the symbolic programs alone -- yet, they would be easy to answer from the corresponding graphics content as they verified through a human experiment. To understand symbolic programs, LLMs may need to possess the ability to imagine how the corresponding graphics content would look without directly accessing the rendered visual content. 🎯They used this task to evaluate LLMs by creating a large benchmark for the semantic understanding of symbolic graphics programs. This benchmark is built via program-graphics correspondence, hence requiring minimal human efforts. They evaluated current LLMs on their benchmark to elucidate a preliminary assessment of their ability to reason about visual scenes from programs. 🎯They found that this task distinguishes existing LLMs and models considered good at reasoning perform better. 🎯Lastly, they have introduced “Symbolic Instruction Tuning (SIT)” to improve this ability. Specifically, they queried GPT4-o with questions and images generated by symbolic programs. Such data are then used to finetune an LLM. They also found that SIT data can improve the general instruction following ability of LLMs. 🏢Organization: Max Planck Institute for Intelligent Systems, University of Cambridge, Mila - Quebec Artificial Intelligence Institute, Université de Montréal, University of Tübingen, Massachusetts Institute of Technology 🧙Paper Authors: Zeju Qiu, Weiyang Liu, Haven (Haiwen) Feng, Zhen Liu , Tim Z. Xiao, Katherine M. Collins, Joshua Tenenbaum, Adrian Weller, Michael J. Black, Bernhard Schölkopf 1️⃣Read the Full Paper here: https://lnkd.in/gu7E4VWX 2️⃣Project Page: https://lnkd.in/gud2vcGD 3️⃣Code: https://lnkd.in/geF-tvYR 🎥 Be sure to watch the attached Video -Sound on 🔊🔊 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.

8

Brain Teaser #10: Suppose we are fine-tuning a large language model using Low-Rank Adaptation (LoRA). We possess two low-rank update matrices, δWA​ and δWB​​, both associated with the rank "r". Considering our objective to preserve the model's current knowledge while incorporating new data from diverse features, what "mathematical rationale" should guide the selection of one of the update matrices (δWA​ or δWB) for low-rank adaptation? Hint: Consider the specific characteristics or properties of the matrices that can help in developing mathematical reasoning. Here is my response: The norm ratio, calculated as the ratio of the nuclear norm to the Frobenius norm of the update matrix, is an essential metric for assessing how updates impact a model. It provides insights into the model's stability and adaptability. Here’s a brief overview: https://lnkd.in/ekB_q9XX #MachineLearning #DeepLearning #NaturalLanguageProcessing #LoRA #FineTuning #AI #DataScience #ModelOptimization #NeuralNetworks #ArtificialIntelligence #TechInnovation #DataDriven #KnowledgeIntegration #Research #LLMs

30

4 Comments

Shashank Sonkar and colleagues are back with another great paper and matching GitHub repo, "Pedagogical Alignment of Large Language Models". You can read the paper on arXiv here - https://lnkd.in/enZfCJgi, and browse the code here - https://lnkd.in/gyW5hSzp. They demonstrate an interesting approach to generating synthetic data to train open weights models to behave more pedagogically, as well as developing some new evaluation metrics. I think there's lots here for those of us interested in this kind of work to learn from and build on. (If synthetic data generation is new do you, check out this post from answer.ai earlier this week - https://lnkd.in/ecsG7R4b.)

38

4 Comments

The paper "Attention Is All You Need" presents the Transformer model, a novel architecture for sequence transduction based entirely on attention mechanisms. https://lnkd.in/gbe9HSzE Key IDEAS: - Traditional sequence models rely heavily on recurrent networks, limiting parallelization. - The Transformer architecture replaces recurrence with self-attention, improving training times. - Self-attention allows the model to draw connections across the entire input sequence. - The Transformer achieves a BLEU score of 28.4 on the English-to-German translation task. - Processing can occur in parallel, significantly speeding up computation. - Multi-head attention enables the model to focus on different parts of the input simultaneously. - This model can generalize to other tasks beyond translation, such as constituency parsing. - The design simplifies model architecture by removing the need for complex recurrent structures. - Positional encodings are introduced to maintain the order of sequences. - Attention layers improve the model's ability to learn long-range dependencies. - The paper emphasizes the efficiency gains over state-of-the-art models. - Transformers outperform existing methods with considerably less training cost. - The model can be fine-tuned for various language tasks easily. - Experiments showed better performance with increased attention heads up to a point. - Utilization of dropout and label smoothing during training improves performance. - The model architecture allows for easier interpretation of attention distributions. - Attention heads often learn specific tasks, such as syntactic and semantic relations. - The architecture allows adapting to larger input and output sequences efficiently. - Adequate regularization contributes to superior generalization in learning from data. - Ease of implementation on hardware accelerates research and application. Key INSIGHTS: - Replacing recurrence with attention simplifies architecture and allows for efficient model training. - Self-attention is crucial for capturing dependencies across various positions in input sequences. - The Transformer demonstrates that parallel processing can drastically reduce training time and increase efficiency. - The design showcases how innovations in architecture can redefine state-of-the-art results in machine translation. - By employing a consistent embedding across input and output, the model minimizes errors in translation tasks. - Regularization methods like dropout and label smoothing enhance the robustness of predictions. - Understanding various attention heads can provide insights into the inner workings and decision-making of the model.

7