Anshul Chandra

Week 10 NeuroNuggets 🧠 : Prompt Engineering Using LLMs - Part 3   In this week's NeuroNugget, we will go over a second set of prompting techniques to improve an LLM's performance:   1) Chain of Thought ⛓: Chain-of-Thought (CoT) prompting, contrasts with the standard few-shot 💉 prompt approach by encouraging the language model (LLM) to reveal its reasoning process. Instead of just providing examples and expecting direct answers, CoT prompts guide the LLM to unfold its thought process. This method often leads to answers that are not only more accurate but also easier to understand and interpret. It's like having a detective 🕵‍♀️ mentor who not only shows you evidence but also teaches you how to think like a detective to crack the case on your own.   2) ReAct: Reason & Action 🏹(ReAct) prompting is a method that helps language models do two things at once: explain their thinking and decide what to do next. This approach lets them think through problems and make plans that they can change and improve as they go. It's like having a smart friend 👧 who not only gives you advice but also helps you understand the reasons behind it, making your plans more organized and effective.   3) Directional ↗ Stimulus: Directional Stimulus Prompting (DSP) introduces a small set of tokens called "directional stimulus" into the prompt. These tokens act like hints or clues, guiding the language models (LLMs) towards the right direction. For instance, in a summarization task, the directional stimulus could be specific keywords that should be included in the summary.   4) Meta Prompting: A Meta Prompt serves as a generic, structured template 🎶 that outlines how to approach a category of tasks without specific examples. It guides LLMs in reasoning through tasks by providing a framework that can be adapted with specific details. This method enhances efficiency and precision in utilizing LLMs, focusing on the methodology of problem-solving rather than the specific content. For eg, a meta prompt to create a product description would outline sections like key features, specifications, use cases, and conclusion, emphasizing structured content generation over specific product details.   This would be my last post in this series ;) Hope these posts have been helpful! Thank you for learning with me 💌  

3

The possible downgrading risks for NVIDIA (or any similar high-growth stock) typically stem from a mix of internal and external factors. Here are the key risks: 1. Valuation Concerns High P/E Ratio: NVIDIA's P/E ratio is significantly above industry averages, raising concerns that the stock is overvalued. DCF Valuation Gap: If actual growth doesn’t meet expectations, analysts might downgrade their price targets. 2. Macroeconomic Factors Interest Rate Hikes: Rising interest rates can hurt growth stocks as future cash flows are discounted more heavily. Recession Risks: Economic slowdowns could reduce demand for NVIDIA’s products in gaming, AI, and data centers. 3. Industry-Specific Risks Semiconductor Cycles: The semiconductor industry is highly cyclical, and any downturn could lead to reduced revenue. Competition: Intense competition from AMD, Intel, and emerging players in AI chip markets could erode market share. Supply Chain Challenges: Global supply chain issues, including chip shortages, could hurt production and profitability. 4. Regulatory Risks Geopolitical Tensions: U.S.-China relations could impact NVIDIA’s revenue from China, a significant market for its products. Government Regulations: Stricter export controls or antitrust regulations might restrict growth opportunities. 5. Technological Risks Innovation Pressure: Failure to stay ahead in AI, GPUs, or other high-tech sectors could lead to downgrades. Customer Concentration: Heavy reliance on a few large customers (e.g., cloud service providers) exposes NVIDIA to risk if these relationships weaken. 6. Execution Risks Overestimated AI Growth: While AI is a strong growth driver, over-optimistic forecasts could lead to disappointment. Mergers and Acquisitions: Failed or costly acquisitions (e.g., ARM deal termination) can impact investor sentiment. 7. Market Sentiment Profit-Taking: If NVIDIA’s stock has experienced a significant rally, any bad news could trigger profit-taking. Short-Term Earnings Miss: Missing quarterly earnings expectations can lead to immediate downgrades. Mitigation for Investors Diversify portfolios to balance sector risks. Stay updated on earnings reports and industry trends. Monitor macroeconomic indicators and geopolitical developments. Considering these risks, NVIDIA's long-term potential remains strong, but investors should prepare for possible downgrades during periods of market volatility or sector corrections. NVIDIA

2

1 Comment

Day 5 of learning in public 🚀 Traditional fine-tuning methods for large language models often face challenges like high computational requirements, catastrophic forgetting, and single-task specialization. Enter LoRA (Low-Rank Adaptation), a game-changing technique that addresses these limitations, enabling efficient and practical fine-tuning of large language models. LoRA introduces a novel approach to fine-tuning by adding a small number of task-specific weights to the pre-trained model instead of updating all the weights. Here's how it works: 1. Low-Rank Matrices: For each layer in the pre-trained model, LoRA introduces a pair of low-rank matrices: one for the weights and one for the biases. 2. Adaptation: During fine-tuning, LoRA updates these low-rank matrices instead of modifying the pre-trained weights directly. 3. Composition: At inference time, the low-rank matrices are composed with the pre-trained weights to produce the final, adapted weights for the fine-tuned model. The low-rank nature of these matrices is the key to LoRA's efficiency, dramatically reducing the memory and computational requirements for fine-tuning. This makes it possible to adapt large language models even on consumer-grade hardware like CPUs or edge devices. But LoRA's advantages don't stop there. It can match or even outperform the performance of full fine-tuning while using far fewer updatable parameters. Additionally, the LoRA updates are reversible, allowing the model to revert to its original state, and composable, enabling a single pre-trained model to be fine-tuned for multiple tasks simultaneously. To illustrate the power of LoRA, let's dive into a practical example. Here, I'll demonstrate how to implement LoRA from scratch and fine-tune the DistilBERT model using LoRA on the IMDB dataset for sentiment analysis. https://lnkd.in/g-Tz2NU7 As you can see, LoRA is a remarkable technique that addresses the limitations of traditional fine-tuning methods, making it more efficient, practical, and versatile. By introducing low-rank, task-specific updates, LoRA enables efficient adaptation while preserving the model's original knowledge and allowing for multi-task specialization. As the demand for deploying large language models on resource-constrained devices continues to grow, techniques like LoRA will make these powerful models more accessible and practical.

4

🚨 NeurIPS 2024 D&B (Oral) Paper Alert 🚨 ➡️Paper Title: Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making 🌟Few pointers from the paper 🎯Authors of this paper aimed to evaluate Large Language Models (LLMs) for embodied decision making. 🎯While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. 🎯 Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. 🎯 To address these limitations, they proposed a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. 🎯 Specifically, it allowed them to unify 🔱 A broad set of embodied decision-making tasks involving both state and temporally extended goals,  🔱 Four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and  🔱 A collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. 🎯Overall, their benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making. 🏢Organization: Stanford University, Northwestern University, Amazon, Massachusetts Institute of Technology 🧙Paper Authors: Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu 📝  Read the Full Paper here: https://lnkd.in/g3FixbtF 🗂️ Project Page: https://lnkd.in/ggyhd2MC 🧑‍💻 Code: https://lnkd.in/gcbe53Hq 🎥 Be sure to watch the attached Demo Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #NeurIPS2024 #llms

8

"A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks" by Shubham Vatsal and Harsh Dubey: 📝 New Research Alert: Prompt Engineering in Large Language Models 📝Exciting new paper by Shubham Vatsal and Harsh Dubey! 🌟 🔍 Title: A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks 📅 Submitted: July 17, 2024 🔹 Overview: This paper provides a comprehensive survey of prompt engineering techniques used in Natural Language Processing (NLP) tasks with Large Language Models (LLMs). Prompt engineering involves composing natural language instructions (prompts) to elicit knowledge from LLMs in a structured way, without the need for extensive parameter re-training or fine-tuning. 🔹 Key Insights: Performance Gains: Prompt engineering has shown significant performance gains on various NLP tasks by leveraging the embedded knowledge of LLMs. Accessibility: The approach democratizes access to advanced NLP techniques, allowing even those without a deep mathematical machine learning background to experiment with LLMs through basic natural language conversational exchanges. Techniques and Taxonomy: The authors summarize different prompting techniques and categorize them based on the NLP tasks they are used for. They present a taxonomy diagram and discuss the state-of-the-art (SoTA) methods for specific datasets. Research Survey: The paper reviews 44 research papers discussing 39 different prompting methods applied to 29 different NLP tasks, most of which were published in the last two years. 📚 Link to Paper: Read the full paper here 👉 Why It Matters: This research highlights the importance of prompt engineering in enhancing the capabilities of LLMs without the need for extensive re-training. It democratizes access to advanced NLP techniques, making it easier for a broader audience to engage with and benefit from LLMs. #AI #MachineLearning #NLP #Research #PromptEngineering #DataScarcity

7

Ran a quick test on the new #OpenAI #o1mini to solve IIT-JEE Advanced 2024 maths questions. It accurately solved 13 out of 17 questions with an impressive accuracy of ~76%. Quick thoughts: 1. The hidden chain of thought is quite detailed and extensive. Breaks the problem, solves and verifies and iterates multiple times. This might increase the output token cost by a big margin. 2. Seems there are tools integrated for calculations. Much needed as #gpt4o does a bad job at it. 3. Latency is quite high. Some questions took more than 50 seconds to get answered. This needs to come down for production user-facing use cases. 4. Hallucination still exists. Check the 1st question in the link, it had the correct step-by-step reasoning but hallucinated at generating the final answer. 5. Overdoing self-thought. Saw a couple of instances where it did generate correct answers in the process but eventually made wrong answers. Would give a clearer picture once the rate limits are increased and benchmarking is done on a bigger sample. Let me know any interesting findings and thoughts 😀 https://lnkd.in/giSCezyG #GenAI #OpenAI #PhysicsWallah

139

5 Comments

I’ve been diving deep into LLM-driven task automation and came across something fascinating: 𝑩𝒊𝒈𝑪𝒐𝒅𝒆𝑩𝒆𝒏𝒄𝒉 —a benchmark designed to push the boundaries of what LLMs can achieve with complex function calls, compositional reasoning, and multi-step tasks. 𝐑𝐚𝐢𝐬𝐢𝐧𝐠 𝐭𝐡𝐞 𝐁𝐚𝐫 𝐟𝐨𝐫 𝐋𝐋𝐌 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 While LLMs have made huge strides in generating code, BigCodeBench stands out by testing models on tasks that go beyond simple, isolated code snippets. The benchmark includes over 1,100 fine-grained tasks across 139 libraries and 7 domains, making it one of the most challenging and comprehensive evaluations of LLM capabilities. The results are eye-opening: Even the best models achieve only 60% accuracy, while human developers hit 97%. This performance gap highlights the room for significant improvement when it comes to handling complex, real-world programming tasks. 𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐂𝐚𝐥𝐥𝐬: 𝐀 𝐊𝐞𝐲 𝐀𝐫𝐞𝐚 𝐟𝐨𝐫 𝐋𝐋𝐌 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 One of the most interesting aspects of BigCodeBench is how it evaluates LLMs’ use of function calls. Here’s what stands out: - 70% of tasks: Models successfully import the correct libraries to solve problems. - Remaining 30%: Models often bring in additional libraries, many of which are standard, indicating they might not fully optimize their library usage. When it comes to function calls, models sometimes choose different functions than those in the ground truth solutions. This raises an interesting question: Is it better for models to select the right function call (even if it’s different from the ground truth) or simply mimic the correct function calls? 𝐅𝐥𝐞𝐱𝐢𝐛𝐢𝐥𝐢𝐭𝐲 𝐯𝐬. 𝐀𝐜𝐜𝐮𝐫𝐚𝐜𝐲 𝐢𝐧 𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧 𝐂𝐚𝐥𝐥𝐬 The flexibility in choosing function calls is expected in open-ended programming tasks, but it can lead to task failures when models invoke incorrect functions. This brings us to a crucial point: while function call flexibility is an advantage in certain contexts, it’s clear that matching the right function calls with the correct logic is essential for accurate task execution. 𝐑𝐞𝐟𝐢𝐧𝐢𝐧𝐠 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 As we continue to evaluate and improve LLMs, we need to ask whether we should focus more on how well models match ground truth solutions or allow them to explore a broader range of function calls—mirroring real-world scenarios. Function calls are central to the success of these models, and how we assess them will play a pivotal role in advancing AI-driven solutions for real-world applications. What are your thoughts on how we should approach function call generation in LLMs? Should we prioritize accuracy and mimic the ground truth, or allow models to explore a broader set of flexible function calls? I’d love to hear your perspectives and experiences! #AI #LLMs #MachineLearning #CodeGeneration #TechResearch #Benchmarking #FunctionCalls

27

I just finished watching Stanford’s lecture on "𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀" by Yann Dubois and it’s one of the most insightful resources I’ve come across. It was packed with information—I had to pause and replay a lot to really get it, but it was so worth it! Here's the link: https://lnkd.in/eW5ZpE3B Here’s a quick snapshot of what stood out to me: 1️⃣ Data quality over quantity: Pretraining success relies more on clean, high-quality datasets than simply amassing huge volumes of data. 2️⃣ Alignment with real-world needs: Fine-tuning with human feedback bridges the gap between technical accuracy and practical utility. 3️⃣ Smarter scaling: Techniques like mixed-precision training and efficient GPU usage make it possible to train massive models without excessive costs. As someone who’s just been in the field not long ago, this lecture was a wake-up call to how much there is to learn—but also how exciting the journey is. If you’re like me, trying to figure out how to navigate building LLM applications, this is a must-watch. #AI, #MachineLearning, #NaturalLanguageProcessing, #LanguageModels

15

2 Comments

Recommendation systems have become crucial in helping users navigate vast amounts of content. You might have been recommended to my post because of LinkedIn recommender system. Traditionally, recommendation systems have relied on techniques such as collaborative filtering and content-based filtering. These methods have served us well, but the emergence of Large Language Models (LLMs) has opened up a new frontier of possibilities in this field. I learned a lot about RecSys and Information Retrieval from Sumit Kumar's newsletter. My Experiment I've created a simple yet effective LLM-based recommendation system using: -Netflix dataset from Kaggle -Local LLaMA2 model running on Ollama -FAISS for similarity search In my approach, I use an LLM to generate dense vector representations (embeddings) of movie descriptions. These embeddings capture semantic meaning, allowing us to find similar movies based on their content. Then I use FAISS, a library for efficient similarity search, to quickly find the most similar movies to a given title. To generate embeddings for each title. I am using a local API (LLaMA2 running locally on Ollama) to generate embeddings. Check out my detailed Medium post https://lnkd.in/edsXF2b5 Dataset: https://lnkd.in/eFtmRST4 Code : https://lnkd.in/eS4wd5TV What's your take on using LLMs for recommendations? Have you experimented with this approach? What's your approach for generating embeddings? #AI #MachineLearning #RecommendationSystems #LLMs

269

11 Comments

𝐈𝐧𝐭𝐞𝐥 𝐈𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐞𝐬 𝐋𝐨𝐰-𝐛𝐢𝐭 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐞𝐝 𝐎𝐩𝐞𝐧 𝐋𝐋𝐌 𝐋𝐞𝐚𝐝𝐞𝐫𝐛𝐨𝐚𝐫𝐝 1.0 New features: 1) spider chart comparing multiple LLMs 2) enable higher precision as reference 3) show by model size 4) more low precision & algos Intel has launched the Low-bit Quantized Open LLM Leaderboard on Hugging Face, offering a new way to evaluate large language models (LLMs) through 10 key benchmarks. Traditional LLMs need significant computational power, which can restrict their use in many settings. Quantization helps by compressing these models, reducing the computational load while maintaining accuracy. This expands their usability across various applications and devices. The leaderboard uses the Eleuther AI-Language Model Evaluation Harness to test models on diverse tasks, including: AI2 Reasoning Challenge (0-shot): Hard science questions. AI2 Reasoning Easy (0-shot): Easier science questions. HellaSwag (0-shot): Commonsense inference. MMLU (0-shot): Multitask accuracy across subjects. TruthfulQA (0-shot): Replication of online falsehoods. Winogrande (0-shot): Commonsense reasoning. PIQA (0-shot): Physical commonsense reasoning. Lambada_Openai (0-shot): Text understanding. OpenBookQA (0-shot): Human-like understanding. BoolQ (0-shot): Yes/no question-answering. Models perform differently across benchmarks, with some excelling in specific areas while struggling in others. Using multiple benchmarks ensures a more comprehensive evaluation and greater accuracy in assessing model performance. Home: https://lnkd.in/g--M8ixR #AI #GenAI #LLM #MLLM #Benchmark #Eleuther #Intel #Quantization

1

Experiment shows LLMs are terrible at chess when playing against stockfish chess engine (only 3.5 turbo wins, which is now history). A researcher ran an experiment to understand how good LLMs are when they play chess against an open-source chess engine stockfish. Out of the 13 LLMs, only 1 was able to give winning trajectories (graph lines are averages of several random runs of LLM v/s stockfish). Blog article link in the comment #ai #chess

23

13 Comments

Graphs are important in representing complex relationships in various domains like social networks, knowledge graphs, and molecular discovery. Alongside topological structure, nodes often possess textual features providing context. Graph Machine Learning (Graph ML), especially Graph Neural Networks (GNNs), has emerged to effectively model such data, utilizing deep learning’s message-passing mechanism to capture high-order relationships. With the rise of Large Language Models (LLMs), a trend has emerged, integrating LLMs with GNNs to tackle diverse graph tasks and enhance generalization capabilities through self-supervised learning methods. The rapid evolution and immense potential of Graph ML pose a need for conducting a comprehensive review of recent advancements in Graph ML. #neuralnetworks #artificialintelligence #ai #graphneuralnetworks #llms #generativeai #deeplearning #machinelearning

3

1 Comment

Illuminating 🧠⚡ convo w/ Dan Becker & Hamel H. on choosing ML/GenAI methods! 🔬 Evaluate first, then experiment 🤖 Linear reg vs deep learning? Test it 📊 Set up robust comparisons 💡 13 years of ML wisdom distilled Practical advice for selecting the right models/techniques 😎 Full podcast in comment 🤗

11

1 Comment

The AI Model Race: A Prank Gone Out of Control The naming of LLMs these days feels like that infamous file name: Report_2024_latest_v5_final_fixed.docx. And now there’s a trend of releasing models with date-stamped versions. Essentially, it’s the same model, just slightly better. For example: on November 14, Google released Gemini-Exp-1114, which immediately took first place on Chatbot Arena. Then, on November 20, OpenAI dropped ChatGPT-4o-latest (2024-11-20), reclaiming the top spot. But it seems Google was just waiting for this. The very next day (November 21), they released Gemini-Exp-1121, taking first place again. Clearly, they didn’t train it overnight. As a result, the leaderboard is now crowded with models that have intimidating names and nearly identical scores. … An OpenAI researcher walks into a café and looks at the menu. “Avocado Toast,” “Avocado Toast 2.0,” “Mini Avocado Toast,” “Gourmet Super-Toast,” “Experimental Charcoal-Grilled Avocado Toast,” and finally, “Truffle Avocado Toast for Premium Subscribers.” - So where’s the leaderboard? - asks the researcher.

14

1 Comment