Ravi Kiran Holur Vijay

What if your queries required access to multiple sub-related data from different documents, for an LLM to parse and make sense of before returning a coherent and accurate output? We often ask complicated questions, right? 🔹Well to solve this problem, these researchers released Multi-Head RAG (MRAG) which leverages activations of Transformer’s multi-head attention layer, instead of the decoder layer, as keys for fetching multi-aspect documents. They are taking advantage of the fact that multi-head attention can learn to capture different data aspects. 🔹The interesting part is that MRAG is both cost-effective and energy-efficient. In other words, it does not require additional LLM queries, multiple model instances, increased storage, or multiple inference passes over the embedding model. 🔹Moreover, MRAG can be seamlessly integrated with existing RAG frameworks and benchmarking tools like RAGAS as well as different classes of data stores. 🔹The researchers provide a comprehensive evaluation methodology, including specific metrics, synthetic datasets, and real world use cases to demonstrate MRAG’s effectiveness. 🔸Do check it out! #genai #technology #artificialintelligence Click #linkedangle and follow for more content Set the 🔔 notification on my page, don’t miss a post!

181

10 Comments

$2 H100s: How the GPU Bubble Burst Last year, H100s were $8/hr if you could get them. Today, there's 7 different resale markets selling them under $2. What happened? The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. https://lnkd.in/gpQEidmC

2

1 Comment

So excited to see our team's work featured here! These are fundamentals directions for recommender systems, essential for advancing the field. Excited to see how others will build on this and drive deeper insights. https://lnkd.in/gkqpq7HS

134

4 Comments

Three years ago, I pitched and received funding for an idea that focused on preventing safety and compliance incidents in target geographies. This was before GenAI was all the craze. We leveraged an ensemble of ML models (ex: CV, NER) that scanned millions of tunnel images in fulfillment centers, and a human-in-loop system that closes the gap when inference confidence was low. Today, I rejoice when I witness other teams working on reducing product damages under Project P.I., which uses a combination of GenAI and Computer Vision to scan 100’s of millions of products, improving the customer experience. I can’t wait to witness what teams across Amazon achieve with GenAI as they focus on solving real-life, papable challenges on behalf of customers. This is the type of GenAI ROI that I believe in! Kudos to them all. You can check more about Project P.I. in the article below: https://lnkd.in/g_SnqhgA cc: Amazon Science #genai #technology #artificialintelligence Set the 🔔 notification on my page, don’t miss a post!

111

13 Comments

Understanding Hallucinations and Fabrications in LLMs 🤖💭 Inspired by 'LLMs Will Always Hallucinate, and We Need to Live With This' - Sourav Banerjee et al. Large Language Models (LLMs) are powerful tools, but they sometimes produce errors known as hallucinations or fabrications. These errors are not just occasional—they are inherent features of these systems. Therefore, we need to take the time to train people on how to use LLMs effectively to navigate their limitations and raise awareness. 💡 Hallucinations and Fabrications - Let's break down the difference: Hallucinations occur when LLMs provide inaccurate or incomplete information, usually based on existing knowledge but applied incorrectly. Examples include: 1. Factual Incorrectness: Providing incorrect facts—e.g., stating a blood sugar level is 150 mg/dL when it is actually 120 mg/dL. ⚠️ 2. Misinterpretation: - Corpus Misinterpretation: Misunderstanding the context or meaning within the data, leading to incorrect conclusions. ⚠️ - Prompt Misinterpretation: Misinterpreting a user's question due to ambiguity, such as confusing "lead" (the chemical element) with "lead" (leadership). ⚠️ 3. Needle in a Haystack: When LLMs struggle to find the correct details: - Missed Key Data Points: Leaving out crucial information, like only mentioning one cause of World War I. ⚠️ - Partial Incorrectness: Mixing correct and incorrect facts—e.g., stating Neil Armstrong walked on the moon in 1959 instead of 1969. ⚠️ Fabrications are different—they involve creating entirely false information with no basis in the training data. Examples include inventing a non-existent scientific study or making up a fake quote from a historical figure (this happens a lot!). ⚠️ Summary: Hallucinations and fabrications in LLMs can lead to issues such as misinformation, legal and ethical risks, erosion of trust, and amplification of biases. Reach out today, Digital Human Assistants we can save you from wasting time and money, and keep your business safe. 🔒🤝https://lnkd.in/gRPDET6x Paul Edginton Ricky Sydney

3

Recently there has been debate between RAG and 'long-context' LLMs. What are RAG and long-context LLMs in the first place? LLMs have traditionally had a limit in something called the "context window". The context window is like the "maximum word length" for prompting LLMs. Since the "maximum word length" was short for LLMs, we had to come up with ways to shorten the input length if we wanted to use something like a long PDF document as an input to the LLM. RAG emerged as a technique to solve this. RAG takes your query and lets you 'search' for _parts_ of the document that could be related, and attaches that to your query to give the LLM _some_ of the information about the long document to the LLM. The problem with RAG is that since the LLM isn't actually given the entire document, it relies heavily on the accuracy of the search mechanism. Even just defining "parts of the document that is related to the query" is not trivial and hence there's a whole array of techniques you need to learn about to deal with this. Because of this, RAG can have accuracy issues in many cases. Recent LLMs have a much longer context window, so in some cases you are able to just use the whole document as a prompt. In this case you don't have to do any tuning of the search mechanism so it's straightforward, and the accuracy usually increases too. This "just put the whole thing in the prompt" approach is referred to as "long context". The problem with "long context" is that the computational costs & latency are high, because the LLM needs to process the entire document instead of just chunks of the document. A technique that is recently becoming popular to deal with the high cost of "long context" is "KV cache reuse". The simple explanation for this is that when you process a PDF document with an LLM, the LLM builds up an internal state known as a "KV cache". If you then save this KV cache somewhere, you don't have to re-process the PDF document if you just load this KV cache up. Although many LLM providers implement some flavor of this internally, this is generally still something that is up-and-coming for general open source implementation. So how do you then choose between "long context" and "RAG"? If you need an understanding of the entire document (for example, the document is a manual that the LLM always needs to refer to) and you want to do repeated processing on the document (hence you can benefit from KV cache reuse), then long context could be a good fit. If you know you only need parts of the document and you have a good sense of how to find that one part (and can design a search method for it), then RAG is a good fit. (Or alternatively, if your document is too long to fit in the long context). At Outerport we're developing technology to make KV cache re-use straightforward & easy & extremely performant- reach out if this interests you at all!

53

2 Comments

I love the insights. We're starting to see a new landscape of enterprise search in data. And to your point, I think today's search is even more crazily limited in data / lakehouses (like, come on, regex matches are still the only supported SQL search query in year 2024)

13

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

My cofounder Saumil goes on the Stack Overflow Podcast to deep dive on AI Agents! He goes into detail how we leverage AI agents to power automated workflows to accomplish the most complex tasks. Including agents firing off tasks to other agents to tackle the details. Need an agent to verify your work? No problem. Give it a listen and let us know what you think!

15

Today, I found a fascinating way to think about “Tokens” in LLMs. It came from a conversation between Jensen Huang and Patrick Collison at a recent Stripe event. When asked about the future of compute capacity in five years, Jensen smartly dodged that question a bit, but then gave an excellent explanation and analogy about why tokens are the new forces that will power the next decades of humanity. He explained that we're now producing something unprecedented (and at scale): floating point numbers that possess “value”, which we now call tokens. These tokens are valuable because they encapsulate “intelligence”. People are now taking these tokens and transforming them into English, French, images, videos, chemicals, proteins, robotic movements, etc. And many are working hard to expand the range of concepts and ideas we can create with these tokens. He then goes on to made a compelling comparison between tokens and electricity. In the past industrial revolution, we successfully found a way to convert “atoms” into “electrons” (by boiling water to power electricity turbines). And now, we've discovered a way to convert "electrons" into "tokens" (by using energy to power data centers that train and run LLMs). When electricity was first introduced, few people understood its value. Today, paying for kilowatts is routine. The same will happen with tokens. Right now, only early adopters and builders are paying tokens. Soon, everyone will be paying for tokens on a daily basis to supercharge productivity and power new products and services. Many new industries will be born from and built on top of tokens. When I first heard about this way of thinking, I got goosebumps. Perhaps it’s because Jensen has a good storytelling skill that helps him sell what he builds, but I have to admit this way of thinking gave me a brief surge of pride. I'm proud of humanity's collective effort and creativity, which have taken us from living under rocks to building machines that can "think." That's nothing short but a miracle.

18

5 Comments

Day 13 of #30DaysOfAI 🚀 Getting structured data from Large Language Models (LLMs) has always been a challenge. While LLMs typically return markdown content by default, asking for structured formats like JSON often leads to errors or inconsistencies. That’s why OpenAI’s Structured Outputs is a game-changer! 🎉 With Structured Outputs, you can provide a JSON Schema to define exactly how you want the data to be structured. This ensures the LLM consistently generates the right format every time, making it far more reliable for applications that depend on structured data. Here’s an example: Imagine you’re building an invoice management application. The app scans an invoice, extracts relevant information, and then returns it as a structured JSON format. With Structured Outputs, you can precisely define the type of data to extract—ensuring accuracy and making it easy to store or categorize invoices. Want to learn more? Check out this guide on Structured Outputs by Shreya Khandelwal to see how you can start using this feature effectively. How would you use Structured Outputs in your projects? Let’s discuss this in the comments!

1

Nice to have 'Top New Release' status on Amazon for the latest book (my 25th) that explores an alternative paradigm for quantum computation. Illustrations (111) by Dr. Narendra Madhav Joshi. My take is that we owe it to ourselves at this early stage of the industry's lifecycle to explore alternative approaches. Here is a brief overview: "This book attempts to reveal something of the vastly different and fundamentally creative quality of computing that must accompany any computation involving the quantum-levels. To linearly project digital computing laurels manifest as increasing speeds and the ability to process vaster amounts of information, as the inevitable trajectory of quantum computing is perhaps, in the aphorism attributed to the Buddha, to look only at the finger and to miss the moon and the sky that it is pointing to. The cover figure highlights all that is being missed, summarized as an egg-like structure synonymous with the term ‘Hiranyagarbha’ in Sanskrit. This depicts the womb of creation abundant with many layers and patterns reflecting luminescence, due to the constant superposition and entanglement inherent to the quantum computation that derives from a light-centered interpretation of quantum dynamics." The book is available here: https://lnkd.in/gXu4H9uW #quantumcomputation #quantumtech #technologyleadership #cosmology #systemsthinking

16

5 Comments

You want to do Machine Learning on large scale Marketing solutions ? Check below 🚀

3

$10B to train an SOTA LLM model. I've been asking fellow Product, Engineering and GTM leaders if SOTA LLMs are delivering positive business outcomes. Answer - > without a value chain of other AI models like ETL, RPA, ASR, TTS, w/ lots of fine tuning and non AI pre-post processing, enterprise customers aren't see enough value to pay more. In fact, they want to pay way LESS to get the same jobs done. This reminds me that not much has changed for product and engineering leaders. We still have to listen to the voice of our customers and translate that into a product that has positive unit economics for the companies we work for. Its very expensive to build and maintain AI solutions. Very expensive to run AI solutions. The right question isn't should I leverage AI in my product, rather if I use AI will it improve the net profit margin. $$$

15

2 Comments

LLM Data Engineering Pattern 3: Aggregating Unstructured Insights at Scale Modern financial analysis often requires synthesizing information from a vast array of unstructured sources—regulatory filings (e.g., 10-K, 10-Q), research reports, market commentary, and even audio transcripts of earnings calls. In many cases, these sources must be combined to form a comprehensive view of an industry’s risk profile. Historically, analysts have relied on manual selection and review of documents, a labor-intensive process that does not scale well in today’s data-rich environment. Consider analyzing credit risk in the U.S. commercial real estate sector: 1) Regulatory Filings: An LLM can systematically parse through hundreds of 10-K and 10-Q filings to extract commentary on asset valuations, loan-to-value ratios, or refinancing challenges. 2) Research Reports: It can synthesize qualitative assessments from research houses covering property market trends, default probabilities, or vacancy rates. 3) Earnings Call Transcripts: It can highlight executives’ forward-looking statements regarding rental income stability, interest rate exposures, or occupancy forecasts. By integrating these unstructured data points, an analyst can address questions like: “Given the commentary on sector vulnerabilities in these filings, research reports, and macroeconomic indicators, what are the key risk factors influencing the commercial real estate market?” Financial people, how are you doing this today? #llm #dataengineering #llmdataengineering #financialdata #finance

20

1 Comment

🚀 We've open-sourced TimesFM – a decoder-only foundation model for time-series forecasting 📈 Key Highlights: ✅ Large-Scale Training Data: Trained on a corpus of over 100 billion real-world time points, including Google Trends and Wikipedia pageviews. ✅ Zero-Shot Forecasting: Offers high-quality forecasts even on unseen datasets, without requiring fine-tuning or additional training. ✅ Outstanding Benchmark Performance: Achieved competitive results across a comprehensive suite of benchmarks. 🤗 Explore the model on Hugging Face and GitHub. I'm encouraged by the early feedback and look forward to seeing how #TimesFM will be used in real-world applications for forecasting and anomaly detection. 🙏 A special thanks to the talented research team behind this achievement: Abhimanyu Das Rajat Sen Yichen Zhou Weihao Kong

45

1 Comment

🚀Hi Folks, I'm Taking on the Product #Teardown_Challenge along with Arindam Goswami! 🚀 As part of an exciting challenge by Product Space, I’m stepping into the role of Senior VP of Product at #Zepto Quick Commerce! Zepto’s already crushing it with 10-minute deliveries across 30 cities in India, but there’s so much more potential to unlock! 💡 Here’s the mission: ➡️ Increase Zepto’s Average Order Value (AOV) to ₹600 🛒 ➡️ Boost product discovery and engagement to encourage bigger, more frequent orders 🔍 ➡️ Elevate the purchase experience to keep customers coming back for more 🔄 And I need YOUR help to make it happen! 🙌 🎯 Your insights on quick commerce platforms like Zepto, Blinkit, and Swiggy Instamart are key to shaping these strategies. I’ve put together a short survey, and your feedback will be crucial in tackling this challenge! 💥 Thanks for your support—let’s crack this together! 🙌 Link to the form - https://lnkd.in/g5v2y4wR #ProductTeardown #QuickCommerce #ProductManagement #ProductSpace

10

2 Comments