Norge Vizcay

Interesting to read actual evidence of what one could expect as an inevitability of scale!

21

3 Kommentare

At the AWS Summit Berlin this year, I was asking around everyone one question- "Do you have any LLM application running in production?" The most common response involved hesitation followed by a "No- we have experimented a bunch though". Almost 1.5 years after the chatgpt revolution began, this is the reality as far as industrial adoption of LLMs is concerned. In fact, these are the usage trends: - Most companies have done some form of experimentation and maybe have some prototypes they're struggling to trust- so, nothing in production. - Some companies are actively using some LLM-based applications to help improve their internal processes and workflows (as opposed to exposing LLM results to their end customers). - Still fewer companies have an LLM-based app that their customers interacts with in some way (either directly via a chatbot or so OR by consuming the output of an LLM in some form). - An extremely tiny portion of companies have succeeded in using opensource LLMs in production in addition to the enterprise ones. Despite all the cool developments in the area, the adoption rate as far as productionization of these models go, remains low. I'll ask around the same question at next year's AWS summit and hopefully, will get a lot of yeses ;) #awssummit #llm #ai #opensource #chatgpt

32

3 Kommentare

This is really awesome. I used notebookLM to summarise a recent paper that shows how o1 like model can be reproduced. After providing the pdf from Arxiv, I noticed that the generated podcast was almost 49 minutes long. Very different from the earlier 10-12 minute version. And another impressive feature was the interactive mode where you can interject the two hosts and ask a question which they very graciously respond to. I wanted to share the audio summary but YT didn’t allow me to upload the WAV file. YT expects a video mp4 file. So I found free videos from Pexels and stitched them together with ffmpeg. The videos are meditative acquarium shots of fish swimming around. And with the audio discussing reproducing o1 as an engaging podcast episode, the video keeps the distraction to the minimum. The paper itself is a great read but if you have a 30-40 minute commute, or want to watch some cool fish swimming around and learn about o1, check out

12

False sharing can have a serious impact on performance, especially in compute-bound tasks like matrix multiplication. I recently documented my journey to tackling this challenge and optimizing matrix multiplication in a parallel programming environment. If you’re curious about how false sharing affects performance and how to address it, check out my detailed report here: https://lnkd.in/g64FSrfH

4

I've never met Hartmut Nevem, at least not in this universe. His blog post makes me wonder if we worked on qubits together in another. :) *Willow’s performance on this benchmark is astonishing: It performed a computation in under five minutes that would take one of today’s fastest supercomputers 10^25 or 10 septillion years. If you want to write it out, it’s 10,000,000,000,000,000,000,000,000 years. This mind-boggling number exceeds known timescales in physics and vastly exceeds the age of the universe. It lends credence to the notion that quantum computation occurs in many parallel universes, in line with the idea that we live in a multiverse, a prediction first made by David Deutsch."

13

2 Kommentare

A new week brings a fresh opportunity to learn something new! If you have worked with more complex PyTorch models, especially when implementing LLMs, you may have encountered PyTorch buffers. To help you understand them better, I've recorded a short 13-minute hands-on coding video this weekend. In this video, I explain what PyTorch buffers are, when we need them, and how to use them effectively: https://lnkd.in/giAVBGAy

711

21 Kommentare

[research update] use a Transformer's multi-head attention activations to fetch RAG style, to augment multi faceted problem breakdown for LLMs in ."Multi-Head RAG: Solving Multi-Aspect Problems with LLMs": 1. Enhances LLMs by integrating document retrieval (RAG) into the LLM context. 2. Challenge: Existing RAG solutions struggle with queries needing multiple documents with varied contents due to distant embeddings. 3. Multi-Head RAG (MRAG): Introduces a novel approach using Transformer's multi-head attention activations to fetch diverse documents. 4. Activation Utilization: Different attention heads capture varied data aspects, improving complex query retrieval. 5. Improved Relevance: MRAG shows up to 20% improvement in relevance over standard RAG baselines. 6. Evaluation Methodology: Provides metrics, synthetic datasets, and real-world use cases to demonstrate effectiveness. 7. Seamless Integration: MRAG can be integrated with existing RAG frameworks and benchmarking tools like RAGAS. 8. Multi-Aspect Problem Solving: Designed specifically to handle multi-aspect queries more effectively. 9. Data Stores Compatibility: Compatible with various data store classes. 10. Potential Applications: Benefits scenarios requiring diverse information retrieval, enhancing LLM responses. Experiment The paper introduces MRAG and evaluates it using synthetic datasets and real-world data. It compares MRAG's performance against standard RAG models. The experiment focuses on the model's ability to retrieve relevant documents across various query aspects, demonstrating up to a 20% improvement in relevance over baselines. Use Case MRAG is tested on real-world scenarios like customer support, where queries often require information from multiple documents with diverse contents. The model's capability to retrieve and integrate multi-aspect data enhances the quality of responses.

81

5 Kommentare

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Kommentare

SWE-Fixer: open-source LLMs for effective and efficient Github Issues Resolution. One of key LLMs applications for software engineering challenges, has emerged in resolving real-world tasks on GitHub by fixing code based on issues reported by users. Most of current approaches in that direction, rely on proprietary LLMs, which limits reproducibility, accessibility, and transparency. To address these challenges, this paper introduce SWE-Fixer, a novel open-source LLM designed to effectively and efficiently resolve GitHub issues. 𝗞𝗲𝘆 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻𝘀: - propose a novel pipeline-based method that leverages open-source models, achieving state-of-the-art Best@1 performance on SWE-Bench Lite and Verified benchamarks - curate a large-scale dataset comprising 110K instances with rigorous filtering techniques to ensure both extensiveness and reliability of training data 𝗠𝗼𝗱𝘂𝗹𝗲𝘀 i) Code File Retrieval - adopted a coarse-to-fine strategy, to identify relevant files for modification in a repository - file documentations (skeletons) were used as input, which includes module docstrings, class headers, class methods, and function signatures - First, BM25 is used to retrieve the 30 most relevant files to the issue, treating issue description as a query and code files as documents.  - Next, we finetune a LLM to further pinpoint which files require modification from BM25-retrieved files. ii) Code Editing - involves generating a patch to resolve an issue based on relevant files - For training, used gold defective files, i.e., those within the patches, and retrieved files for inference - For better understanding issue, content of all retrieved files are provided as context - For generating valid code modifications, defined a structured output that includes three key components: file path, original code block and modified code block 𝗠𝗼𝗱𝗲𝗹 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 - adopted a structured approach, JsonTuning, with input JSON structure includes task input elements, task instructions, and output control information, while output JSON structure consists of task output elements. - for data collection, used Github REST API to crawl python repos with more than 100 pull requests - Chain-of-Thought Data Construction, with usage of gold patches as part of input to guide the teacher model in generating both the reasoning process and the corresponding patches 𝗦𝗲𝘁𝘂𝗽 - finetuned Qwen2.5 base series models to implement SWE-Fixer, which consists of a 7B code retriever and a 72B code editor 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - On SWE-Bench Lite and Verified benchmarks, SWE-Fixer achieves state-of-the-art performance among open-source models with scores of 23.3% and 30.2%, respectively. - SWE-Fixer framework achieves highest Best@1 performance. - SWE-Fixer outperforms several methods based on proprietary models, including those leveraging GPT-4, GPT-4o, and Claude-3-Opus. 𝗕𝗹𝗼𝗴: https://lnkd.in/evARHnWz 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eCet2y6X 𝗖𝗼𝗱𝗲: https://lnkd.in/ePnCykNF

27

VARIANCE AND STANDARD DEVIATION OF LOG RETURNS This paper explains why the classical (arithmetic) definitions of variance and standard deviation should not be used with log returns. Instead, we should use the geometric variance and geometric standard deviation, defined as the exponentials of the arithmetic variance and arithmetic standard deviation. This is a work in progress... https://lnkd.in/grxBec4W

2

There is a new LLM & RAG framework in town that you must know about, as it might replace LangChain and LlamaIndex ↓ The new RAG framework targets to be the PyTorch library for LLM applications. It prioritizes: - simplicity - modularity - robustness - a readable codebase ...over out-of-the-box one-liners that are hard to understand and extend. ...which we often encounter when working with LangChain and LlamaIndex. . 𝗦𝗼, 𝘄𝗵𝗼 𝗶𝘀 𝘁𝗵𝗲 𝗻𝗲𝘄 𝗸𝗶𝗱 𝗶𝗻 𝘁𝗼𝘄𝗻? → 𝘓𝘪𝘨𝘩𝘵𝘙𝘈𝘎 It was initiated by Li Yin, who started implementing LLM solutions before they were cool and realized how limited one is when building custom solutions. She highlighted that each use case is unique in its data, business logic, and user experience. Thus, no library can provide out-of-the-box solutions. Ultimately, LightRAG aims to provide a robust and clean codebase you can 100% trust, understand and extend. With the ultimate goal of quickly customizing your own LLM, RAG, and agent solutions. . If you are into LLMs, RAG and agents, consider checking it out on GitHub and support it with a ⭐️ (or even contribute) ↓ 🔗 𝘓𝘪𝘨𝘩𝘵𝘙𝘈𝘎: https://lnkd.in/d-fcDZ3A #machinelearning #mlops #datascience . 💡 Follow me for daily content on production ML and MLOps engineering.

391

29 Kommentare

OpenCoder doesn't get enough love They open-sourced the entire pipeline to create QwenCoder-level code models. This includes: - Large datasets - High-quality models - Eval framework Tons of great lessons and observations in the paper 📝 Paper: arxiv.org/abs/2411.04905 #special #linkedin #Family #Pakistan #skills #islam #government #AI #ML #MachineLearning #tesla #microsoft #openai

5

How to optimize Ollama for CPU-only inference? I am wondering if anyone has documentation, blog posts, ideas, ... really anything(!) about optimizing inference for CPU-only. The reason is that most people do not have nice GPU(s) in their notebooks and rely on API providers or remote hardware. Would be great to give those people a good hint on which model they could use locally. We are currently on a good way of doing a massive benchmark about CPU-only inference so i appreciate your input greatly!

1

4 Kommentare

#RAG with advanced retrieval technique RAPTOR for long contexts. When working with long-context documents, we cannot just chunk the documents and embed them. Instead we would want to have a good approach for minimalist document splitting for long context LLMs. This is where RAPTOR comes into picture. Recursive Abstractive Processing for Tree Organized Retrieval [RAPTOR] is a new and powerful indexing and retrieving technique for LLM in a comprehensive manner. It adapts a bottom-up approach by clustering and summarizing text segments(chunks) to form a hierarchical tree structure. → The leaves are a set of starting documents. → Leafs are embedded and clustered. → Clusters are then summarized into higher level (more abstract) consolidations of information across similar documents. → This process is done recursively, resulting in a “tree” going from raw docs (leaves) to more abstract summaries. We can apply this at varying scales; leaves can be: → Text chunks from a single doc → Full docs With longer context LLMs, it’s possible to perform this over full documents. This tree structure is key to RAPTOR function as it captures bot high level and detailed aspects of text, which is particularly useful for complex thematic queries and multi-step reasoning in questioning and answering tasks. This process involves segmenting documents into shorter texts called chunks and then embedding the chunks using an embeding model. These embeddings are then clustered by a clustering algorithm. Once clusters are created, the text associated with each cluster is summarized by using a LLM. The summaries generated form nodes in a tree with higher level nodes providing more abstract summaries. Know more about RAPTOR in this research paper: https://lnkd.in/gDgZP6Vj Also, read this article by Plaban Nayak: https://lnkd.in/g-7HpADg ----------------------------------------------------------------- BTW, you also need a robust database in RAG. I recommend using SingleStore Database. Sign up and get $600 worth free credits. Try SingleStore database for free: https://lnkd.in/gCAbwtTC

229

1 Kommentar

Seems parquet goes like AI embeddings encoding, hence compression!

1

🚨 𝐘𝐨𝐮 𝐚𝐫𝐞 𝐬𝐭𝐫𝐮𝐠𝐠𝐥𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐲𝐨𝐮𝐫 𝐑𝐀𝐆 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧? 𝐘𝐨𝐮𝐫 𝐑𝐀𝐆 𝐀𝐏𝐏 𝐢𝐬 𝐧𝐨𝐭 𝐚𝐜𝐜𝐮𝐫𝐚𝐭𝐞 𝐞𝐧𝐨𝐮𝐠𝐡 𝐚𝐧𝐝 𝐒𝐥𝐨𝐰? 🎯🐌 𝐕𝐞𝐜𝐭𝐨𝐫-𝐑𝐀𝐆 (Retrieval-Augmented Generation ) and now increasingly 𝐆𝐫𝐚𝐩𝐡-𝐑𝐀𝐆 has been instrumental in grounding LLM's with external knowledge. 📚 However, challenges such as retrieval latency, errors, and system complexity persist. 😰 𝐂𝐚𝐜𝐡𝐞-𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐞𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 (CAG) offers a promising solution by preloading and precomputing relevant resources into the model’s context window, thereby eliminating real-time retrieval during inference. 🥇 𝐀𝐝𝐯𝐚𝐧𝐭𝐚𝐠𝐞𝐬 𝐨𝐟 𝐂𝐚𝐜𝐡𝐞-𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐞𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 1️⃣ 𝐑𝐞𝐝𝐮𝐜𝐞𝐝 𝐋𝐚𝐭𝐞𝐧𝐜𝐲: By eliminating real-time retrieval, CAG enables faster inference. 2️⃣ 𝐈𝐦𝐩𝐫𝐨𝐯𝐞𝐝 𝐑𝐞𝐥𝐢𝐚𝐛𝐢𝐥𝐢𝐭𝐲: Minimises retrieval errors while maintaining context relevance. 3️⃣ 𝐒𝐢𝐦𝐩𝐥𝐢𝐟𝐢𝐞𝐝 𝐃𝐞𝐬𝐢𝐠𝐧: Provides a streamlined, retrieval-free alternative to RAG, achieving comparable or superior results with lower complexity. 🛑 𝐋𝐢𝐦𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬 𝐨𝐟 𝐂𝐚𝐜𝐡𝐞 𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐞𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 1️⃣ 𝐋𝐢𝐦𝐢𝐭𝐞𝐝 𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐒𝐢𝐳𝐞: Requires the entire knowledge source to fit within the context window, making it less suitable for tasks involving extremely large datasets. 2️⃣ 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐋𝐞𝐧𝐠𝐭𝐡 𝐂𝐨𝐧𝐬𝐭𝐫𝐚𝐢𝐧𝐭𝐬: Performance may degrade with very long contexts. 𝐀𝐝𝐯𝐚𝐧𝐜𝐞𝐦𝐞𝐧𝐭𝐬 in LLM's with extended context windows are rapidly increasing, enhancing CAG’s applicability. ⚙️ For a deeper dive into this topic, check out the recent paper (link in first comment) of the National Chengchi University IMBA. Credits to Maryam Miradi, PhD to bring my Attention to this method. 👏🏼 #AI #MachineLearning #NaturalLanguageProcessing #Innovation #TechTrends #cacheaugmentedgeneration

12

1 Kommentar

How I Squeeze 60K RPS out of SQLite on a $5 VPS: There are tons of tutorials on how to set up SQLite, and sometimes they contradict each other. Here's what worked for me: 1. Configuring PRAGMAs: We need to send the following PRAGMA commands right after opening the connection: PRAGMA journal_mode = WAL; - Enables write-ahead log so that your reads do not block writes and vice-versa. PRAGMA busy_timeout = 5000; - SQLite will wait 5 seconds to obtain a lock before returning SQLITE_BUSY errors, which will significantly reduce them. PRAGMA synchronous = NORMAL; - SQLite will sync less frequently and be more performant, still safe to use because of the enabled WAL mode. PRAGMA cache_size = -20000; - Negative number means kilobytes, in this case 20MB of memory for cache. PRAGMA foreign_keys = true; - Because of historical reasons, foreign keys are disabled by default; we should manually enable them. PRAGMA temp_store = memory; - Moves temporary tables from disk into RAM, speeding up performance a lot. Do NOT use cache=shared! Some tutorials recommend configuring it, but this is how you get nasty SQLITE_BUSY errors. It is disabled by default, so you don't have to do anything extra. 2. Use immediate transactions: If you know that a transaction can possibly do a write, always use BEGIN IMMEDIATE or you can get a SQLITE_BUSY error. Check your framework; you should be able to set this at the connection level. 3. Open two connection pools: Another trick is to open two connection pools: one for reads only and another for reads/writes. Set the connection limit of the write pool to 1, and the connection limit of the read pool to some reasonably high number, e.g., the number of your CPU cores. Hope this helps!

13

1 Kommentar