Zichu L.

A very useful checklist to follow when you have to put in production Transformer based models

5

The cost of training top models is increasing 2-3X per year, and at this rate will reach $1 billion per model by 2027. The findings are from Epoch AI and follow their recent analysis of increases in compute for foundation models running at 4-5X per year. They've also provided some interesting data on the relative breakdown of cost components like staff, hardware, and energy. A few thoughts after reading this: - I would assume staff costs won't increase as much as hardware and energy costs, so the relative contribution of human labor to future AI models will decrease, in yet another sign of capital's increasing dominance over labor in the industry (just look at NVIDIA's ever-rising stock price). - If compute is increasing 4-5X per year but cost is increasing 2-3X, is it safe to assume that training runs are getting more efficient? - Some people, like Anthropic CEO Dario Amodei, have said we'll see $1 billion training runs in the next year. Does this mean the 2-3X pace is about to increase? That Epoch is wrong in its estimate? Or that Amodei is overstating the cost, possibly to put fear into competitors who can't keep pace? - Only a few companies are going to be able to train future models. We're likely going to see further consolidation. Perhaps we get to a future with only two large general purpose models, like the Windows/MacOS and Android/iOS duopolies we've seen in the past. Article here: https://lnkd.in/g5SejXen

7

Unlock the full potential of your Large Language Models (LLMs) with Intel® Extension for PyTorch (IPEX) and the Intel® LLM Library for PyTorch (IPEX-LLM). Download this whitepaper to explore the optimization of LLMs, including their performance, resource utilization, and response times in real-world applications. Link - https://intel.ly/3BBJ4ey

2

As machine learning models grow in size, so too does their carbon footprint. As AI scales, it's important that we quantify and mitigate these emissions. In our new short course "Carbon Aware Computing for GenAI Developers" you'll learn from Google Cloud Developer Advocate Nikita Namjoshi how to: - Query the ElectricityMaps API for real-time data on regional electricity grid carbon intensity - Route your model training jobs to data centers in regions primarily powered by low-carbon sources like wind, solar, hydro and nuclear - Measure the carbon footprint of your ML training, inference, storage and API usage with the Google Cloud's Carbon Footprint tool - Optimize training job scheduling to run when clean energy is most abundant in a given region. You can sign up here: https://lnkd.in/gqXnMJzX

5,149

172 Comments

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

180

2 Comments

🔍 Enhancing NLP with Retrieval-Augmented Models: The Rise of RAG and RAU In the realm of Natural Language Processing (NLP), integrating external data to refine outputs is reshaping how language models perform. Researchers from East China University of Science and Technology and Peking University have detailed advancements in Retrieval-Augmented Language Models (RALMs), specifically focusing on Retrieval-Augmented Generation (RAG) and Retrieval-Augmented Understanding (RAU). These frameworks are pivotal in addressing the limitations of traditional Large Language Models (LLMs) like hallucinations and the need for domain-specific knowledge. RALMs enhance NLP tasks by pulling relevant external information to support the language model's output, offering solutions in various interaction modes. For instance, sequential interactions involve retrievers identifying relevant documents to inform the model’s predictions, enhancing accuracy. Parallel interactions allow retrievers and language models to operate independently, synchronizing their outputs to optimize results. The versatility of RALMs is supported by diverse retrieval methods—sparse, dense, internet, and hybrid—each tailored to maximize performance across different NLP tasks. Sparse methods use straightforward techniques such as TF-IDF, while dense retrieval employs deep learning to improve accuracy. Hybrid retrieval combines these methods to leverage their strengths comprehensively. The practical applications of RAG and RAU are vast, spanning machine translation, dialogue generation, and text summarization. In dialogue generation, for example, RALMs produce contextually relevant responses in multi-turn interactions, significantly enhancing user engagement and system responsiveness. In summary, the development of RAG and RAU marks a significant step forward in NLP technology, merging external data retrieval with powerful language processing to overcome traditional LLM challenges. This integration not only enhances model performance but also broadens the scope of NLP applications, making it a critical area for future research and development. 👉 Explore the full paper below or download link in the comments below. If you find it valuable, please like and share this post – your engagement shapes the content I share. Also, follow me if you haven't already. Thanks! #NLP #AI #MachineLearning #DataRetrieval #LanguageModels #RAG #RAU

2

2 Comments

Understanding causality is a formidable challenge for Large Language Models (LLMs) and is considered a pivotal benchmark for assessing their proximity to human cognitive abilities. Despite research efforts employing prompt-based training on causal relationships, achieving satisfactory results remains elusive. The absence of a standardized causal dataset for benchmarking exacerbates this issue. In light of this, a recent study proposes a solution in the form of CausalBench, a comprehensive benchmark designed to evaluate LLMs' understanding of causality. This benchmark addresses the limitations of existing evaluation methodologies, offering a diverse set of tasks derived from the causal research community. By encompassing three distinct causal learning tasks, CausalBench enables a thorough comparison of LLM performance against traditional causal learning algorithms. https://lnkd.in/dvV4B7FS

3

I’ve been fortunate enough to do AI shows with Sam Altman, Yann LeCun & Arthur Mensch. But this show today with David Cahn at Sequoia Capital is the best of all of them: 🧠 AI’s $600BN question 🏃 Why no one will never train a frontier model on the same data center twice 🏗️ Servers, steel & power: The 3 pillars of the future of AI Not one to be missed, my 8 key takeaways 👇 1. The Oligopoly That is Blowing Billions on AI - Microsoft, Azure, & Google now represent $7T market cap. - Of course they will spend billions to protect their oligopoly - What is most interesting is Meta is the only incumbent that is financing this not with cloud being their cash cow. 2. How Startups Could Win with the Cost of Compute Coming Down - Big tech companies are producers of compute & startups are consumers of compute. - Overproduction of compute = lower prices.  - This gives startups higher margins & makes them more valuable. 3. Is Compute the Currency of the Future or Do the Costs Come Down and It Is Commoditised? - We are forgetting “compute” is a euphemism for a $2BN data center in the middle of nowhere. - They have GPUs & liquid cooling systems that need to be constantly optimized & upgraded. - We cannot simply build 15 years worth of compute all at once.  No one's ever going to train a frontier model on the same data center twice. 4. Data Centre Teams and Model Teams Need to Be Coupled - You cannot have separate teams running data centers & building models. - Elon & Zuck control all their data centres. - It will not work as these models get better & better. 5. Why the Data Center Is the Most Important Asset - There is not much difference between these models today. - AI researchers are jumping from one lab to another. - Scaling laws will become more dominant as these models get bigger. 6. Servers Steel and Power: The Three Bottlenecks of AI Three things: 1. Servers: NVIDIA, AMD, Broadcom… 2. Steel: Construction & real estate developers for data centers. 3. Power: All the energy required for AI. - More factories will be built in 12 months for AI and we will see another industrial revolution. 7. You Have to Have a Cash Machine to Play the Big Model Game - Amazon has AWS - Meta has Instagram - Microsoft has Azure - You need a cash machine to compete and it CANNOT be the AI business. 8. There is One Definition of Success in Venture: Are you a Slugger? - If you are generating billion dollar gains, you are a slugger. - Doug Leone, Pat Grady, Andrew Reed are all sluggers. - You cannot be good at this business without being a slugger. (links in comments) #founder #funding #business #investing #vc #venturecapital #entrepreneur #startup #seed #funding #sequoia

190

26 Comments

Nobody calls in a SWAT team to change a light bulb. This overkill approach doesn't work in everyday life, and it won't be the way forward for AI either. On one side, you have players like OpenAI who believe in going all-in on gigantic models. Their explicit goal is AGI, and they're willing to spend seemingly without limit to get there. But Snowflake, Databricks, and others in the "small-but-mighty" camp envision a different future. They believe the key to unlocking AI's potential isn't solely about scale. Instead, it's about combining multiple, specialized models into compound AI systems that can handle complex tasks from start to finish. Rather than forcing one huge model to learn everything, it's about teaching a team of smaller models to work together efficiently. More in my latest newsletter:

74

9 Comments

Could RAFT (Retrieval-Augmented Fine-Tuning) be the next evolution of RAG architecture? I am following this with great interest. https://lnkd.in/eHDx3rHz

13

Cerebras Introduces CePO (Cerebras Planning and Optimization): An AI Framework that Adds Sophisticated Reasoning Capabilities to the Llama Family of Models Cerebras has introduced CePO (Cerebras Planning and Optimization), an AI framework designed to enhance the reasoning and planning capabilities of the Llama family of models. CePO integrates optimization algorithms with Llama’s language modeling capabilities, enabling it to address complex reasoning tasks that previously required multiple tools CePO’s core innovation lies in embedding planning capabilities directly into the Llama models. This eliminates the need for external optimization engines, allowing the models to reason through multi-step problems, manage trade-offs, and make decisions autonomously. These features make CePO suitable for applications in logistics, healthcare planning, and autonomous systems where precision and adaptability are essential CePO enhances Llama models with a specialized planning and reasoning layer. This layer employs reinforcement learning and advanced constraint-solving techniques to facilitate long-term decision-making. Unlike traditional AI systems, which often require predefined rules or domain-specific training data, CePO generalizes its optimization strategies across various tasks. A key technical feature of CePO is its integration of neural-symbolic methods. By combining neural network learning with symbolic reasoning, CePO achieves both adaptability and interpretability. It also includes a dynamic memory module that enables it to respond effectively to evolving scenarios, improving performance in real-time planning tasks..... Read the full article here: https://lnkd.in/gjE8uikP Check out more details: https://lnkd.in/gv66CTvM Cerebras Systems Emma Call Connor Anderson Jessica Liu Ryan Humble Ganesh Venkatesh Daria Soboleva Natalia Vassilieva Angela Yeung

74

Check out my new tutorial on how to create a multimodal database! You get to use text- and image- embedding models in addition to Llama 3.2 Vision for image captioning. Decided to try out Groq, Hugging Face, LanceDB, and Hex after meeting them in various events here in the bay area. Cool Products! #ai #multimodal #llama #vision

42

🚀 Our vision at Nixtla is to become the go-to solution for time series and empower our great community. We also believe that practitioners should easily be able to test and benchmark new ideas. 🙏 This release of NeuralForecast is particularly exciting because it is the first (to my knowledge) open-source implementation of KANS for TimeSeries. Kolmogorov-Arnold Networks (KANs) are an interesting alternative to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights").  🤔 Should you start using KANs now for all your forecasting needs? Probably not, but they are worth exploring. 🔥 We are also extremely thankful for the wonderful contributions of the community. A particular shout-out to Jasmine Rienecker and Dr. Fabian Bergermann.

63

5 Comments

The ongoing debate surrounding the use of long-context LLMs versus RAG centers on their respective advantages and limitations. Google has recently released a paper around hybrid approach. One notable finding from the study is that 60% of queries result in identical predictions when processed by both RAG and long-context LLMs. Self-route seems to be a promising hybrid architecture to get benefit of both worlds. #LLMs #llm #rag #genai #AI

25

I wrote a blog post on a thorny but underappreciated problem in ML-based marketing personalization: event attribution. It's how you go from the actions you take and customer behavior you observe to creating training experiences for your model. Good attribution makes a huge difference in driving uplift, there are a lot of degrees of freedom in configuring it, and at the same time, it's rarely discussed in ML/bandits/RL literature.

47

4 Comments