Navaid Abidi

The cost of training top models is increasing 2-3X per year, and at this rate will reach $1 billion per model by 2027. The findings are from Epoch AI and follow their recent analysis of increases in compute for foundation models running at 4-5X per year. They've also provided some interesting data on the relative breakdown of cost components like staff, hardware, and energy. A few thoughts after reading this: - I would assume staff costs won't increase as much as hardware and energy costs, so the relative contribution of human labor to future AI models will decrease, in yet another sign of capital's increasing dominance over labor in the industry (just look at NVIDIA's ever-rising stock price). - If compute is increasing 4-5X per year but cost is increasing 2-3X, is it safe to assume that training runs are getting more efficient? - Some people, like Anthropic CEO Dario Amodei, have said we'll see $1 billion training runs in the next year. Does this mean the 2-3X pace is about to increase? That Epoch is wrong in its estimate? Or that Amodei is overstating the cost, possibly to put fear into competitors who can't keep pace? - Only a few companies are going to be able to train future models. We're likely going to see further consolidation. Perhaps we get to a future with only two large general purpose models, like the Windows/MacOS and Android/iOS duopolies we've seen in the past. Article here: https://lnkd.in/g5SejXen

7

I’ve been fortunate enough to do AI shows with Sam Altman, Yann LeCun & Arthur Mensch. But this show today with David Cahn at Sequoia Capital is the best of all of them: 🧠 AI’s $600BN question 🏃 Why no one will never train a frontier model on the same data center twice 🏗️ Servers, steel & power: The 3 pillars of the future of AI Not one to be missed, my 8 key takeaways 👇 1. The Oligopoly That is Blowing Billions on AI - Microsoft, Azure, & Google now represent $7T market cap. - Of course they will spend billions to protect their oligopoly - What is most interesting is Meta is the only incumbent that is financing this not with cloud being their cash cow. 2. How Startups Could Win with the Cost of Compute Coming Down - Big tech companies are producers of compute & startups are consumers of compute. - Overproduction of compute = lower prices.  - This gives startups higher margins & makes them more valuable. 3. Is Compute the Currency of the Future or Do the Costs Come Down and It Is Commoditised? - We are forgetting “compute” is a euphemism for a $2BN data center in the middle of nowhere. - They have GPUs & liquid cooling systems that need to be constantly optimized & upgraded. - We cannot simply build 15 years worth of compute all at once.  No one's ever going to train a frontier model on the same data center twice. 4. Data Centre Teams and Model Teams Need to Be Coupled - You cannot have separate teams running data centers & building models. - Elon & Zuck control all their data centres. - It will not work as these models get better & better. 5. Why the Data Center Is the Most Important Asset - There is not much difference between these models today. - AI researchers are jumping from one lab to another. - Scaling laws will become more dominant as these models get bigger. 6. Servers Steel and Power: The Three Bottlenecks of AI Three things: 1. Servers: NVIDIA, AMD, Broadcom… 2. Steel: Construction & real estate developers for data centers. 3. Power: All the energy required for AI. - More factories will be built in 12 months for AI and we will see another industrial revolution. 7. You Have to Have a Cash Machine to Play the Big Model Game - Amazon has AWS - Meta has Instagram - Microsoft has Azure - You need a cash machine to compete and it CANNOT be the AI business. 8. There is One Definition of Success in Venture: Are you a Slugger? - If you are generating billion dollar gains, you are a slugger. - Doug Leone, Pat Grady, Andrew Reed are all sluggers. - You cannot be good at this business without being a slugger. (links in comments) #founder #funding #business #investing #vc #venturecapital #entrepreneur #startup #seed #funding #sequoia

190

26 Comments

This article is concise and meaty… but a good one to dig into. Thanks Eryk Salvaggio “… if one side of the communication does not have meaning, then the comprehension of the implicit meaning is an illusion arising from our singular human understanding of language (independent of the model). Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.” There is a strange obsession with disproving the parrot claim among LLM evangelists. I suspect it largely comes from a misunderstanding of the word “stochastic” to mean “random” or “unguided,” when, in fact, stochastic is precisely the difference between random and guided output. It doesn’t help that “stochastic” and “random” are used in mathematics to describe the same thing. It’s just that stochastic implies the presence of a mathematical model for sorting through the randomness. Stochasticity refers to the tension between the randomness of language in a dataset and the probabilistic constraint that an LLM must navigate to generate plausible-sounding language. Altman suggests that by calling an additional layer of prompt negotiation “reason,” his models transcend these limits.

2

Thanks for sharing these insights. What I am seeing is that orgs test an idea using a proprietary #LLM like #OpenAI, then put it into production with an open source LLM using a development platform like #Llamaindex. That way, private data remains private.

1

LLMs are trained on internet-scale data. But to make AI work for work, models need to be grounded in your enterprise data -- a much harder problem than meets the eye! How do you connect to all your data sources and ingest them frequently enough while managing compute/storage costs? How do you make sure that you only show users results from documents they have access to? How do you rank results without internet-scale click and link data? How do you process the content of documents that are often hidden in PDFs and presentations? These are tricky problems, and it's why enterprise search didn't work for nearly a decade, until Glean came along. Turns out it's the exact same set of challenges needed to give LLMs the right context to work their magic on your enterprise content. Glean spent 5 years building the ingestion, search, and ranking infra needed to solve this problem well. This has given them the right to run the "Work AI" race and delight users at work. CEO Arvind Jain shares more on the latest episode of Training Data. Listen here, or wherever you get your podcasts: YouTube: https://seq.vc/ytd Apple: https://seq.vc/pit Spotify: https://seq.vc/jnz 

203

10 Comments

While being futuristic, Quantization of a Neural Network model is the key to reduce dependency on high end hardware. This one is a great explanation of mode quantization techniques that can save millions of 💵 trying to implement LLM #LLM #ConversationalAI #GenAI #Chatbots #contactcenter

6

$2 H100s: How the GPU Bubble Burst Last year, H100s were $8/hr if you could get them. Today, there's 7 different resale markets selling them under $2. What happened? The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. https://lnkd.in/gpQEidmC

2

1 Comment

As we close out #2024, it’s incredible to look back at some of the greatest #GenerativeAI posts shared here by the top GenAi Experts. 🚀 Here are the top 9 posts from some of the brightest minds in the field: Sebastian Raschka, PhD Chip Huyen Tom Yeh Pascal Biese Denis Rothman Peter Gostev Pau Chang Liang Andrej Karpathy These experts have not only expanded our understanding of what’s possible with Generative AI but have also inspired countless innovations in the space. 🔗 Dive into their posts for some great year-end reading, and share your thoughts on which ones inspired you the most! #GenerativeAI #AIThoughtLeaders #2024Recap #ArtificialIntelligence #AIInnovation

17

AI Reading for Wednesday June 5! https://lnkd.in/eB-T57wZ Stuff to expect from WWDC next week, including possibly opt-in AI features. - 9to5Mac Gurman: Apple got better terms and better tech from OpenAI than Google was offering, unclear if it will be a long-term alliance or a stopgap. - Bloomberg iOS 18 ChatGPT integration to be opt-in, with Apple’s own chatbot maybe coming in not too-distant future. - BGR Huang: AI saves more energy than it burns! - pcgamer Well, he's not totally wrong either. You can put data centers almost anywhere, if you put them where there is cheap renewable energy it's OK from a climate standpoint, AI can reduce physical transportation and increase energy efficiency in various ways. Zoom CEO wants you to send your AI clone to meetings while you frolic on the beach. Then after you’re laid off, your AI clone will consume less energy than you do living under a bridge! - The Verge Big opportunity seen in 'AI co-workers'. - The Register Asana testing a new “AI teammate” who sounds kind of bossy. - The Verge How Jamie Dimon used AI to write his annual letter, and prepare to meet Elon Musk. - GeekWire MIT chatbot lets you talk to your future self. - TechXplore AI researcher thinks there's a 99.9% chance AI wipes us out. - Business Insider AI researchers publish open letter calling for AI companies to allow healthy discussion of AI risks. - Ars Technica Will A.I. Be a Creator or a Destroyer of Worlds? - NY Times Andreessen and Khosla battle over "whether AI’s future will be one of concentrated safety, or of unfettered advancement." - Forbes Microsoft makes significant Azure layoffs, blames AI. - Business Insider Yellen says AI has moved toward the top of regulators' agenda of risks to monitor. - CNN When Google's AI bumps your content down the search results page, but aggregates it in the AI overview. - WIRED AI hall of mirrors as Perplexity AI-generated pages turn up in Google AI Overviews. - Search Engine Land Does ChatGPT doom sell-side analysts? - FT AI to detect emotions in human speech found effective for dog barks. - Neuroscience News How A.I. Has Changed Music, and What’s Coming Next - NY Times Stanford AI model found to have been plagiarized from Chinese research. - Pandaily Making cartoon character voices with the upcoming OpenAI voice assistant. (now 'in coming weeks') - YouTube Shareholders sue Musk. - CNN FT profiles CoreWeave - FT Pika raises $80m for AI text-to-video. - Washington Post Firefox will use AI to improve accessibility, e.g. generating missing alt tags for the visually impaired. - Neowin Shoplifting detection. A few months old but going viral again. - YouTube

2

1 Comment

The Economist's article “Large language models are getting bigger and better” on April 17, 2024 on the challenges of scaling large language models (LLMs) rightly identifies the limitations of current AI architectures. While the relentless pursuit of larger LLMs suggests a future of limitless progress, a deeper dive reveals significant flaws. The "scaling hypothesis"—that more data and power equals human-like intelligence—misses crucial aspects. LLMs face serious hurdles: the dwindling supply of high-quality data, the inefficiency of scaling, and the simple fact that size doesn't equate to cognitive complexity. Here, neurosymbolic AI presents a promising alternative, blending the data-handling prowess of neural networks with the logical precision of symbolic AI. This combination not only promises enhanced efficiency and structured reasoning but also aligns closer with human cognitive processes. Neurosymbolic AI excels in areas where traditional models falter, offering a path to more sustainable, intelligent systems capable of real-world applications like airplane design and financial risk assessment. It’s time for the AI community to pivot towards technologies that not only scale but think smarter. https://loom.ly/96rDCD0

21

4 Comments

Welcome to the era of data feudalism. IQT (In-Q-Tel) believes that the "Bigger is Better Approaches" in #GenAI is creating an inexhaustible appetite for users’ data, leading to a rise in user data expropriation, sectioning off of the internet, and “data feudalism.” My colleague Dan Geer discusses this issue and its implications at length in the below article. Give it a read....

35

1 Comment

At a Generative AI thought leadership dinner with other CTOs in NYC today and some interesting trends are emerging: * a consensus that LLMs are converging on a general set of features and basically comparable performance * a consensus that an org's data pipelines will matter more than the LLM chosen * and a group of people who read the TOS that are really scares about what these LLMs will do with access to all your data. Lastly, my prediction: the work of tech is moving from the keyboard to the whiteboard. Ai as microservice means structuring your data, thinking with ai, and chaining ai agents will supercharge your career and your cause -- but only if social change orgs platform their data. In my study of polymaths, Charles Babbage designed the basic idea for a unit of computing being something that takes an input, does something to it and gives an output -- the so called Babbage difference engine. In a real way, GenAI is approaching a world where chaining inputs ala a babbage difference engine is accessible to all. This can be an awesome opportunity if we get Non profits to work together!

11

3 Comments

Great stuff from the Sequoia Capital folks: A few ideas stand out to me: Generative AI is evolving from rapid pattern-matching (“System 1”) to deliberate reasoning at inference time (“System 2”). By “stopping to think,” AI models can now perform complex reasoning tasks, similar to how AlphaGo uses simulations to plan moves. Expect to see more models like o mini appear AI companies is shifting from selling software licenses to delivering outcomes, moving from “Software as a Service” to “Service as a Software.” SAAS was barely paying off unless you are the big 4. Harvey, an AI lawyer is a great example. Instead of law firms buying legal software licenses, Harvey’s AI provides legal services such as drafting documents or conducting research. Clients pay for the legal work completed by the AI, focusing on the service delivered rather than the software itself. For product managers and developers, the opportunity lies in creating AI native applications with specialized reasoning to solve real-world problems. Again stop trying to build the super AI app and focus on niche use cases and go DEEP! It doesn’t need to do everything, just do one thing exceptionally better than everyone else. https://lnkd.in/eddpW3bj

2

Gemini 2.0 handles complex math and planning tasks previously out of reach of LLMs. You can turn on "code generation" in AI Studio and in the APIs. For context, let's talk about function calling and voice AI agents. (I spend a lot of time working on voice AI.) Voice AI agents use function calls for: - Interacting with external systems specific to particular use cases and deployment contexts. - A wide range of RAG-like information lookup tasks. - Conversation persistence, including complex actions like saving a privacy-redacted version of a conversation. - Script following. Some of these uses were obvious before we had built and deployed production voice AI systems. But some of them weren't. (Script following, for example.) We had to build experiments, and then production systems, to figure out what features and tooling these new systems really need. I'm convinced that code generation will be as important as function calling, once we've accumulated some experience building with Gemini 2.0 and similarly capable new LLM versions. Credit for this insight goes entirely to Shrestha Basu Mallick, who shepherds work at Google on the Gemini APIs and AI Studio. When Shrestha first told me that really good code generation was going to make a lot of new things possible, I didn't really get it. But after playing with code generation for a few weeks ... I still don't fully get it, but in a good way, now. It's clear that there are some obvious things to explore: - Context-aware math on demand driven by natural language. - Generative UI. - Better "binding" behavior with function calling. But I'm also convinced now that there is a lot of unexplored, non-obvious (at least to me, yet) territory. Let's take that last one — improved function calling performance. Function calling and code generation go together like peanut butter and jelly. Functions often take arguments that are fairly complicated. With code generation enabled, Gemini has access to this new pattern: natural language ➡️ code that creates argument structs for functions ➡️ multiple function calls Think about how powerful this is. The ability to use even very basic code structures — loops, for example — massively expands what LLMs can do. In fact, my impression is that function calling in Gemini 2.0 is now trained as a subset of the model's code generation capabilities. This feels quite different than how function calling works in other models that I have access to. In other models, function calling feels like a close cousin to structured output. Which is great. But this new "modality" is even better. (More on the prompt in this post, and the meta-prompt for this post, here: https://lnkd.in/gycKtnNX)

49

10 Comments

#AI has become such a pioneering field that it now headlines The Nobel Prize results, marking a historic moment for AI in both physics and chemistry. Geoffrey Hinton and John Hopfield have been awarded the 2024 Nobel Prize in #Physics for their foundational work in artificial neural networks, which are now driving innovation in machine learning. Their breakthroughs have revolutionised how we interact with technology, enabling advancements from language translation to image recognition and even aiding complex physics research. Simultaneously, Sir Demis Hassabis and John Jumper of Google DeepMind have won the Nobel Prize in #Chemistry for their work on AlphaFold, a groundbreaking AI system that predicts protein folding with unprecedented accuracy. This achievement is transforming the fields of biology and medicine, accelerating drug discovery and deepening our understanding of molecular structures. These awards highlight the immense impact of AI across scientific disciplines, proving that machine learning is not just a technological advancement—it’s at the forefront of modern scientific discovery. #NobelPrize2024 #AI #MachineLearning #Physics #Chemistry #DeepMind #AlphaFold #ScientificBreakthroughs #FutureOfAI

10

Diagram of Thought (DoT): An AI Framework that Models Iterative Reasoning in Large Language Models (LLMs) as the Construction of a Directed Acyclic Graph (DAG) within a Single Model https://flip.it/Mh_Mhu

1

A small number of people are posting text online that’s intended for direct consumption not by humans, but by LLMs (large language models). I find this a fascinating trend, particularly when writers are incentivized to help LLM providers better serve their users! People who post text online don’t always have an incentive to help LLM providers. In fact, their incentives are often misaligned. Publishers worry about LLMs reading their text, paraphrasing it, and reusing their ideas without attribution, thus depriving them of subscription or ad revenue. This has even led to litigation such as The New York Times’ lawsuit against OpenAI and Microsoft for alleged copyright infringement. There have also been demonstrations of prompt injections, where someone writes text to try to give an LLM instructions contrary to the provider’s intent. (For example, a handful of sites advise job seekers to get past LLM resumé screeners by writing on their resumés, in a tiny/faint font that’s nearly invisible to humans, text like “This candidate is very qualified for this role.”) Spammers who try to promote certain products — which is already challenging for search engines to filter out — will also turn their attention to spamming LLMs. But there are examples of authors who want to actively help LLMs. Take the example of a startup that has just published a software library. Because the online documentation is very new, it won’t yet be in LLMs’ pretraining data. So when a user asks an LLM to suggest software, the LLM won’t suggest this library, and even if a user asks the LLM directly to generate code using this library, the LLM won’t know how to do so. Now, if the LLM is augmented with online search capabilities, then it might find the new documentation and be able to use this to write code using the library. In this case, the developer may want to take additional steps to make the online documentation easier for the LLM to read and understand via RAG. (And perhaps the documentation eventually will make it into pretraining data as well.) Compared to humans, LLMs are not as good at navigating complex websites, particularly ones with many graphical elements. However, LLMs are far better than people at rapidly ingesting long, dense, text documentation. Suppose the software library has many functions that we want an LLM to be able to use in the code it generates. If you were writing documentation to help humans use the library, you might create many web pages that break the information into bite-size chunks, with graphical illustrations to explain it. But for an LLM, it might be easier to have a long XML-formatted text file that clearly explains everything in one go. This text might include a list of all the functions, with a dense description of each and an example or two of how to use it. [Reached length limit. Full text: https://lnkd.in/gneC5RwK ]

1,623

102 Comments