Muhammad Jehangir Amjad

Today, I found a fascinating way to think about “Tokens” in LLMs. It came from a conversation between Jensen Huang and Patrick Collison at a recent Stripe event. When asked about the future of compute capacity in five years, Jensen smartly dodged that question a bit, but then gave an excellent explanation and analogy about why tokens are the new forces that will power the next decades of humanity. He explained that we're now producing something unprecedented (and at scale): floating point numbers that possess “value”, which we now call tokens. These tokens are valuable because they encapsulate “intelligence”. People are now taking these tokens and transforming them into English, French, images, videos, chemicals, proteins, robotic movements, etc. And many are working hard to expand the range of concepts and ideas we can create with these tokens. He then goes on to made a compelling comparison between tokens and electricity. In the past industrial revolution, we successfully found a way to convert “atoms” into “electrons” (by boiling water to power electricity turbines). And now, we've discovered a way to convert "electrons" into "tokens" (by using energy to power data centers that train and run LLMs). When electricity was first introduced, few people understood its value. Today, paying for kilowatts is routine. The same will happen with tokens. Right now, only early adopters and builders are paying tokens. Soon, everyone will be paying for tokens on a daily basis to supercharge productivity and power new products and services. Many new industries will be born from and built on top of tokens. When I first heard about this way of thinking, I got goosebumps. Perhaps it’s because Jensen has a good storytelling skill that helps him sell what he builds, but I have to admit this way of thinking gave me a brief surge of pride. I'm proud of humanity's collective effort and creativity, which have taken us from living under rocks to building machines that can "think." That's nothing short but a miracle.

18

5 Comments

Ilya Sutskever confirmed at NeurIPS '24 pre-training scaling law will not be the same. Internet data, the AI fossil fuel, is running out. It is not the end of the AI hype, the rapid rise of foundational AI has outpaced our ability to fully grasp their long-term societal impact. AI agents and applications are poised to transform every facet of our lives, and we will see it happening in coming few years. There will still be improvements in model architecture, and synthetic data creation, so models will improve, just not at the lightning speed they did till now. Looking forward, the single most important milestone in AI is creating systems capable of advancing themselves—AI researchers that can innovate and improve foundational models autonomously. We need to assess how far are we from this stage.

9

Yes, the future is AI Agents navigating the Web in the decentralised manner, using Croissant for Machine Learning to check content ownership and take licenses into account. It's a kind of "TomTom" for the future Web, represented as moving knowledge graph. We knew all of that for years and it's coming.

2

1 Comment

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

47

2 Comments

Quantization is a game changer for deploying large language models like Mistral 7B on consumer hardware. This course offers invaluable techniques to harness powerful AI models even with limited resources.

2

Much has been said about many companies’ desire for more compute (and data) to train large foundation models. I think it’s under-appreciated that we also have nowhere near enough compute available for inference on foundation models. Years ago, when I was leading teams at Google, Baidu, and Stanford that focused on scaling up deep learning, many semiconductor makers, data center operators, and researchers asked me if AI would continue to make good use of more compute if they kept delivering it. For many desktop workloads, like running a web browser, a faster CPU doesn’t help much beyond a certain point. So do we really need faster and faster AI processors? Each time, I confidently replied “yes!” and encouraged them to keep scaling up compute. (Sometimes, I added half-jokingly that I had never met a machine learning engineer who felt like they had enough compute. 😀) Fortunately, this prediction has been right so far. However, beyond training, we are also far from exhausting the benefits of faster and higher volumes of inference. Today, a lot of LLM output is for human consumption. A human might read around 250 words per minute, which is around 6 tokens per second (250 words/min / (0.75 words/token) / (60 secs/min)). So it might seem there’s little value to generating tokens much faster than this. But in an agentic workflow, an LLM might be prompted repeatedly to reflect on and improve its output, use tools, plan and execute multiple steps, or implement multiple agents that collaborate. So, we might generate hundreds of thousands of tokens or more before showing any output to a user. This makes fast token generation very desirable and makes slower generation a bottleneck to taking better advantage of existing models. That’s why I’m excited about the work of companies like Groq, which can generate hundreds of tokens per second. Recently, SambaNova also showed it can hit hundreds of tokens per second. Incidentally, faster, cheaper token generation will also help make running evaluations (evals), which can be slow and expensive today since it involves iterating over many examples, more palatable. Fortunately, both training and inference are rapidly becoming cheaper. I spoke with Cathie Wood and Charles Roberts of the investment firm ARK, which is famous for its bullish predictions on tech. They estimate that AI training costs are falling 75% a year. If they are right, a foundation model that costs $100M to train this year might cost $25M to train next year. Further, they report that for “enterprise scale use cases, inference costs seem to be falling at an annual rate of ~86%, even faster than training costs.” I don’t know how accurate these specific predictions will turn out to be, but with progress in both semiconductors and algorithms, I do see training and inference costs falling rapidly. This will be good for application builders and help AI agentic workflows lift off! [Original text: https://lnkd.in/dJ9tVGh7 ]

2,894

80 Comments

OpenCoder: An Open-Source Code LLM Redefining Model Efficiency 🌟 The OpenCoder team has launched a game-changer in the world of code language models. Unlike many LLMs that focus on sheer size, OpenCoder brings something unique to the table: a data-focused training approach. By putting quality before volume, OpenCoder’s 1.5B and 8B models rival some of the biggest players, like DeepSeekCoder and QwenCoder, with only a fraction of their token count. The core strength of OpenCoder lies in its meticulous data processing pipeline and two-stage fine-tuning. With a training dataset composed of 2.5 trillion tokens, 90% raw code and 10% code-related web data, OpenCoder refines every aspect of its corpus. Advanced filtering and deduplication strategies, applied to 607 programming languages, optimize model efficiency and precision, ensuring high-quality data drives its training. Key Features: - Complete Training Transparency: Open-source datasets and training pipelines enable reproducibility and further research. - Instruction Fine-Tuning: Two-stage tuning sharpens both core CS concepts and real-world coding skills, prepping OpenCoder for production-level code generation. - Available on Ollama: Accessibility in deployment means more avenues for experimentation and improvement. OpenCoder's data-driven strategy showcases the potential of a leaner approach. By focusing on quality data rather than quantity alone, OpenCoder achieves file-level deduplication that outperforms traditional repo-level methods, making it both efficient and robust. 🚀 Read more here: https://lnkd.in/eEnG3yJA #OpenSource #MachineLearning #CodeLLM #AI #ModelEfficiency #DataQuality #Innovation #CodeGeneration #OpenCoder #TechNews #AIResearch #Programming #LargeLanguageModels #ArtificialIntelligence

1

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15

AI memory is fundamental for conversational AI and AI agents. It is critical for building personalized experiences and improving reasoning. Check out Letta’s course to understand how to use AI memory to enhance the customization and quality of your AI app.

15

Thanks Hamdi Amroun, PhD - this is a great paper to share. In the real world business context we often need AI to reason over large pieces of text, and often you need/want multi-document summarization (MDS). 'When testing five LLMs with benchmarks using news and conversation datasets, they found up to 75% of content in MDS summaries was hallucinated, with notable increases towards summaries' end. Alarmingly, even for non-existent topics, models like GPT-3.5-turbo and GPT-4 generated fabricated content 79% and 44% of the time, respectively. Analysis of 700+ generated insights showed that hallucinations often arose from failures to follow instructions or overly generic content.' I agree with Hamdi's sentiments that despite improvements in post-processing techniques, robust solutions are urgently needed to reduce hallucinations, as inaccurate summaries can lead to business risks (notably misinformation and misrepresentation). I would add that there is still a lot of utility in LLMs/Multimodal models but we need to be vigilant - this is not about AI malevolence or intentional deception. LLMs do not possess consciousness or intent; they generate content based on patterns in data they were trained on.

5

1 Comment

Today's guest is the amazing Google DeepMind research scientist, Dr. Rosanne Liu (pictured here in San Francisco this past weekend with myself and Jason Yosinski, who introduced me to Rosanne). Rosanne:  • Is a Research Scientist at Google DeepMind in California. • Is Co-Founder and Executive Director of ML Collective, a non-profit that provides global ML research training and mentorship. • Was a founding member of Uber AI Labs, where she served as a Senior Research Scientist. • She has published deep learning research in top academic venues such as NeurIPS, ICLR, ICML and Science, and her work has been covered in publications like WIRED and the MIT Tech Review. • Holds a PhD in Computer Science from Northwestern University. Today’s episode, particularly in the second half when we dig into Rosanne’s fascinating research, is relatively technical so will probably appeal most to hands-on practitioners like data scientists and ML engineers. In today’s episode, Rosanne details:  • The problem she founded the ML Collective to solve. • How her work on the “intrinsic dimension” of deep learning models inspired the now-standard LoRA approach to fine-tuning LLMs. • The thorny problems with LLM evaluation benchmarks and how they might be solved. • The pros and cons of curiosity- vs goal-driven ML research. • The positive impacts of diversity, equity and inclusion in the ML community. The "Super Data Science Podcast with Jon Krohn" is freely available on all major podcasting platforms and we publish a video version on YouTube. This is Episode #797! #superdatascience #machinelearning #ai #deeplearning #llms

114

11 Comments

🎯Data was, is, and will be the only fuel. Analogy with the fossil fuel is a good one, since the data we have today is as dirty as oil and coal. What big tech companies tried is to put a lot of "garbage" data, and what they did get was fantastic. Yet, we are at the point of diminishing returns. Data is the beginning and the end of this cycle.” Great thread, particularly this comment 👆from Ilija Lazarevic

20

2 Comments

I’ve been fortunate enough to do AI shows with Sam Altman, Yann LeCun & Arthur Mensch. But this show today with David Cahn at Sequoia Capital is the best of all of them: 🧠 AI’s $600BN question 🏃 Why no one will never train a frontier model on the same data center twice 🏗️ Servers, steel & power: The 3 pillars of the future of AI Not one to be missed, my 8 key takeaways 👇 1. The Oligopoly That is Blowing Billions on AI - Microsoft, Azure, & Google now represent $7T market cap. - Of course they will spend billions to protect their oligopoly - What is most interesting is Meta is the only incumbent that is financing this not with cloud being their cash cow. 2. How Startups Could Win with the Cost of Compute Coming Down - Big tech companies are producers of compute & startups are consumers of compute. - Overproduction of compute = lower prices.  - This gives startups higher margins & makes them more valuable. 3. Is Compute the Currency of the Future or Do the Costs Come Down and It Is Commoditised? - We are forgetting “compute” is a euphemism for a $2BN data center in the middle of nowhere. - They have GPUs & liquid cooling systems that need to be constantly optimized & upgraded. - We cannot simply build 15 years worth of compute all at once.  No one's ever going to train a frontier model on the same data center twice. 4. Data Centre Teams and Model Teams Need to Be Coupled - You cannot have separate teams running data centers & building models. - Elon & Zuck control all their data centres. - It will not work as these models get better & better. 5. Why the Data Center Is the Most Important Asset - There is not much difference between these models today. - AI researchers are jumping from one lab to another. - Scaling laws will become more dominant as these models get bigger. 6. Servers Steel and Power: The Three Bottlenecks of AI Three things: 1. Servers: NVIDIA, AMD, Broadcom… 2. Steel: Construction & real estate developers for data centers. 3. Power: All the energy required for AI. - More factories will be built in 12 months for AI and we will see another industrial revolution. 7. You Have to Have a Cash Machine to Play the Big Model Game - Amazon has AWS - Meta has Instagram - Microsoft has Azure - You need a cash machine to compete and it CANNOT be the AI business. 8. There is One Definition of Success in Venture: Are you a Slugger? - If you are generating billion dollar gains, you are a slugger. - Doug Leone, Pat Grady, Andrew Reed are all sluggers. - You cannot be good at this business without being a slugger. (links in comments) #founder #funding #business #investing #vc #venturecapital #entrepreneur #startup #seed #funding #sequoia

190

26 Comments

"'Progress from o1 to o3 was only three months, which shows how fast progress will be in the new paradigm of RL [Reinforcement Learning] on chain of thought to scale inference compute,' writes OpenAI researcher Jason Wei in a tweet. 'Way faster than pretraining paradigm of new model every 1-2 years.' I think basically no one is pricing in just how drastic the progress will be from here." - Jack Clark, Import AI Newsletter

15

Ilya like all other LLM vendors is now actively trying to cover up the 10s of billion$ in LLM tech debt by claiming “Agentic AI” as the new panacea to solve intelligence - even lowering the bar for AGI. You can't brute force your way to solve intelligence. There is no "i" in AI without "Cognition". Self-learning and adapting in real-time isn't going to happen with LLMs or the wrapper "Agents" sitting on top of these LLMs. Hopefully they are not going to sit on "Agentic AI" for another few years to arrive at this conclusion as they did with the scaling laws. We need a Neuro-Symbolic approach centered around human-like Cognition to solve intelligence that doesn’t require trillions of data points, billion$ in computational resources and even more $$ in energy requirements while it cannot learn & adapt continuously yet after all this it hallucinates. INSA is the exact opposite of this. INSA: Integrated Neuro-Symbolic Architecture https://lnkd.in/ggFTxuUz The insanity of huge language models: https://lnkd.in/d5pcyf6R

32

11 Comments

Nearly a decade ago, I wrote out a complete and rigorous proof of Cantor's theorem that the set of sequences of natural numbers is bigger than the set of natural numbers...using only words of one syllable. https://lnkd.in/g4wWYVYc This is not the only time I revisited the result, much more recently I formally published a number of generalisations of this using different fundamental assumptions (for experts: taking Lawvere's fixed-point theorem and the contrapositive diagonal argument and showing how the proof works in a substructural type of logic). But this one-syllable-word proof was so much fun. I was reminded of this because someone got in touch with me, saying how it is something he periodically re-read for pure interest's sake, and asking if I could recover some material that was on the now-vanished Google+ to which I linked. I had in fact saved that discussion, which had contributions of similar pieces from a number of other mathematicians, including the Fields medal winner Tim Gowers. I will put a pdf of that thread somewhere public, soon, and link to it. But my correspondent was so taken by the exercise that he said he'd go so far as to purchase a physical book containing a collection of such "short words proofs", if one existed. The idea of publishing a slim volume of fun little proofs certainly is appealing, in some respects, because I love producing physical artifacts that reflect mathematics, but the market must surely be tiny!

7

2 Comments

This is a great first step in understanding how scaling dynamics extend beyond language to other types of sequential data. Time is a universal axis, understanding how models learn optimally over data collected over time is a crucial step in the broader AI research agenda.

26

Cost-aware inference is important, but currently LLM prompts are not interoperable. Prompts that work good on one LLM may show worse performance on the other, even more advanced LLM or even on the different versions of the same LLM. Once interoperability problem is solved it will also be possible to implement cost-aware inference that strategically manages computational resources for prompt processing, ensuring sustainable costs. This approach includes model routing https://www.notdiamond.ai/ - automatically selecting the most appropriate model for each task. Startups in this area are actively supported by the industry veterans.

4

2 Comments

🚀 Exciting News: UC Berkeley Introduces Sky-T1-32B-Preview for $450! UC Berkeley has introduced the Sky-T1-32B-Preview, an open-source reasoning model that is causing a stir in the AI community. This innovative model competes with OpenAI's o1 model, showcasing remarkable performance capabilities. This groundbreaking development highlights the potential for advanced AI functionalities at an affordable price point of just $450. The Sky-T1-32B-Preview not only matches the o1 model in benchmarks like Math500 (scoring 82.4%) but also excels in LiveCodeBench-Easy with an impressive score of 86.3%, demonstrating its proficiency in both mathematical and coding tasks. What makes this model truly stand out is its adherence to open-source values. The NovaSky team at UC Berkeley has generously shared all essential resources for replicating and improving the model, including training data, code, and model weights. Furthermore, they have provided training infrastructure scripts and a detailed technical report with wandb logs, creating a valuable resource for researchers, developers, and AI enthusiasts looking to explore or contribute to the realm of AI reasoning models. #AI #UCBerkeley #OpenSource #ArtificialIntelligence

2