Amelio Vázquez-Reina

Today, I found a fascinating way to think about “Tokens” in LLMs. It came from a conversation between Jensen Huang and Patrick Collison at a recent Stripe event. When asked about the future of compute capacity in five years, Jensen smartly dodged that question a bit, but then gave an excellent explanation and analogy about why tokens are the new forces that will power the next decades of humanity. He explained that we're now producing something unprecedented (and at scale): floating point numbers that possess “value”, which we now call tokens. These tokens are valuable because they encapsulate “intelligence”. People are now taking these tokens and transforming them into English, French, images, videos, chemicals, proteins, robotic movements, etc. And many are working hard to expand the range of concepts and ideas we can create with these tokens. He then goes on to made a compelling comparison between tokens and electricity. In the past industrial revolution, we successfully found a way to convert “atoms” into “electrons” (by boiling water to power electricity turbines). And now, we've discovered a way to convert "electrons" into "tokens" (by using energy to power data centers that train and run LLMs). When electricity was first introduced, few people understood its value. Today, paying for kilowatts is routine. The same will happen with tokens. Right now, only early adopters and builders are paying tokens. Soon, everyone will be paying for tokens on a daily basis to supercharge productivity and power new products and services. Many new industries will be born from and built on top of tokens. When I first heard about this way of thinking, I got goosebumps. Perhaps it’s because Jensen has a good storytelling skill that helps him sell what he builds, but I have to admit this way of thinking gave me a brief surge of pride. I'm proud of humanity's collective effort and creativity, which have taken us from living under rocks to building machines that can "think." That's nothing short but a miracle.

18

5 Comments

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

📚 "LLMs Will Always Hallucinate, and We Need to Live With This" by Sourav Banerjee and the team. 🧠 This is a foundational paper - if you're an AI / LLM practitioner or champion (or naysayer), this is worth reading - many of you will know this but the evidence is vital. 🔍 Summary: As Large Language Models become more ubiquitous across domains, it becomes important to examine their inherent limitations critically (hence my work on the AI Trust / Verisimilitude Paradox). 🤖 This work argues that hallucinations in language models are not just occasional errors but an inevitable feature of these systems. 🎭 The researchers demonstrate that hallucinations stem from the fundamental mathematical and logical structure of LLMs. It is, therefore, impossible to eliminate them through architectural improvements, dataset enhancements, or fact-checking mechanisms. 🧮 As I have said before - we are playing a game of error minimization, so we need to understand risk and risk mitigation. 🎯 There is still utility in LLMs, but they need to be handled and managed with care. ⚠️ We can save you time, money and help you safely navigate the ‘Age of AI’ #AI Risk Guy Digital Human Assistants Paul Edginton Ricky Sydney https://lnkd.in/gRPDET6x

3

2 Comments

Something that stuck with me during my interview with Naveen Rao, VP of Generative AI at Databricks: "I take the hard path." He built an AI-focused chip company Nervana in 2014 when machine learning was still academic theory and the chip industry was deeply out of favor. Naveen saw computing architecture would need to fundamentally change for ML workloads. And then in 2019, observing the power of LLMs like BERT, Naveen foresaw the need for enterprise-grade services to manage their training, deployment, and customization. That vision led to a $1.3B acquisition by Databricks in 2023. What I love about Naveen's approach is his willingness to build for markets that don't exist yet. "There isn't a market, employees don't even know about it. So how the hell do you build a company?" You educate. You convince. You get people to see what you see. It's brutal work, but it's important work. You're reshaping how people think about problems.

176

8 Comments

My third essay for Yule LOGIC is about LLMs and temperature: https://lnkd.in/gqutMN-R

10

1 Comment

Sharing my experiment on creating a Streamlit based chatbot, that combines inputs from Agentic RAG using llamaindex (with openai and MistralAI), and langchain google genai (with Gemini-1.5-Flash), to extract information from images embedded within the PDF files. Was a nice experience to build this chatbot. Still many open questions, ways to improve the solution, in a production scenario..but nice to combine features from multiple libraries and come up with a solution. Please check it out! Thanks. #chatbot #gemini #openai #langchain #llamaindex https://lnkd.in/gPdtxhQF

19

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

16

3 Comments

$10B to train an SOTA LLM model. I've been asking fellow Product, Engineering and GTM leaders if SOTA LLMs are delivering positive business outcomes. Answer - > without a value chain of other AI models like ETL, RPA, ASR, TTS, w/ lots of fine tuning and non AI pre-post processing, enterprise customers aren't see enough value to pay more. In fact, they want to pay way LESS to get the same jobs done. This reminds me that not much has changed for product and engineering leaders. We still have to listen to the voice of our customers and translate that into a product that has positive unit economics for the companies we work for. Its very expensive to build and maintain AI solutions. Very expensive to run AI solutions. The right question isn't should I leverage AI in my product, rather if I use AI will it improve the net profit margin. $$$

15

2 Comments

The cost of training top models is increasing 2-3X per year, and at this rate will reach $1 billion per model by 2027. The findings are from Epoch AI and follow their recent analysis of increases in compute for foundation models running at 4-5X per year. They've also provided some interesting data on the relative breakdown of cost components like staff, hardware, and energy. A few thoughts after reading this: - I would assume staff costs won't increase as much as hardware and energy costs, so the relative contribution of human labor to future AI models will decrease, in yet another sign of capital's increasing dominance over labor in the industry (just look at NVIDIA's ever-rising stock price). - If compute is increasing 4-5X per year but cost is increasing 2-3X, is it safe to assume that training runs are getting more efficient? - Some people, like Anthropic CEO Dario Amodei, have said we'll see $1 billion training runs in the next year. Does this mean the 2-3X pace is about to increase? That Epoch is wrong in its estimate? Or that Amodei is overstating the cost, possibly to put fear into competitors who can't keep pace? - Only a few companies are going to be able to train future models. We're likely going to see further consolidation. Perhaps we get to a future with only two large general purpose models, like the Windows/MacOS and Android/iOS duopolies we've seen in the past. Article here: https://lnkd.in/g5SejXen

7

Congrats VERSES (OTCQB:VRSSF) Gabriel René and Dan Mapes and Dr. Karl Friston for introducing an "efficient alternative to deep learning, reinforcement learning and generative AI called Renormalizing Generative Models (RGMs) that address foundational problems in artificial intelligence (AI), namely versatility, efficiency, explainability and accuracy, using a physics based approach." #ai #activeinference #RGM #spatialweb #AGI #cognitivecomputing https://lnkd.in/gcYSM4eN

23

I am excited to share the latest blog, co-authored by myself, Junhwi Kim and Vyas Sekar, detailing how we managed to successfully localized and improved LLM by combining Fine-Tuning and RAG. https://lnkd.in/gKDkgGPq

70

3 Comments

🤔 "What is AI memory, after all?" We get asked this question a lot - and for good reason. While LLMs are great at generating text, their ability to "remember" is often limited. Just like humans rely on memory to connect conversations and store information, LLMs benefit from memory systems to provide context-rich, personalized, and efficient responses. At cognee, we believe AI memory is the next step toward truly intelligent systems. By integrating memory architectures, we're helping LLMs go from “stateless” responses to adaptive, context-aware interactions that feel much more natural. Here’s a quick take: 💡 Short-term memory handles immediate context (like your current conversation). 📚 Long-term memory stores knowledge across sessions (like remembering user preferences or facts). Why does it matter? ✅ It enables continuity across interactions (so AI doesn’t “forget” you). ✅ Helps reduce hallucinations by anchoring responses in stored facts. ✅ Powers efficient workflows for things like document processing or customer support. We’ve explored this in our latest blog. If you've been curious about why AI memory matters, give it a read! The link is in the comments 👇 Now, over to you: When building AI systems, do you ever wish they could hold onto context better? How do you deal with this challenge? 💬

22

2 Comments

Ilya Sutskever confirmed at NeurIPS '24 pre-training scaling law will not be the same. Internet data, the AI fossil fuel, is running out. It is not the end of the AI hype, the rapid rise of foundational AI has outpaced our ability to fully grasp their long-term societal impact. AI agents and applications are poised to transform every facet of our lives, and we will see it happening in coming few years. There will still be improvements in model architecture, and synthetic data creation, so models will improve, just not at the lightning speed they did till now. Looking forward, the single most important milestone in AI is creating systems capable of advancing themselves—AI researchers that can innovate and improve foundational models autonomously. We need to assess how far are we from this stage.

9

1. IMO, agent patterns are one of the most interesting ways to extend LLMs. memary is based on leveraging a graph database (neo4j) for long-term memory (to get around the limited memory inherent in even the largest context windows in LLMs) and augmenting it with other tools like web search and Computer Vision. https://lnkd.in/gfFcBqJx Looks interesting. 2. More about how consolidation effects the entire healthcare industry: https://lnkd.in/gXJKqNJB The Change hack/ransomware attack not only affected claims, service, refills, etc. but now CMS & NCQA are delaying the entire HEDIS reporting process (used for Medicare/Medicaid quality ratings) due to the impact. 3. NARA bans ChatGPT use agency-wide. https://lnkd.in/gW8T94JU Federal Government agencies see this as a risk, including their information leaking and being used to train models (which may then output the wrong information or show PII/PHI other proprietary information). 4. Oh great, now I can't even be angry anymore :) https://lnkd.in/gcrN9mjx

1

1 Comment

#RAG DOES NOT REDUCE HALLUCINATIONS IN #LLMS: Think I've called this out a few times, but the validation is nice - https://lnkd.in/eAi6zxW2. The article makes two well established points: 1) "Hallucination is a problem of reasoning and not relevance! Any amount of relevant text fed through RAG to a machine will retain the original perplexity and entropy in the system to hallucinate independent of the text"; and 2) "The initial challenge arises from the retrieval phase’s reliance on semantic similarity, which is quantified by inner product spaces or cosine similarities. This metric, while capturing semantic closeness, does not differentiate between factually accurate and inaccurate information." This should be clear to everyone, but the industry still shills flimsy RAG scaffolding anyway (shovels to gold miners).

104

32 Comments

“It’s all about core product creation at the companies that use our AI,” SandboxAQ CEO Jack Hidary told VentureBeat. “And so if you want to create a drug, a diagnostic, a new material or you want to do risk management at a big bank, that’s where quantitative models shine.” OOh and you want an enterprise use case? How's this LQMs for a better version of a Monte Carlo simulation. A Monte Carlo simulation is a classic form of computational algorithm that uses random sampling to get results. With the SandboxAQ LQM approach, a financial services firm can scale in a way that a Monte Carlo simulation can’t enable. “If I have a portfolio and I want to know what the tail risk is given changes in this portfolio,” said Hidary. “What I’d like to do is I’d like to create 300 to 500 million versions of that portfolio with slight changes to it, and then I want to look at the tail risk.”

6

A couple of recent college graduates from India against AI finetuning giants like Anyscale and Run:ai? I met Arko C at a NASSCOM event last October. He was building Xylem AI (YC S24) , a platform that allows you to train and deploy LLMs in production. So naturally most Indian VCs rejected him. Because how could a team of three recent college grads compete with the likes SV startups like Anyscale that have raised hundreds of millions of dollars, have teams of experienced AI researchers and a stash of H100 GPUs? Arko hustled and closed two Fortune 500 companies for paid contracts. I personally know unicorns that haven't managed to do this. He then got into YC in the last round in May. There's a lot to still be proved out, but Arko exemplifies how valuable grit is in building startups. I’m doing a webinar with him tomorrow on challenges of using LLMs in production. Link to register in the comments. Disclaimer: I'm a small investor in Xylem.

153

5 Comments