Robert Kim

Optillm: An optimizing inference proxy for LLMs Optillm is an optimizing inference proxy that has over a dozen techniques that aim to improve the accuracy of the responses using test-time compute. Over the last couple of months we have set several SOTA results using smaller and less capable models like gpt-4o-mini. Recently, we have added support for plugins that enable capabilities like memory, privacy and code execution to optillm. Plugins are just python scripts that you can also write yourself, optillm would then load them at start from the directory. You can now also combine the plugins and techniques using & and | operators. E.g. We recently evaluated the new FRAMES benchmark from Google. Using a combination of plugins and techniques (we used readurls&memory-gpt-4o-mini) we were able to get 65.7% accuracy on the benchmark which is very close to what Google reported in their paper with Gemini Flash 1.5 (66.5) which has a context length that is almost 10 times that of gpt-4o-mini. Do check out Optillm at https://lnkd.in/gN6_kNky

21

1 Comment

💥💥💥 LLMs are in-context RL learners, but not great because they can’t explore well. How do we teach LLMs to explore better? Solution: Supervised fine-tuning on full exploration trajectories. 👉 EVOLvE: Evaluating and Optimizing LLMs For Exploration Allen Nie, Yi Su, Bo Chang, Jonathan N. Lee, Ed H. Chi, Quoc V. Le, Minmin Chen Abstract Despite their success in many domains, large language models (LLMs) remain under-studied in scenarios requiring optimal decision-making under uncertainty. This is crucial as many real-world applications, ranging from personalized recommendations to healthcare interventions, demand that LLMs not only predict but also actively learn to make optimal decisions through exploration. In this work, we measure LLMs' (in)ability to make optimal decisions in bandits, a state-less reinforcement learning setting relevant to many applications. We develop a comprehensive suite of environments, including both context-free and contextual bandits with varying task difficulties, to benchmark LLMs' performance. Motivated by the existence of optimal exploration algorithms, we propose efficient ways to integrate this algorithmic knowledge into LLMs: by providing explicit algorithm-guided support during inference; and through algorithm distillation via in-context demonstrations and fine-tuning, using synthetic data generated from these algorithms. Impressively, these techniques allow us to achieve superior exploration performance with smaller models, surpassing larger models on various tasks. We conducted an extensive ablation study to shed light on various factors, such as task difficulty and data representation, that influence the efficiency of LLM exploration. Additionally, we conduct a rigorous analysis of the LLM's exploration efficiency using the concept of regret, linking its ability to explore to the model size and underlying algorithm. 👉 https://lnkd.in/d6BeKYZe #machinelearning

2

🌟 Live RAG Comparison Test: Pinecone vs. MongoDB vs. Postgres vs. SingleStore ❓Why It Matters: 🔹The role of Retrieval-Augmented Generation (RAG) has become increasingly significant in today's #GenAI landscape. RAG combines the best of both worlds of generative and retrieval-based models to provide more contextually accurate and rich responses in LLM apps.  🔹Latency is crucial for RAG in LLM apps because it directly impacts the user experience by determining how quickly and interactively the model can retrieve and integrate relevant information to generate accurate responses 🔹Rohit Bhamidipati will offer a hands-on RAG performance evaluation of leading back-end/database vendors such as Pinecone vs. MongoDB vs. Postgres vs. SingleStore, comparing throughput, efficiency, and latency for large datasets. 📚 What You'll Learn: ➡️ The mechanics of RAG and its impact on enhancing language model responses. ➡️ How Vector Databases like Pinecone, MongoDB, PostgreSQL, and SingleStore facilitate the functionality of RAG. ➡️ Comparative analysis showcasing real-time performance metrics of these databases. ➡️ Best practices for integrating these technologies into your AI and ML projects to boost efficiency and accuracy. ⚡️ Can't make it? No worries! All registrants will receive a copy of the webinar recording and additional resources via email post-session ✅ Live demo and code-share session will be offered for a hands-on experience 📅 Event Details: 🔹 Thursday, May 9 @ 10-11 AM PDT 🔹 #Free Registration: https://lnkd.in/g6EvB8NG #artificialintelligence #rag #ad

19

1 Comment

🚀 Check out my latest blog post on the UPR Reranker 🔍 Learn how UPR Reranker reorders passages for higher accuracy and performance, and see how to use into AutoRAG with a simple YAML configuration. 📈 You can compare UPR reranker performance between 10+ rerankers in AutoRAG. 👉 Read about UPR reranker : https://lnkd.in/gYffgYdy #RAG #AutoRAG #NLP #AI #UPRReranker #Reranker

5

1 Comment

LLMs' capability follows a "bathtub curve" of usefulness. At the lowest level, they excel as a super-autocomplete, generating concise code snippets. Conversely, at the peak, they effectively convey high-level task-relevant concepts. However, in the middle ground, they struggle.

1

Machine Learning Engineer - Founding Team - London £200k If you answer yes to 4/5 of these questions, then you want to check out this job (link in comments) 1️⃣ Do you have experience with quantization techniques (e.g., PEFT, DeepSpeed, TensorRT) to optimize ML models? 2️⃣  Have you ever managed or worked with a multi-node HPC cluster? 3️⃣ Are you familiar with Deep Learning using PyTorch Library? 4️⃣ Do you have experience with Kubernetes container orchestration? 5️⃣  Have you worked with some of the recent Large Language Models (e.g., OpenAI, LLaMA)? #LLM #ArtificialIntelligence #Quantization

21

3 Comments

Generalisation is Key to AGI 🗝️ I keep hearing that we're on the verge of achieving AGI, but I believe current approaches won't get us there. Right now, we're placing all our hopes on neural networks, thinking that by pouring in more compute and data, we'll evolve them into AGI. Neural networks are effective tools for pattern recognition and prediction but often struggle with out-of-distribution generalisation. This means they excel in tasks similar to their training data but can falter when encountering novel or unfamiliar scenarios. The root of this issue lies in their focus on learning specific patterns rather than broader, underlying principles that could be applied to diverse contexts. For example, a neural network trained to identify cat breeds might perform well on familiar images of cats but could struggle with recognising a cat in unusual situations, such as wearing a costume or captured from a unique angle. This challenge in generalisation can impact their ability to reason and plan effectively, given the complexity and unpredictability of real-world scenarios. These limitations also have significant ethical implications. The demand for extensive data can lead to privacy concerns and amplify existing biases within the data. Furthermore, the potential for failures in unfamiliar contexts, such as in autonomous vehicles or medical diagnostics, underscores the risks associated with current neural network technologies. To move toward Artificial General Intelligence, it is crucial to explore alternative approaches that can better generalise to new and diverse data. These methods should focus on understanding fundamental principles and relationships rather than merely memorising patterns from training sets. #ArtificialGeneralIntelligence #DataEthics #TechInnovation

15

if you know, you know yann, yawns as in it's boring to work on llms. so do some 'ting else. wow. the day that innovation dies in yann's head. necessity is the mother of invention. so it's said. and for a good reason. it is our sense of being a bit upset of the status quo that is the fuel, that motivates us to search, discover, build, invent, anew. yet there is some 'ting else which is missing. the will. the will or deep burning desire, that is beauty within our (hu)man nature. and our perspective is that it begins here. for this reason lecunn, cunningly suggests not to try. sounds a bit hypocritical coming from someone who toiled in academic hell along with yoshua bengio and geoffrey hinton. if there is anyone after receiving finally recognition by way of splitting the 2019 turing prize, you would imagine that lecun would le be supportive of ingenuity. or rather to dare to dream. so we find this statement curious. at a time where we are justified at the very beginning of (ai) or agi or whatevs it seems a bit rich for this chevalier to suggest that the majors have it in the bag. how arrogant. or rather how predictably french. i mean the french are known for their superiority complex. in many ways it suited them. there are michellin star restaurants thanks in large part to this arrogance. there is also fashion, design, literature, music, culture. indeed it is cultural although horribly tribal. and so this tribal sense we believe leads lecunn to be le spoiler. or le party pooper. llms are not done. they are models. this requires a sense of math, design, nay a daring to test limits as nothing before. this is the inception of invention, is it not. so being as lecunn is controversial (and we do love him for this) we ask lecunn to put a le shut up. and encourage for others to delve into the (un)known with a sense of joy, adventure and without limits. one being that the message or narrative is done. is he perhaps a bit worried. that quicker than he or his peers could, come up dare we say something novel. built on the shoulders of this giant. are we simply talking your book. as in the investments have arrived and we are greedy. we'd like this all for ourselves. yeah. abundantly clear. n'est pa? source: https://lnkd.in/eUVsq_J8 < . >

1

1 Comment

🤖 Creating a fully autonomous AI developer to write and maintain code is still the holy grail of software development. There are many brilliant teams currently working on this, and each of them have their own unique approach. The team at Cosine have made a massive step towards making this a reality. “Cosine achieves state-of-the-art results on the SWE-bench benchmark” They have achieved a score of 43.8% on SWE Bench, which was verified by OpenAI. What's even more incredible is they have managed to do this with a lean team and a relatively limited budget. Though it’s important to note Cosine isn’t looking to replace human developers entirely, but rather augmented them with human-like assistants that they can collaborate with on any kind of coding task. Yang Li + Sam Stenner + Alistair Pullen Read more about their collaboration with OpenAI 👇 https://lnkd.in/gu7xXVPn

21

Have Open Source LLMs finally caught up with Private LLMs? Historically, Open Source Performance lagged Private Models by 6-12 months. However, Llama 3.1 has narrowed the gap even further. 🚀 As Foundation Models evolve, the gap between open-source models like Llama 3.1 and closed-source models like ChatGPT-4 is shrinking. Chamath Palihapitiya said it best: "Foundation models will have no economic value; they will be broadly available and entirely free." 🌐 Here’s what I believe: The real moat in AI will shift away from proprietary models. Why? Open-source models like Mistral and Llama, trained on open data, are eroding the dominance of proprietary models. The focus is moving towards usability.🔍 True defensibility in AI will come from: - User experience - Brand - Rich feature sets Most users prioritize ease of use and value over model accuracy and benchmarks. Companies are shifting to enable "Outcomes" for end users, paving the way for "Services as a Software." 📈 That's why: - Anthropic hired Instagram's co-founder Mike Krieger as their CPO to enhance user experience. Take Midjourney: It leads in Image Model-as-a-Service due to its image quality, not just model performance. Even Apple is connecting with other LLMs instead of building their own, integrating these features seamlessly into their products. The landscape is changing. How will your strategy adapt? 💡 #AI #OpenSource #MachineLearning #UserExperience #TechInnovation #AITrends #FoundationModels #BusinessStrategy #DigitalTransformation #FutureOfTech

17

1 Comment

Neural Magic ML team is introducing 2:4 Sparse Llama: The first sparse foundation model built on Llama 3.1 8B. We show that it is possible to combine sparsity and quantization for extreme compression.

20

2 Comments

One year ago, we started Epsilla (YC S23) with a big dream. Today, as we celebrate our first birthday, I want to share the story that brought us here. Many people ask why I left my job as a senior director at TigerGraph, took a huge pay cut, and founded my own company. Since I was a kid in elementary school, I've watched the story of Bill Gates building the Microsoft empire, and dreamed of one day building my own company. This dream grew stronger when I came to the US in 2014 pursuing my master's degree at Cornell, and applied to Y Combinator with my roommate, Hao Ma, with a simple anonymous social app idea. Even though it didn’t work out then, it planted the seed in my heart. During my time at Cornell, I met 4 other entrepreneurial students, Landice Gao, Xiaoyan Wu, Jiaqi Su, and Kejia Tao, and started a company focused on indoor navigation. I wished it could have gone further, but as an international student, I needed a job that could sponsor my working visa to stay in the US. So I joined TigerGraph at 2015, a pre-series A startup at that time. Since day 1, I was trying my best to grow as fast as I can. At TigerGraph, I learned a lot from industry leaders like Amol Ghoting, Li Chen, Mingxi Wu, Ruoming Jin, Yu Xu, Like Gao, Tim Zhang, Songting Chen, Shuo Yang, and many other talented colleagues. Most importantly, I met my future co-founders, Jing Qin and Eric Y., and our future advisor and angel investor, Jay (JieBing) Yu, PhD. When ChatGPT launched in November 2022, it blew my mind. I thought such advanced technology would appear in 2045 (according to "The Singularity Is Near"), but it was already here. However, I quickly realized that large language models (LLMs) didn’t have memory and their knowledge was fixed. Our friend Weimo Liu mentioned vector database to us, and I saw the huge potential of combining AI with data. This led to the birth of Epsilla, aiming to build the data and knowledge foundation for LLMs. Becoming a CEO from a technical background has been tough, especially for an introvert like me. I am so proud of my growth so far. In the past year, I’ve talked to more people than I have in my entire life. I’m grateful for all the interesting people I’ve met around the world, which wouldn’t have happened without starting this journey. Looking ahead, I’m very dedicated to the success of Epsilla. Our journey is just beginning, and I’m excited to see what the future holds for us. Thank you to everyone who has been part of this journey—our customers, partners, team, investors, and supporters. Your belief in our vision keeps us going every day. Here’s to one year of Epsilla, and many more to come! 🚀✨

105

32 Comments

The video of my talk at C++ on Sea 2024 has been published on YouTube today. It concerns algorithms to convert floating-point numbers to string. How you would go about solving this problem? At first sight, it might look trivial: just a bunch of divisions and modulus by 10. Then, you might realised that for floating-point numbers things are not that easy. You might become hopeless and find it very complicated. That was my journey until I had my Aha! moment. I found a novel algorithm which is simpler and faster than everything I came across before. Curious? If so, watch my talk and if you find worthwhile, please give it a "like" on YouTube and share the video with your friends and colleagues. Note for non C++ programmers: the talk has little to do with C++. It's really about the algorithm which can be implemented in many programming languages and I'm hopeful it will. https://lnkd.in/eP6Nwnyv #cpp #algorithms #mathematics

75

4 Comments

Hugging Face demonstrated test-time scaling with Llama 1B, turning it into a "reasoner" that outperformed Llama 8B (8x larger!) in math. Test-time scaling has opened up a whole new dimension in model performance and AI research. I wrote about this when OpenAI released its o-1 model (the first 'reasoner' model). You can read it here: https://lnkd.in/gi737nsA At its core, a reasoner model uses Chain of Thought (CoT) reasoning—breaking problems into logical steps, solving them sequentially, and backtracking when needed. Scaling test-time compute enables this reasoning through two techniques: 1/ Self-Refinement: Models iteratively improve outputs, correcting errors over multiple passes. 2/ Search & Verifiers: Generate multiple outputs and evaluate them with verifier models (e.g., Process Reward Models for steps, Outcome Reward Models for final answers). Hugging Face’s contribution stands out because they’ve shown test-time scaling can be applied to any model, even smaller, open-source ones. They’ve also made it easier for others to replicate and advance the technique—helping the OS community catch up. That said, inference scaling is still far from efficient. Llama 1B required ~20 reasoning steps to outperform Llama 8B, a model only 8x larger. Nevertheless, inference scaling already achieves two critical things: – Pushing the frontier: Scaling inference pushes the boundaries of accuracy and delivers better results. – Smaller hardware requirement: Smaller models can match the performance of larger ones, reducing reliance on expensive infrastructure. The next frontier isn’t just about bigger models—it's about smarter inference. Excited to see where this leads!

62

4 Comments

💡 Introducing Pype's Prompt Optimizer Check out the demo video to build leaner and effective prompts in minutes using Pype. 2 months back, when we started building Pype, we knew that Test-Driven Development (TDD) can be a game-changer for building reliable AI agents. At Pype, we've developed a Prompt Optimizer to streamline this TDD approach. How Pype helps you? 📝 Draft and Test Prompts 🤖 Auto-Enhance Without Overfitting 📊 Version and Compare Prompts 🚢 Deploy and track performance live With a step by step approach, it's intuitive and systematic. Plus, you can: ➕ Add Context Blocks for clarity 🌡 Experiment with Advanced Prompting Techniques 🎛 Switch between Different LLM Models 📨 Generate Test Cases even without production data 🏷 Annotate samples and add feedbacks Ready to build killer LLM apps? Visit www.pypeai.com to try it out. It's free to use. For complex use cases, we offer personalized onboarding support. 💼 Who's Using It? Companies are already leveraging Pype for tasks like RFP triaging, content moderation, and AI tutoring. If you are building an AI agent, give Pype a try 🚀 We would love to hear back from you.

122

9 Comments

DeepSeek V3 - new open model that is better than Claude Sonnet at the Aider Polyglot coding benchmark and - perhaps most importantly - has 10x cheaper inference costs. The massive price drop means you can experiment with having a Aider build you a feature for $2 instead of $20. $20 is a little too pricey for rapid experimentation in your development workflow. Unsurprisingly, Deep Seek's site is down. Other Model Inference providers like Hyperbolic seem to have also taken Deep Seek V3 down for maintenance https://lnkd.in/gtn8Hb36

9

I built a tool entirely with v0 to compare retry algorithms to visually understand their behaviour. You can use it to compare linear retry, exponential backoff, and exponential backoffs with capped duration. Check it out at http://compareretries.com

44