Utku Irmak

What an end to year 2024, OpenAI o3 has achieved state of the art results on ARC-AGI benchmarks. From ARC-AGI's blog (https://lnkd.in/gKw3bU4U): "To sum up – o3 represents a significant leap forward. Its performance on ARC-AGI highlights a genuine breakthrough in adaptability and generalization, in a way that no other benchmark could have made as explicit. o3 fixes the fundamental limitation of the LLM paradigm – the inability to recombine knowledge at test time – and it does so via a form of LLM-guided natural language program search. This is not just incremental progress; it is new territory, and it demands serious scientific attention." This is indeed a significant leap and in 2025, I expect few things: * OpenAI, Google and others will keep pushing the boundaries with what we can do with reasoning models, models will keep getting better - we have not yet reached the ceiling. * We will see openweights models as well which will be comparable to o3 (we already have few going in that direction like QwQ, Deepseek - both from China). * Products will make an accelerated effort for catching up both of these: https://lnkd.in/gRBuFMGi

45

TRIPLE BAM! If you missed it, here's the recording of our live Q/A with Jay Alammar and Joshua Starmer PhD! We talked about many AI topics, answered lots of great questions, and talked about two great books: Hands-On Large Language Models: Language Understanding and Generation, and The Illustrated Guide to Neural Networks and AI!!! https://lnkd.in/dcPHs7YQ

44

1 Comment

Amidst a range of commercial black box LLMs from large enterprises, we are also seeing Open-Source Machine Learning Frameworks for Advancing Large Language Models in Healthcare. Check out Hippocrates, an open-source LLM framework specifically developed for the medical domain https://lnkd.in/gw5xHdUD

1

If you think compute scaling will plateau, think again. We are in the baby states of inference time compute. VLLMs coupled with computer tools, reward learning and RL can leverage orders of magnitude more inference time compute than what we have seen with foundation models. Here’s a concrete training architecture (replace \pi with a large pre-trained vLLM and add it to the reward learner too, \R is the space of all computer tools)

22

1 Comment

We recently had Jon-Saad Falcon from Stanford AI Group give a talk at SambaNova Systems. He talked about his research ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. You can check out the talk at - https://lnkd.in/eXkBTb-j RAG systems are really hard to evaluate. Lot of hand holding required with annotated data or manual inspection. Excited to see this research and its outcome to help alleviate this problem. Key to this idea is a composite system using LLM as a judge. LLM judges are trained using synthetic data on domain queries/answer pairs. Its exciting to see how our customers use SambaNova Systems Platform to enable these workflows efficiently. Exceptional speed allows for fast synthetic data generation! And ability to host multiple models on a single system with extremely fast model switching allows for efficient low cost deployment. Try out 405B running at 120+ tokens/second at https://fast.snova.ai/ Want to enable your own such workflows by hosting fine-tuned checkpoints? Sign up here - https://lnkd.in/ecAsSDRH

62

Professor Lawrence is spot on in here. Most of the ML solutions that have served us well until now have less to do with the current hype on generative AI and more with a specialized domain knowledge combined with classical modeling techniques and a great deal of software engineering. Certainly what we call Generative AI today will have an important role to play as it is introducing a new way to interact with computers to do stuff for you. However thinking that it will solve all industry/society problems is a very big stretch. https://lnkd.in/e_UX_YUd

6

Thanks Hamdi Amroun, PhD - this is a great paper to share. In the real world business context we often need AI to reason over large pieces of text, and often you need/want multi-document summarization (MDS). 'When testing five LLMs with benchmarks using news and conversation datasets, they found up to 75% of content in MDS summaries was hallucinated, with notable increases towards summaries' end. Alarmingly, even for non-existent topics, models like GPT-3.5-turbo and GPT-4 generated fabricated content 79% and 44% of the time, respectively. Analysis of 700+ generated insights showed that hallucinations often arose from failures to follow instructions or overly generic content.' I agree with Hamdi's sentiments that despite improvements in post-processing techniques, robust solutions are urgently needed to reduce hallucinations, as inaccurate summaries can lead to business risks (notably misinformation and misrepresentation). I would add that there is still a lot of utility in LLMs/Multimodal models but we need to be vigilant - this is not about AI malevolence or intentional deception. LLMs do not possess consciousness or intent; they generate content based on patterns in data they were trained on.

5

1 Comment

Hello All! It is great to be back with another article in my series “Neural Networks Intuitions - 18. Generative Pretrained Transformer(GPT) Series". The past 1.5-2 years have been very much dominated by Large Language Models(LLMs) ever since the release of GPT-3.5 and GPT-4, their fantastic ability to solve wide range of natural language tasks — right from question answering, summarization, visual QA and even code generation. And anyone who has followed this recent trend would know about the talks regarding the emergence of AGI which led to two school of thoughts — one set of people who truly believe that Language Models can lead us to AGI, while the other set sharply denies this statement by arguing that although there are a few signs of reasoning, there is no true extrapolation as such and all of it comes down to the training data(and its scale). The latter group's criticism is that unless there is visibility into the training data distribution and having a test set(without contaminating the test data with train set) along with a proper evaluation methodology, it is unfair to say that these LLMs can actually lead to AGI. Now in order to truly validate some of these closed source GPT models(such as GPT-4, Gemini, Claude etc), we need to be aware of the training data distribution — which is highly unlikely. But another way to understand if LLMs or more specifically instructed tuned LLMs can actually generalize to novel distribution or not is by actually digging deep into the evolution of these architectures/algorithms — more specifically the evolution of GPT models. In this article, I will be going over a series of GPT papers - GPT-1, GPT-2, GPT-3 and InstructGPT, and try to understand what makes these models truly generalizable or universal. Do read and comment your thoughts/opinions :) https://lnkd.in/di8_VpJA #neuralnetworks #largelanguagemodels #GPT #incontextlearning #nlp

23

1 Comment

FriendliAI offers Friendli Model Optimizer, the state-of-the-art LLM compression tool to improve inference speed and reduce resource consumption while maintaining high model accuracy! Learn more about it in our latest blog! #friendliai #friendli #llm #compression #quantization #optimizer #fmo

15

This week, I'm interviewing extraordinary Bradley Voytek — neuroscience prof at UC San Diego, founding member of UCSD's Data Science Institute, and he was Uber's first data scientist. Got questions for him? Questions at the intersection of neuroscience and data science would be particularly interesting, given that Prof. Voytek's research specialty is combining large-scale data science and machine learning to study how brain regions communicate with one another, and how that communication changes with aging and disease. (UCSD is arguably the world's best neuroscience research institute, so this is quite a special opportunity for ya!) More detail on Brad: • Professor in UC San Diego's Department of Cognitive Science, the Halıcıoğlu Data Science Institute, and the Neurosciences Graduate Program. • Alfred P. Sloan Neuroscience Research Fellow. • Kavli Fellow of the National Academies of Sciences. • Joined Uber as their first data scientist, when it was a 10-person startup, helping build their data science strategy and team. • Outreach work has appeared in Scientific American, NPR... and Comic-Con! • Co-authored the amusing book "Do Zombies Dream of Undead Sheep?" The "Super Data Science Podcast with Jon Krohn" is available on all major podcasting platforms and a video version is on YouTube. Brad's episode will likely be #829 and be released on October 22nd. #superdatascience #machinelearning #ai #datascience #neuroscience

86

1 Comment

𝐌𝐮𝐬𝐭-𝐫𝐞𝐚𝐝, 𝐟𝐫𝐞𝐞 𝐛𝐨𝐨𝐤 𝐭𝐡𝐚𝐭 𝐩𝐫𝐨𝐯𝐢𝐝𝐞𝐬 𝐚 𝐩𝐚𝐭𝐡𝐰𝐚𝐲 𝐟𝐫𝐨𝐦 𝐆𝐀𝐈 𝐭𝐨 𝐀𝐆𝐈! ​ Yann LeCun has expressed skepticism about the potential of LLMs to achieve Artificial General Intelligence (AGI), citing limitations in memory, planning, and grounding in real-world understanding. However, we argue that 𝗟𝗟𝗠 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝘃𝗲 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 (𝗟𝗖𝗜)—particularly with multiple multimodal LLMs working together—offers a promising architecture to address these challenges and progress toward AGI. ​ Two key architectural innovations underlie LCI: ​ 𝐒𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐔𝐧𝐜𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐚𝐧𝐝 𝐂𝐨𝐧𝐬𝐜𝐢𝐨𝐮𝐬 𝐑𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐏𝐥𝐚𝐧𝐧𝐢𝐧𝐠: Inspired by the human mind’s dual-layer structure, LCI divides its architecture into foundational and adaptive layers. The foundation layer, akin to unconscious processing, leverages extensive datasets to build robust pattern recognition, encoding essential responses much like human instincts. The adaptive layer, mirroring conscious thought, enables rapid adaptation and contextual reasoning. This dual-layer approach allows LLMs to perform complex tasks with foundational knowledge while adjusting to new information on the fly, much as a child learns new concepts within a developed cognitive framework. ​ 𝐃𝐢𝐬𝐭𝐢𝐧𝐜𝐭 𝐑𝐨𝐥𝐞𝐬 𝐟𝐨𝐫 𝐄𝐚𝐜𝐡 𝐋𝐋𝐌-𝐀𝐠𝐞𝐧𝐭: Each LLM-agent is assigned a specific role, such as executive (knowledge processing), legislative (behavior guardrails), and judicial (adapting to cultural norms). ​ Additionally, our validation of the LCI framework has been thoroughly tested through real-world deployment in applications across various domains, including healthcare, sales planning, investment, and debiasing news. These empirical deployments highlight the practical viability and adaptability of the LCI framework, showcasing its ability to enhance reasoning and decision-making through collaborative intelligence. ​ The eleven aphorisms presented in this free book illustrate the philosophical foundation of the LCI framework, backed by empirical studies. Through LCI, we envision a clear pathway to AGI, rooted in collaborative intelligence and flexible, context-aware adaptation. https://lnkd.in/gn74DkNj (free) https://lnkd.in/gszSvfqP ($6.98 printing fee)

9

👏CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks

25

Large language models (LLMs) are increasingly being explored for their potential in medicine, but applying them effectively to complex clinical scenarios remains a challenge. The paper "MDAgents: an adaptive collaboration of LLMs for medical decision-making" by Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park (NeurIPS 2024) introduces a novel framework aimed at addressing this gap. MDAgents structures collaboration among LLMs dynamically, assigning roles—either independent or group-based—based on the complexity of the medical task. This design mimics real-world clinical decision-making processes, where collaboration is tailored to the case at hand. Evaluated on 10 medical benchmarks, MDAgents achieved leading performance on 7 tasks and demonstrated an 11.8% accuracy boost when incorporating external medical knowledge and moderator reviews. A case study further illustrates how the framework synthesizes differing perspectives among LLMs to reach accurate, consensus-driven diagnoses. This framework stands out for its adaptability and its emphasis on collaboration. Ablation studies underscore the importance of the system’s components, from its ability to classify medical complexity to the integration of multi-modal reasoning. By reflecting the dynamic, consultative nature of clinical decision-making, MDAgents provides a thoughtful approach to enhancing LLM-assisted medical diagnosis. To me, this paper is interesting because it takes a practical step toward aligning AI capabilities with the intricacies of healthcare. By emulating real-world medical collaboration, MDAgents not only improves performance but also lays the groundwork for more robust and trustworthy AI systems in clinical settings. What are your thoughts on how AI frameworks can better reflect real-world decision-making processes in fields like healthcare? paper: https://lnkd.in/gd5fX4Ef #artificialintelligence #AIhealthcare #LLMs #AIAgents

11

I am excited to share that Dr. Feifang Hu, my co-founder and PhD advisor, will present at the Regulatory-Industry Statistics Workshop (RISW) on Sept 27, 2024. He will discuss practical considerations of covariate adjusted inferences in various classic and innovative designs. Join us to explore how statistical innovations and AI are revolutionizing clinical drug development! #RISW2024 #ClinicalTrials #CovariateAdjustment #AI

20

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

89

5 Comments

Beating o1-preview on AIME 2024 with Chain-of-Code reasoning in Optillm In the past week, there has been a flurry of releases of o1-style reasoning models from DeepSeek, Fireworks AI and Nous Research. In our open-source optimizing inference proxy, optillm (https://lnkd.in/gN6_kNky) we have implemented several techniques that use additional inference time compute to improve accuracy and work with a variety of base models. Today, we are happy to announce that by using chain-of-code (coc) plugin in optillm we are able to beat OpenAI's o1-preview on AIME 2024 (pass@1) using SOTA base models from both Anthropic and Google DeepMind. For reference, also see the original paper that introduced the idea of CoC: Chain of Code: Reasoning with a Language Model-Augmented Code Emulator -https://lnkd.in/gtq7hbjx We have done an independent implementation of coc in optillm as the original source code was not released.

29

2 Comments

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15