Abhishek Sethi

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

46

2 Comments

I just came across a fascinating paper that helped me crystalize a part of our work at Neon to achieve very high computational efficiencies on our models: switching from NxN to Nx1 convolutions. Drawing on insights from BitNet, which quantized LLMs to binary values, this paper introduces matrix multiplications with matrix-multiplication-free dense layers with ternary weights, offering significant efficiency gains. The performance benefits from pure algorithmic sense are simple to grasp. Matrix multiplication is O(n^3) and, in the best case, O(n^~2.4). Linear operations are O(N), but you will trade that for depth, so the correct value is n_layers * O(N). But that's just the tip of the iceberg. Now let's look at a H100 GPU -- what makes that fast? HBM3 memory and a pile of other tricks that reduce latency to half of an A100. nVidia invested in memory latency and bandwidth because they are a big part of keeping a monster GPU fed. Matrix multiplications cause non-local access and thus add to the complexity needed to overcome those non-local accesses. That is why fast MatMul ops only help at large matrix sizes. Linear operations, on the other hand, are cache-friendly, pre-fetchable, and very friendly for SIMD operations. These hardware advantages allow models to work better not only GPUs but also CPUs and NPUs. These findings are not just theoretical. They have practical implications, demonstrating that a well-designed network can effectively replace the cross-correlative weighing functions provided by matrix multiplication with depth-wise operations. These operations are not only common in tokenization but also find applications in various other areas. This shift towards more architecture and efficiency-aware designs is a promising development, opening up new possibilities for the use of LLMs and generative models on a wider range of devices. For a more in-depth exploration, check out the links: 📜Scalable MatMul-free Language Modeling: https://lnkd.in/gVpMhjdU 📜BitNet: Scaling 1-bit Transformers for Large Language Models: https://lnkd.in/gzFbtduJ 💡Example of MatMul use in CLIP (AI by Hand): https://lnkd.in/gVU6CYhh #ComputationalEfficiency #MachineLearning #EfficiencyAwareDesigns #LLMs #GenerativeModels #ResearchPapers

25

Modeling in the sciences is in fact almost always generative modeling. What a BOLD statement to start a paper with here --> 1.1 Motivation One major division in machine learning is generative versus discrimi- native modeling. While in discriminative modeling one aims to learn a predictor given the observations, in generative modeling one aims to solve the more general problem of learning a joint distribution over all the variables. A generative model simulates how the data is generated in the real world. “Modeling” is understood in almost every science as unveiling this generating process by hypothesizing theories and testing these theories through observations. For instance, when meteorologists model the weather they use highly complex partial differential equations to express the underlying physics of the weather. Or when an astronomer models the formation of galaxies s/he encodes in his/her equations of motion the physical laws under which stellar bodies interact. The same is true for biologists, chemists, economists and so on. Modeling in the sciences is in fact almost always generative modeling

1

1 Comment

Experimentation platforms need to have a symbiotic relationship between platform engineering and data science: I’ve been engineering experimentation platforms from scratch for the last 8 years. Even after building random allocation, data pipelines, and the scorecard, the work never stops there. The science around experimentation keeps growing. Think about how experimentation has grown so much from differences in means of randomized data. 1. There was an evolution of what treatment effect we are modeling. The average treatment effect, to the conditional treatment effect, to the time dynamic treatment effect. This evolution allowed us to analyze data segments, and look for patterns in the treatment effect over time, such as novelty effects or diminishing returns. 2. There was an evolution in inference. Bayesian testing is more and more popular, and so is always valid inference like the SSRM. 3. Then there was also an evolution in the type of experiment we run. From AB testing, to quasi experimentation, to multiarm bandits, to encouragement experiments. It’s hard to keep up with all these advancements in science. And it’s almost impossible to have an experimentation platform keep up in implementation. To have a truly stellar experimentation platform, you have to find ways to work smarter. Platform engineers should not be the only ones responsible for implementing these changes in experimentation. Data scientists want to be part of the discussion. Invite them and involve them! But how? It’s not so easy though. A production experimentation stack can be very complex, and the barrier to entry is frequently overwhelming to a data scientist. I believe we have to compromise. First, experimentation platform teams can offer a python library to perform metrics computation. Second, the team can offer a separate, stand alone library to perform causal inference on data that came from the first library. These libraries need to form the analytics stack for the experimentation platform. The beauty of this is that the libraries can be used for prod, but can also be used in local notebook environments. If you remove the overhead of the production environment, and enable the DS to use the metrics and inference libraries in local notebooks, they will be able to easily extend the libraries with new innovations. And when the libraries are extended, those new ideas can be merged upstream back into the platform. Our [slightly old] paper describes how we think about democratization, how to modularize for engineers and data scientists, how to reduce the needs for distributed computing so that DS can operate in local environments, and also how to scale it back for the production environment. I’ve been carrying that way of working for the past 6 years across 3 companies. I’d love to hear if this resonates with your team. https://lnkd.in/gsDBEmtX

101

5 Comments

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

386

12 Comments

NuExtract is a large language model developed by NuMind for structured information extraction from unstructured texts. Interestingly, this family of compact models often outperform larger models like GPT-3.5 and Llama3, setting new standards in the realm of data extraction. 🔧 How to Use NuExtract? * Define Your Schema: Create a JSON template that outlines the information you need. * Input Your Text: Provide the text from which you want to extract data. * Get Structured Output: NuExtract processes the text and outputs a structured JSON adhering to your schema. Try out NuExtract on HuggingFace - https://lnkd.in/gbsx-JTq Blog - https://lnkd.in/ghnaWB7j #DataScience #MachineLearning #AI #DataExtraction #NER #Innovation #NuExtract #TechTransformation

26

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15

The potential of Large Language Models to revolutionize manufacturing is explored in a recent paper by Yiwei Li et al., showcasing how LLMs can drive innovation and efficiency across the industry. With a focus on GPT-4V, the paper highlights its ability to execute complex instructions and extract insights from data, offering transformative opportunities. The applications are vast, from optimizing product design and enhancing quality control to talent management and supply chain optimization. The paper also delves into the impact on manufacturing education and the potential for LLMs to shape the industrial metaverse. As Yuval Avidani, a technical content creator, I find this research exciting, offering a glimpse into how LLMs can address challenges and drive sustainable growth in manufacturing. https://lnkd.in/dF_88WCw

2

Implementing Reinforcement Learning (RL) in the context of Large Language Models (LLMs) posed challenges, particularly in designing reward engineering and penalization components. In traditional RL, reward engineering is straightforward, often defined through a scalar reward function. However, implementing the same concept in RL with LLMs required a critic model that evaluates the questions and answers, critiquing them in a manner that simulates the reward function. A recent research paper on Natural Language Reinforcement Learning (NLRL) provides an intriguing perspective on the reward problem in RL by translating traditional RL concepts into natural language constructs. Here's a summary of how it addresses the reward problem: Language-Based Value Functions: Unlike traditional RL, which uses scalar rewards, NLRL employs language-based value functions. These functions articulate the effectiveness of policies and actions in a more interpretable, natural language format. This approach addresses the need for richer, more informative feedback than simple numerical scores. Iterative Refinement of Language Policies and Critics: NLRL incorporates a structured training pipeline where language-based policies and critics are iteratively refined using textual feedback. This method allows the system to handle the complexities of defining rewards in natural language, improving its understanding and evaluation of actions over time through textual feedback loops. Use of Large Language Models (LLMs): The framework leverages the advanced capabilities of LLMs to interpret, generate, and evaluate complex language-based information. This is essential for implementing language-based value functions and policies, enabling the system to generate and refine responses that align closely with desired outcomes. Distilling Enhanced Actions and Evaluations: NLRL employs a process where enhanced evaluations from the language-based value function approximator and improved actions from the language policy improvement operator are distilled. This allows for continuous refinement and learning. Compatibility with Existing LLMs: The framework is designed to be compatible with current LLMs, enabling it to leverage pre-existing language models to process and generate linguistic data effectively. This compatibility is crucial for translating the traditional reward engineering process into a language-based framework. The NLRL framework's introduction of natural language as a medium for representing and evaluating policies and rewards offers a novel solution to the reward problem in RL. By shifting from numerical reward systems to a more flexible, expressive, and intuitive system based on language, NLRL enhances interpretability and adaptability, paving the way for more diverse RL applications. Innovation Hacks AI Inc. #llm #refincorcementlearning #ai

4

We had a great session about NVIDIA NIMs with Anshul Jindal and Mark Moyou, PhD yesterday. 🚀 From Mark: We got lots of info on inferencing with NIMs and different types of NIMs like embedding models, generative models, reranking models.. 🎨 I walked through how you can use NIMs for embedding, retrieval and generation with #Haystack 🚀 And finally, Anshul did a live coding session where he showed how you can deploy and scale your own self-hosted NIMs. If you missed it, or don't even know what I'm talking about but you wanna learn, the recording is up (adding to the comments👇) All resources, including the example repos and guides are in the description. #AI #LLM #Nvidia #opensource

35

2 Comments

Not sure why LLM compression is not discussed more often. Leveraging very large language models to build capable smaller LMs will be a huge win for everyone. Companies like NVIDIA and Meta seem to be pushing hard on pruning and distillation for good reason. The compressed Llama 3.1 and NeMo models mentioned in this paper look very competitive.

471

7 Comments

What a great talk from Swapnil Sayan Saha, on Physics-Aware Tiny Machine Learning! https://lnkd.in/g9g4YWs4 There's exciting times ahead as Physics-aware TinyML is becoming increasingly important in merging the power of machine learning with operating in the physical world. By integrating real-world physics models into ultra-low-power devices, this approach dramatically enhances the accuracy and efficiency of TinyML applications. From improving energy efficiency in smart sensors to enabling more accurate anomaly detection in industrial applications, this approach to machine learning at the edge helps machines better understand their environment - without sacrificing power or performance. As we explore the possibilities, this combination of physics and Edge AI opens up new opportunities in fields like healthcare, environmental monitoring, and autonomous systems. It's a huge leap toward making AI more sustainable and impactful, and allows us to build more useful AI-enabled systems that operate in (and on!) the physical world. #TinyML #EdgeAI #PhysicsAwareAI #AIInnovation #Sustainability #MachineLearning

4

Big shoutout to Elizaveta Korotkova, our intern-turned-scientist, finishing her 2 years with Clarifai with a bang 💥 Our latest blogpost looks into the current benchmarks for Multimodal models. Read here: https://lnkd.in/dDbBFAUE

25

Excited to share our DistiLLM accepted at #ICML2024. DistiLLM is a better (performance), faster (training efficiency), and stronger (theorem) distillation method to significantly enhance the performance of SLMs. #efficientAI #slm

26

Going through the list of papers for #ICML2024 and #ICLR2024 and found some interesting ones that I'll share over the next few days. But here's the big one that got me hooked: Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs Paper: https://lnkd.in/dWkn_d_6 Authors: Yeonhong Park and others. Essentially, the paper proposes: ➡️ Rapid generation of multiple models at different quantization levels → Think about having 2, 3, 4, 5, 6, 7 or 8-bit models always in memory, but with a memory requirement of only an 8-bit model (well, almost, they do need some metadata but its an order of magnitude lower memory requirement). ➡️ Specialized LLM serving engine with the support to serve different quantization levels of the same model. ❓ Why is this important ❓ Imagine the scenario where you have a highly finetuned LLM (or an SLM) that you deploy for your users. As a failsafe, you also need to deploy a secondary LLM that comes into play if the first one goes down. However, as soon as the secondary LLM comes online, the users will notice - it's not a finetuned version of your first model! Or even if you deploy a finetuned version, you end up deploying TWO models, in memory, in parallel. This paper solves this problem - a possible failsafe method for an in-production LLM that failovers to smaller/higher quantization levels based on bandwidth and user requirements. Incredible! For a truly redundant system though this still doesn't work because if the LLM serving engine goes down, all models go down as well. So don't throw away your failsafe LLM just yet, but this might be a handy alternative. Book a free 15 minute call with me: https://lnkd.in/dGnHs_Mn #icml2024 #iclr2024 #llm #nlp #machinelearning #consulting #generativeai #openai #microsoft #iclr #icml #research #paper

12

Professor Lawrence is spot on in here. Most of the ML solutions that have served us well until now have less to do with the current hype on generative AI and more with a specialized domain knowledge combined with classical modeling techniques and a great deal of software engineering. Certainly what we call Generative AI today will have an important role to play as it is introducing a new way to interact with computers to do stuff for you. However thinking that it will solve all industry/society problems is a very big stretch. https://lnkd.in/e_UX_YUd

6