Maunil Vyas

It’s been an honor working with Gil Weinberg! Thanks so much! This demo is from one of the chapters in my thesis on #AI driven raga agnostic automatic accompaniment for alapana (non-rhythmic improvisations) in South Indian Classical Music. Since the algorithm is raga agnostic, the system can accompany for any raga performed! More demos on a violin playing #robot coming soon! #MusicTechnology #MachineLearning #HumanRobotInteraction #HumanComputerInteraction #AutomaticAccompaniment

216

32 Comments

🔎 Demystifying LLMs: A New Video Series! 🔎 Struggling to wrap your head around Large Language Models (LLMs)? I feel you! That's why I'm launching a new video series, LLM Explained, in collaboration with Uthiralakshmi S. from Worcester Polytechnic Institute. Each episode will be a bite-sized 2-minute explainer, tackling a specific LLM concept. This week, we're diving into the foundations: tokenization and embedding! Whether you're a seasoned data scientist or just starting your AI journey, this series is designed to make LLMs accessible and engaging.✨ Stay tuned for the next episode! In the meantime, hit that follow button and join the conversation! If you learned something valuable, share it with your network! 👨‍💻 👥 #GenAI #LLM #LargeLanguaeModels #DeepLearning #NeuralNetwork #AI #MachineLearning #Tokens #Tokenization #Embedding #Education #AIEducation #Data #DataScience #NEU #WPI

18

2 Comments

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

46

2 Comments

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

386

12 Comments

🎉 Excited to Share a New Achievement! 🎉 I’m thrilled to have earned the NVIDIA Generative AI LLMs (NCA-GENL) Certification! 🚀 This certification validates my foundational understanding of developing, integrating, and maintaining AI-driven applications using Generative AI and Large Language Models (LLMs) with NVIDIA solutions. Over the past few months, I’ve delved deep into a variety of resources, including courses and webinars on topics such as Hugging Face's NLP, Building RAG Agents for LLMs, Diffusion Models, DeepLearning.AI's Prompt Engineering, and LangChain for LLM Application Development, to name a few. The journey has been challenging, but incredibly rewarding. I’m particularly excited about the knowledge I’ve gained regarding NVIDIA’s AI tech stack, including DGX, AI Enterprise, NeMo, NIM, and TensorRT. This certification not only strengthens my understanding of the current AI landscape but also equips me with the skills to contribute to developing cutting-edge AI applications. A big thank you to all the amazing resources and communities that helped me along the way. I look forward to applying this knowledge to future projects and continuing to grow in this exciting field! #NVIDIA #GenerativeAI #LLMs #Certification #AI #DeepLearning #MachineLearning #LangChain #HuggingFace

56

3 Comments

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 Comment

🚀 Excited to present our work on "AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations" at the #NeurIPS2024 Workshop on Adaptive Foundation Models! 📅 December 14, 2024, 10:45 AM 📍 West Exhibition Hall A 🔴 Problem: Multimodal web agents struggle to adapt to unseen tasks/websites/domains, relying on large-scale pre-training and fine-tuning, which limits efficiency. 🛠️ Methods: We introduce AdaptAgent, enabling agents to adapt with few-shot human demonstrations via in-context learning for proprietary models and meta-learning for open-weight models. 📈 Results & Importance: AdaptAgent boosts task success by a relative increase of 21.03% to 65.75% with just 1-2 multimodal demonstrations, offering a more efficient, adaptable solution for web agents. 📄 For more details, check out our arxiv paper: https://lnkd.in/g46s2Pys. This work is led by our intern Gaurav Verma during his time at J.P. Morgan AI Research, along with fellow collaborators Nishan S., Zhen Zeng, Tucker Richard Balch, and Manuela Veloso ❤️🤝 PS: We’re also looking for an intern in multimodal agents! 🤖 If you're passionate about multimodal systems, reach out (here or at NeurIPS 2024) or apply: https://lnkd.in/gtN6P5w9. #MultimodalAgents #FewShotLearning #JobOpening

7

"Are large language models superhuman chemists?" So what these researchers did was make a test -- a benchmark. They made a test of 7,059 chemistry questions, spanning the gamut of chemistry: computational chemistry, physical chemistry, materials science, macromolecular chemistry, electrochemistry, organic chemistry, general chemistry, analytical chemistry, chemical safety, and toxicology. They recruited 41 chemistry experts to carefully validate their test. They devised the test such that it could be evaluated in a completely automated manner. This meant relying on multiple-choice questions rather than open-ended questions more than they wanted to. The test has 6,202 multiple-choice questions and 857 open-ended questions (88% multiple-choice). The open-ended questions had to have parsers written to find numerical answers in the output in order to test them in an automated manner. In addition, they ask the models to say how confident they are in their answers. Before I tell you the ranking, the researchers write: "On the one hand, our findings underline the impressive capabilities of LLMs in the chemical sciences: Leading models outperform domain experts in specific chemistry questions on many topics. On the other hand, there are still striking limitations. For very relevant topics the answers models provide are wrong. On top of that, many models are not able to reliably estimate their own limitations. Yet, the success of the models in our evaluations perhaps also reveals more about the limitations of the exams we use to evaluate models -- and chemistry -- than about the models themselves. For instance, while models perform well on many textbook questions, they struggle with questions that require some more reasoning. Given that the models outperformed the average human in our study, we need to rethink how we teach and examine chemistry. Critical reasoning is increasingly essential, and rote solving of problems or memorization of facts is a domain in which LLMs will continue to outperform humans." "Our findings also highlight the nuanced trade-off between breadth and depth of evaluation frameworks. The analysis of model performance on different topics shows that models' performance varies widely across the subfields they are tested on. However, even within a topic, the performance of models can vary widely depending on the type of question and the reasoning required to answer it." And with that, I'll tell you the rankings. You can log in to their website at ChemBench.org and see the leaderboard any time for the latest rankings. At this moment I am seeing: gpt-4: 0.48 claude2: 0.29 GPT-3.5-Turbo: 0.26 gemini-pro: 0.25 mistral_8x7b: 0.24 text-davinci-003: 0.18 Perplexity 7B Chat: 0.18 galactica_120b: 0.15 Perplexity 7B online: 0.1 fb-llama-70b-chat: 0.05

3

GPT-4o says "9.11 is bigger than 9.9". It is extremely fascinating how AI models which can understand quantum mechanics, have read books from Tolstoy to Principia, and are now achieving super-human scores on knowledge benchmarks can fail at something as trivial as interpreting the decimal point system. It would almost be like if a top scientist made groundbreaking discoveries in physics and then said 1x1=2 *. These issues are at the heart of why AI models are considered unreliable. To build a world where AI models are ubiquitous, we need to figure this out ASAP. I think the most important step here is understanding WHY these mistakes happen. To this end, the recent interpretability work done by Anthropic will be crucial. https://lnkd.in/gvMKkGXX Further thoughts in comments. * Any resemblances to actual people are purely coincidental 😜 #AI #Anthropic #LLMs

58

28 Comments

There’s been a lot of discourse around why LLMs can or can’t solve math problems lately and I’ve been wondering why that’s so important to us (because LLMs suck at math.) TL;DR two reasons: 1. Using computers for math is something we’ve been doing since we invented computers and even before and 2. Doing math is not the goal, reasoning is. https://lnkd.in/eVkJQuXy

113

3 Comments

To(ring) or not to ring. That's the question we tried to answer. Last quarter, I along with Pranava Singhal and Harshvardhan Agarwal worked on a project to answer a simple question in Graph ML: Can adding virtual nodes to represent molecular rings improve performance on graph-level tasks? 🌟 Why do rings matter? Rings in Molecules play a role in determining their properties, like hydrophobicity, which influence solubility, and bioavailability. Inspired by this, we proposed a novel RingSAGE algorithm to explicitly tackle these structures during the message-passing stage in GNNs. 💡 Why This Matters Traditional GNNs treat molecular graphs uniformly, missing the unique properties of rings. Our approach introduces virtual nodes to capture ring-specific information. However, this leads to an increase in the number of learned parameters, so for a fair comparison, we compare our method against a variant of GIN with more parameters than our model. 🔍 What We Achieved Our model outperformed standard GNN models on the ZINC dataset, a widely used molecular drug solubility prediction benchmark. We provide a notebook to replicate these experiments, a codebase to use RingSage for your own usecase and a blog elaborating our method and results.

74

6 Comments

Ran a quick test on the new #OpenAI #o1mini to solve IIT-JEE Advanced 2024 maths questions. It accurately solved 13 out of 17 questions with an impressive accuracy of ~76%. Quick thoughts: 1. The hidden chain of thought is quite detailed and extensive. Breaks the problem, solves and verifies and iterates multiple times. This might increase the output token cost by a big margin. 2. Seems there are tools integrated for calculations. Much needed as #gpt4o does a bad job at it. 3. Latency is quite high. Some questions took more than 50 seconds to get answered. This needs to come down for production user-facing use cases. 4. Hallucination still exists. Check the 1st question in the link, it had the correct step-by-step reasoning but hallucinated at generating the final answer. 5. Overdoing self-thought. Saw a couple of instances where it did generate correct answers in the process but eventually made wrong answers. Would give a clearer picture once the rate limits are increased and benchmarking is done on a bigger sample. Let me know any interesting findings and thoughts 😀 https://lnkd.in/giSCezyG #GenAI #OpenAI #PhysicsWallah

139

5 Comments

❓ Struggling to keep up with generative AI’s rapid evolution—or wondering how to turn research into real-world impact? Join me and Ravin Kumar, Senior Research Data Scientist at Google Labs, for a live podcast recording where we’ll tackle these questions and more. 🎙 What we’ll cover: 📚 Generative AI at Google: How tools like Notebook LM bridge cutting-edge research with practical applications for technical and non-technical users alike. 🌍 Democratizing AI: The shift from training models to API-driven solutions, and the trade-offs between open-source and hosted tools. 🧩 Cultural & Practical Challenges: How to adapt generative AI tools for diverse global audiences and improve their user experience. 🛠 AI Roles & Personas: The evolving landscape of AI—from researchers fine-tuning models to developers building with APIs—and how to get started. 🏁 The Competitive Landscape: How companies like Google are pushing the boundaries of generative AI with new tools and features. This is your chance to hear from someone working on the frontlines of AI research and products—don’t miss it! Sign up here: https://lnkd.in/gwmzHunh

11

🔥 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐚𝐧𝐝 𝐋𝐋𝐌 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 - 𝐓𝐨𝐩 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 🔥 🤖 I've conducted a mock interview with my student, Bheesetti Anand, focused on Generative AI and LLM (Large Language Models) interview questions. This session is designed to benefit other students preparing for similar interviews, helping them secure their dream roles in the field of Generative AI. 📔 Here’s why watching the video can help you make a 𝐬𝐮𝐜𝐜𝐞𝐬𝐬𝐟𝐮𝐥 𝐜𝐚𝐫𝐞𝐞𝐫 𝐬𝐰𝐢𝐭𝐜𝐡 𝐭𝐨 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈: ✅ 𝐂𝐨𝐦𝐩𝐫𝐞𝐡𝐞𝐧𝐬𝐢𝐯𝐞 𝐂𝐨𝐯𝐞𝐫𝐚𝐠𝐞: Detailed exploration of top interview questions related to Generative AI and Large Language Models. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Real-world examples and practical applications to understand complex concepts better. ✅ 𝐄𝐱𝐩𝐞𝐫𝐭 𝐆𝐮𝐢𝐝𝐚𝐧𝐜𝐞: Tips and strategies to approach and answer interview questions effectively. ✅ 𝐂𝐚𝐫𝐞𝐞𝐫 𝐏𝐫𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧: Essential knowledge and skills to prepare for interviews and stand out to potential employers. 📽 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐥𝐢𝐧𝐤: https://lnkd.in/g_nkP7fs #generativeai #llm #interview #jobpreparation #mockinterview

14

2 Comments

Researchers at CMU Introduce TriForce: A Hierarchical Speculative Decoding AI System that is Scalable to Long Sequence Generation Quick read: https://lnkd.in/gr_he4kw Researchers from Carnegie Mellon University and Meta AI (FAIR) Introduce TriForce, a hierarchical speculative decoding system designed for scalable long sequence generation. TriForce utilizes the original model weights and dynamic sparse KV cache via retrieval as a draft model, serving as an intermediate layer in the hierarchy. Maintaining the full cache allows for superior KV cache selection using retrieval-based drafting, characterized as lossless compared to eviction-based methods like StreamingLLM and H2O. The hierarchical system addresses dual memory bottlenecks, pairing a lightweight model with a StreamingLLM cache for initial speculations to reduce drafting latency and accelerate end-to-end inference. TriForce introduces a hierarchical speculative decoding system with retrieval-based KV cache selection. The hierarchical system addresses dual bottlenecks, enhancing speed-up. Retrieval-based drafting segments the KV cache, highlighting relevant information. Lightweight models with StreamingLLM cache accelerate initial speculations, reducing drafting latency. TriForce utilizes model weights and KV cache to enhance LLM inference speed for long sequences. The implementation utilizes Transformers, FlashAttention, and PyTorch CUDA graphs, maintaining full layer sparsity while minimizing kernel launching overhead. Paper: https://lnkd.in/ga45hspw Machine Learning Department at CMU

26

Some folks asked if recordings would be made available for the LLM course I am teaching this quarter at UW, Seattle. The answer is yes! Sharing some resources here: Lecture recording: https://lnkd.in/gGCWh8dJ Lectures Playlist: https://lnkd.in/gb9Ns_Ai Course Webpage: https://lnkd.in/gX9_i9G6 #llms #largelanguagemodels #ai #teaching

75

Confounding? Detect it! A brand new paper on detecting and measuring confounding using causal mechanism shifts (presented this week at NeurIPS) In their new NeurIPS 2024 paper, Abbavaram Gowtham Reddy and Vineeth N. Balasubramanian (IIT Hyderabad) propose a framework for detecting and measuring confounding in a number of scenarios, based on shifts in causal mechanisms. The authors propose three complementary measures of confounding (with conditional and multivariate variants), designed to assess relative strengths of confounding in a variety of settings. The proposed measures complement one another depending on the available contextual information and offer a unified way to study confounding across diverse scenarios where data from different contexts is available. The authors generously shared Python implementation of their solution on GitHub. 💡 Paper: https://lnkd.in/dGXbnqAy 💡 Code: https://lnkd.in/dpczZvcU

142

7 Comments

Brain Teaser #10: Suppose we are fine-tuning a large language model using Low-Rank Adaptation (LoRA). We possess two low-rank update matrices, δWA​ and δWB​​, both associated with the rank "r". Considering our objective to preserve the model's current knowledge while incorporating new data from diverse features, what "mathematical rationale" should guide the selection of one of the update matrices (δWA​ or δWB) for low-rank adaptation? Hint: Consider the specific characteristics or properties of the matrices that can help in developing mathematical reasoning. Here is my response: The norm ratio, calculated as the ratio of the nuclear norm to the Frobenius norm of the update matrix, is an essential metric for assessing how updates impact a model. It provides insights into the model's stability and adaptability. Here’s a brief overview: https://lnkd.in/ekB_q9XX #MachineLearning #DeepLearning #NaturalLanguageProcessing #LoRA #FineTuning #AI #DataScience #ModelOptimization #NeuralNetworks #ArtificialIntelligence #TechInnovation #DataDriven #KnowledgeIntegration #Research #LLMs

30

4 Comments

Happy to share that our paper, "UniMTS: Unified Pre-training for Motion Time Series", has been accepted to the Neural Information Processing Systems (NeurIPS) 2024. This is the first pre-trained model specifically designed for human motion time series, excelling in zero-shot activity recognition across a wide range of wearable devices, regardless of body placement or orientation. Our approach follows a CLIP-inspired pre-training framework, where we leverage contrastive learning to align motion data with LLM-enhanced activity descriptions. https://lnkd.in/gjJcVSiR #NeurIPS2024 #neurips #neurips2024 #neurips24 #MachineLearning

93

9 Comments