Bryan Travis Smith - Ph.D / MBA

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

180

2 Comments

Alexandre Andorra, co-founder of PyMC Labs, joins Jon Krohn for an in-depth look at Bayesian statistics. Learn how Bayesian methods tackle complex problems, leverage prior knowledge, and optimize limited data using tools like PyMC, PyTensor, ArviZ, and Gaussian Processes. Don’t miss this in-depth episode! Watch here: https://bit.ly/sds793 Thank you to Crawlbase for supporting this episode of SuperDataScience, enabling the show to be freely available on all major podcasting platforms and on YouTube. #datascience #bayesian #pymc

7

Spatial Reasoning did not ‘emerge’ spontaneously in Large Language Models (LLMs) the way so many reasoning capabilities did. Humans have specialized, highly capable spatial reasoning capabilities that LLMs have not replicated. But every subsequent release of the major models — GPT, Claude, Gemini- promises better multimedia support, and all will accept and try to use uploaded graphics along with texts. #llms #chatgpt4 #gemini #llmops #aiml #ml https://lnkd.in/dFSWKqKs

2

🧠 What’s a Semantic Cache in RAG? Here’s What We’ve Learned Semantic cache is a time and cost saving method for Retrieval Augmented Generation (RAG). Instead of always running a full vector search, a semantic cache stores embeddings of frequently asked queries and their responses. This speeds up your RAG system by skipping redundant searches for similar queries. How do you implement it? Step 1: Store history of answered questions, their embeddings and the LLM response in a semantic cache table. Step 2: When a new query comes in, compare it against the cache table using vector similarity. Step 3: If there’s a match with score over a certain threshold, serve the cached response. If not, run a fresh retrieval and add it to the cache. Now, let’s talk vector search methods: 1️⃣ Cosine Similarity: Measures angle between vectors, works well for normalized data. 2️⃣ Euclidean Distance: Measures straight-line distance but can be skewed by vector magnitude. 3️⃣ Dot Product: Our favorite—it factors in vector size and performs best for our use cases. Want to see how semantic caching powers a RAG system in real life? Check out our demo at https://lnkd.in/g2jFuHTT. 🚀

2

🚀 Want to employ the power of #PyTorch for your scientific models without a steep learning curve? 👀 Check out our latest blog, where Travis Oliphant discusses how PyTorch's integration with the Array API makes it easy to use with familiar libraries like NumPy and scikit-learn.🔥 https://lnkd.in/dDRkEY3v #Innovation

10

- SwiftKV is a novel model transformation and distillation procedure (by folks at Snowflake) designed to optimize large language model (LLM) inference. Very cool findings in the paper (as presented in the vLLM meetup on 11/13). - SingleInputKV: Prefills later layers' KV cache using earlier layer outputs, reducing computation on prompt tokens. The key point is that the first few layers of attention in LLMs are similar (nice find), allowing SwiftKV's SingleInputKV mechanism to skip computations in later layers during inference. By using a single earlier layer's output to compute the KV cache for subsequent layers, it significantly reduces both computation (by 50%) and memory usage (by 62.5%) without sacrificing model performance. - AcrossKV: Merges KV caches of neighboring layers to minimize memory usage and increase batch size for higher throughput. - For models like Llama-3.1-8B and 70B, SwiftKV reduces prefill compute by 50%, memory usage by 62.5%, and achieves up to 2× higher throughput and 60% lower time per token. https://lnkd.in/guhEBW5G

62

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

Models like OpenAI's CLIP combine text and images for search, but they have limitations, particularly with longer texts and complex relationships. This article talks about JINA's new model that improves on CLIP by integrating both text and image embeddings in one model, making it faster and more efficient for multimodal search tasks. Experiments and Findings: → Search with Both Text and Images: Combining both embeddings leads to better and more diverse search results, improving relevance and variety in results. → Image-Driven Diversification: By using images for diversification, Jina-CLIP v1 can present a broader range of results, offering users more varied options. Results Comparison: → Using both text and image embeddings (via averaging) consistently outperformed searches based solely on text or images, providing more precise and diverse retrieval across both broad and fine-grained categories. https://lnkd.in/guNRQpBU

15

NeuralForecast 1.7.4 Released: Nixtla’s Advanced Library Revolutionizes Neural Forecasting with Usability and Robustness In a significant development for the forecasting community, Nixtla has announced the release of NeuralForecast, an advanced library designed to offer a robust and user-friendly collection of neural forecasting models. This library aims to bridge the gap between complex neural networks and their practical application, addressing the persistent challenges faced by forecasters in terms of usability, accuracy, and computational efficiency. NeuralForecast is positioned as a comprehensive toolkit that includes a variety of neural network architectures such as Multi-Layer Perceptrons (MLP), Recurrent Neural Networks (RNNs), Temporal Convolutional Networks (TCNs), and more sophisticated models like NBEATS, NHITS, Temporal Fusion Transformer (TFT), and Informer. This wide range of models ensures users can access state-of-the-art techniques for diverse forecasting needs.... Full read: https://lnkd.in/gVsTDTXd Code: https://lnkd.in/g6ThGxfP Docs: https://lnkd.in/gP2DSZWj

19

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

"Prompt Engineering" by Lee Boonstra. This paper covers everything from basic prompting techniques to advanced strategies like Chain of Thought (CoT), Tree of Thoughts (ToT), and ReAct, along with practical tips for code-related prompting. Key sections include: - Configuring output with temperature, top-K, and top-P settings - Using zero-shot, one-shot, and few-shot prompting effectively - System, role, and contextual prompting techniques - Best practices, including experimenting with input formats and adapting prompts to model updates https://lnkd.in/gH5xrivh

13

ICYMI - Yi Ding spoke about advanced RAG techniques at our South Bay Unstructured Data Meetup! 💪 He shared insights on a new approach to RAG: Small to Slide. This method enhances the performance of multimodal LLMs when dealing with visual data like presentations or documents with images. Read the recap: https://lnkd.in/esGnVFpc #RAG #Meetup #UnstructuredData #Event #AI #LLM #Vectordb #Milvus

28

1 Comment

Exploring non-linear relationships in regression? Transformations can lead to trillions of potential models, but is there a better way? Samuele Mazzanti's article discusses how gradient-boosting models can provide effective solutions with ease. #Regression #DataScience

8

1 Comment

Learn how an image-text multi-modality model can perform image classification, image retrieval, and image captioning #llms #llmops #image #opencv #chatgpt4 #gemini #aiml https://lnkd.in/gJFh9Ruf

1

💡 GPU? Lazy eval? No problem! 💻 Write dataframe code that works beautifully across pandas, Polars, cuDF & more with Narwhals. 🔥 Check out our in-depth blog, by Marco Gorelli on making data science more interoperable: https://lnkd.in/dedkSPPy #Polars #DataFrame #Pandas #Narwhals

14

I just published Vol. 80 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Comparing Commercial and Open-Source LLMs for Systematic Review Query Generation, from TU Wien 📚 A Systematic Comparison of Ranking Algorithms for LLM Evaluation, from the University of Michigan 📚 A Comprehensive Analysis of LLM Representations for Regression Tasks, from Google 📚 Benchmarking Multi-modal Retrieval-Augmented Generation for Web Content, from Ma et al. 📚 Efficient Large Language Model Scaling for Industrial Dense Retrieval, from Huang et al. 📚 A Survey on LLM-as-a-Judge, from Gu et al. 📚 Reproducing and Adapting 2D Matryoshka Training for Information Retrieval, from Wang et al. 📚 A Fast and Exact Similarity Search Engine for Billion-Scale Data Series, from Schafer et al. 📚 A Review of Visual Question Answering in the Era of Large Language Models, from SYSU 📚 Integrating ID Embeddings and Language Models for Enhanced Sequential Recommendations, from Huawei #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

32

1 Comment