Shahebaz Mohammad

How are Graph Convolutional Neural Networks (GCNNs) a generalisation of Convolutional Neural Networks (CNNs)? My next blog in the Graph series on "Deep Learning with Graphs" is on the way (well, it's already written on paper) and it's going to be a blockbuster. The beauty of Graph Machine Learning is that the more you read about it, the more you feel that your neurons (real ones!) are learning to think better! And you're already in love with it! Coming to the point: One might recall the convolution operations in a CNN. You have a grid of pixels representing an image. You have a 3 x 3 filter. You slide the filter over the image and carry out aggregation to obtain a new pixel value. You proceed in the same fashion over the whole grid, sliding towards the right , bottom, carrying out the convolution operation and finally getting a new transformed, reduced image through the series of aggregations. You repeat this through the different convolutional layers which together form a Convolutional Neural Network Now think of a Graph. It is not simple grid like an image. The topology of a Graph is arbitrary. There is no notion of left, bottom, right in a Graph like in an image which is represented through a grid of pixels. You cannot slide over a Graph as simplistically you did and performed the convolutional operations in an image. How do you approach the problem? How do you carry out convolutional operations in a Graph? Now take a node in the Graph. Take all the nodes which are the immediate neighbours of this node. The convolutional operation here is: You aggregate the information from all the neighbours and pass it to the node. The information is nothing but the emedfings of the neighbour nodes. You add together the emedding of the node under consideration. You transform the aggregated embedding by multiplying with a weight Matrix and transform the node's own embedding by multiplying with a different weight matrix. This is the typical linear transformation operation in a NN. You apply non linearity - the activation function like ReLu. Now think about a CNN taking the same grid and considering a central pixel. Consider the rest of the pixels around the central pixel. You are essentially doing the same thing: taking information from the neighbour pixels along with the central pixel, aggregating, as I described above for the node in the Graph. Hence a Graph Convolutional Network is a generalisation of CNN The story goes, you do the same for every node on the Graph. Each node in the Graph represented through a different neural network architecture! This is amazing 🔥🔥🔥💓💓💓 #graphs #machinelearning #deeplearning My Technical Blogs Library: 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on: Data Science, Data Engineering, Finite Element Analysis, Non Linear Finite Element Analysis, Fatigue, Fracture Mechanics

٢٢

LLMs are game changers. Very early on, before the LLM hype, I realized that language models would fundamentally change human-computer interaction. That’s why we developed an AI assistant at OrganAi.se before the LLM hype and were ready once the technology got available. We realized our vision of building an affordable personal assistant! Of course, the product developed much more than we originally planned in every single way. One thing is for sure, you can’t build an AI product that users want, if you don’t build a product that people want even without AI 🤖.

١٩

٣ تعليق

This is probably the best in-depth "musing" on building a Generative AI product I've ever read so far! Some notable learnings that I have picked up in this article: - There is trade-off of speed of development vs consistency in UX when parallelizing tasks across multiple teams to build different independent AI agents - Building the RAG pipeline (80%) was a breeze: they built dev sets and fitted them with prompt engineering and in-house models, but the generation bit (20%) was tough - In-house defensive YAML parser; re-prompting LLM to correct its YAML output mistakes adds non-trivial latency + GPU calls - "Poor man's fine-tuning" with Embedding-Based Retrieval (EBR) with in-memory DB to inject response examples into the prompts. Link in the comment

٣

١ تعليق واحد

GPT-4o says "9.11 is bigger than 9.9". It is extremely fascinating how AI models which can understand quantum mechanics, have read books from Tolstoy to Principia, and are now achieving super-human scores on knowledge benchmarks can fail at something as trivial as interpreting the decimal point system. It would almost be like if a top scientist made groundbreaking discoveries in physics and then said 1x1=2 *. These issues are at the heart of why AI models are considered unreliable. To build a world where AI models are ubiquitous, we need to figure this out ASAP. I think the most important step here is understanding WHY these mistakes happen. To this end, the recent interpretability work done by Anthropic will be crucial. https://lnkd.in/gvMKkGXX Further thoughts in comments. * Any resemblances to actual people are purely coincidental 😜 #AI #Anthropic #LLMs

٥٨

٢٨ تعليق

Ran a quick test on the new #OpenAI #o1mini to solve IIT-JEE Advanced 2024 maths questions. It accurately solved 13 out of 17 questions with an impressive accuracy of ~76%. Quick thoughts: 1. The hidden chain of thought is quite detailed and extensive. Breaks the problem, solves and verifies and iterates multiple times. This might increase the output token cost by a big margin. 2. Seems there are tools integrated for calculations. Much needed as #gpt4o does a bad job at it. 3. Latency is quite high. Some questions took more than 50 seconds to get answered. This needs to come down for production user-facing use cases. 4. Hallucination still exists. Check the 1st question in the link, it had the correct step-by-step reasoning but hallucinated at generating the final answer. 5. Overdoing self-thought. Saw a couple of instances where it did generate correct answers in the process but eventually made wrong answers. Would give a clearer picture once the rate limits are increased and benchmarking is done on a bigger sample. Let me know any interesting findings and thoughts 😀 https://lnkd.in/giSCezyG #GenAI #OpenAI #PhysicsWallah

١٣٩

٥ تعليق

Enhancing the accuracy of Large Language Models (LLMs) is a key focus in ongoing AI research. Previously, we have used a Critic model along with Reward Engineering to identify incorrect responses and incrementally fine-tune the LLM. The introduction of Thought Preference Optimization (TPO) advances this approach by having the LLM generate preliminary thoughts which a judging model then refines into better responses. This method is akin to the human practice of "thinking before responding," a critical safeguard in high-stakes situations, pushing LLMs to adopt a similar evaluative process. Training an LLM to pause, generate internal thoughts, and evaluate these before delivering a final response is complex and resource-intensive. It typically requires large datasets of human-annotated thoughts, which are scarce across diverse domains. Chain of Thought (CoT) prompting is another method where the model is instructed to articulate intermediate reasoning steps, structuring its approach to tasks. While CoT has proven effective in fields like mathematics and logic that require clear reasoning steps, it often underperforms in more subjective areas such as marketing or creative writing. This limitation is exacerbated by training datasets that generally contain human responses without the underlying thought processes, restricting the model’s ability to refine its reasoning across varied domains. Thought Preference Optimization (TPO) aims to equip LLMs with the capability to generate and refine internal thoughts before producing a response. Crucially, it operates independently of human-annotated datasets. It begins by instructing the model to divide its output into two distinct parts: the thought process and the final response. The model generates multiple thoughts for each instruction, which are then evaluated through performance optimization to select the best thought-response pair for further training iterations, allowing the model to gradually enhance its reasoning capabilities. TPO employs a reinforcement learning technique, enabling the model to learn from its thought process. The model generates preliminary thoughts before responding, and a judging model evaluates the resulting responses. This iterative refinement process leads to more thoughtful and accurate answers. The effectiveness of TPO was tested on two benchmarks: AlpacaEval and Arena-Hard. On AlpacaEval, TPO achieved a win rate of 52.5%, outperforming the direct response baseline by 4.1%. Similarly, it registered a win rate of 37.3% on Arena-Hard, surpassing traditional methods by 4.3%. One of the most significant findings is that thinking-based models generally outperformed direct response models across various domains. Even in non-reasoning tasks like creative writing, TPO-enabled models could plan their responses more effectively, leading to better outcomes. https://lnkd.in/d5kVzjxv #llm #ai

٣

١ تعليق واحد

Anthropic recent blog on #Agentic #patterns outlines a compelling framework for building AI agents, emphasizing the beauty of simplicity in design. However, reducing these patterns to mere LLM calls might oversimplify the complexity involved. The cognitive load of an LLM can extend far beyond a single function call, mirroring the intricate decision-making and problem-solving capabilities of a human brain. Such agents need to handle multi-step processes, handle perception, maintain context over extended interactions, and adapt dynamically to new information, much like a human would. These aspects suggest that while the foundational principles of agentic patterns are sound, the implementation in real-world scenarios requires a nuanced understanding of how AI can mimic human #cognitive processes, potentially leading to more robust and versatile AI applications. https://lnkd.in/dCehVmzF

٣٣

٧ تعليق

The possible downgrading risks for NVIDIA (or any similar high-growth stock) typically stem from a mix of internal and external factors. Here are the key risks: 1. Valuation Concerns High P/E Ratio: NVIDIA's P/E ratio is significantly above industry averages, raising concerns that the stock is overvalued. DCF Valuation Gap: If actual growth doesn’t meet expectations, analysts might downgrade their price targets. 2. Macroeconomic Factors Interest Rate Hikes: Rising interest rates can hurt growth stocks as future cash flows are discounted more heavily. Recession Risks: Economic slowdowns could reduce demand for NVIDIA’s products in gaming, AI, and data centers. 3. Industry-Specific Risks Semiconductor Cycles: The semiconductor industry is highly cyclical, and any downturn could lead to reduced revenue. Competition: Intense competition from AMD, Intel, and emerging players in AI chip markets could erode market share. Supply Chain Challenges: Global supply chain issues, including chip shortages, could hurt production and profitability. 4. Regulatory Risks Geopolitical Tensions: U.S.-China relations could impact NVIDIA’s revenue from China, a significant market for its products. Government Regulations: Stricter export controls or antitrust regulations might restrict growth opportunities. 5. Technological Risks Innovation Pressure: Failure to stay ahead in AI, GPUs, or other high-tech sectors could lead to downgrades. Customer Concentration: Heavy reliance on a few large customers (e.g., cloud service providers) exposes NVIDIA to risk if these relationships weaken. 6. Execution Risks Overestimated AI Growth: While AI is a strong growth driver, over-optimistic forecasts could lead to disappointment. Mergers and Acquisitions: Failed or costly acquisitions (e.g., ARM deal termination) can impact investor sentiment. 7. Market Sentiment Profit-Taking: If NVIDIA’s stock has experienced a significant rally, any bad news could trigger profit-taking. Short-Term Earnings Miss: Missing quarterly earnings expectations can lead to immediate downgrades. Mitigation for Investors Diversify portfolios to balance sector risks. Stay updated on earnings reports and industry trends. Monitor macroeconomic indicators and geopolitical developments. Considering these risks, NVIDIA's long-term potential remains strong, but investors should prepare for possible downgrades during periods of market volatility or sector corrections. NVIDIA

٢

١ تعليق واحد

No amount of scaling (throwing billion$ for computational resources & trillions of words) would help LLMs achieve the two key cognitive capabilities that make human intelligence so special: 1. Learn - Autonomously, Adaptively, Conceptually, Contextually & Incrementally. 2. Reason - Conceptually, Contextually & Metacognitively We discussed the criticality of this in greater detail here: https://lnkd.in/ggZm_W-w It is for this reason, LLMs can never get us to AGI: https://lnkd.in/d3vEPJ2P So, what will get us to AGI: https://lnkd.in/ggFTxuUz It's no wonder Demis Hassabis, CEO of Google DeepMind said this about LLMs Intelligence on July 9, 2024 at the Future of Britain Conference hosted by Tony Blair: “We’re still not even at cat intelligence yet, as a general system,” https://lnkd.in/gtyQNZyc It is great to see Yann LeCun arriving at the same conclusions about LLMs. https://lnkd.in/gm9Nfe4P

٣٣٩

٦٧ تعليق

Perhaps the most critical thing in a Retrieval Augmented Generation (RAG) system is the retrieval or searching strategy. Poor retrieval will lead to irrelevant documents going to the LLM leading to irrelevant generated answers or worse hallucinations. Is cosine similarity everything? Do check out below illustration which talks about four of the most common strategies for playing around with retrieval strategies in RAG systems: 1. Similarity-based retrieval: This is based on embedding similarity, you can play around with different metrics like cosine, maximal margin relevance etc. 2. Similarity with Threshold retrieval: Useful where we cap the similarity threshold and can help in removing irrelevant document but can drop recall 3. Multi-query retrieval: Make the LLM generate variations of your query and get documents based on these multi-versions of your initial query, often useful to get more recall or coverage of documents 4. Reranker retrieval: Use a model specialised in ranking documents based on search relevance Often you can combine some of these techniques and even use ensemble retrievers. Do explore them and go beyond just cosine similarity!

١٧

Top 5 Pre-trained Word Embeddings for NLP Teaching machines to understand text is no easy feat. While humans excel at interpreting languages, computers often struggle. This is where word embeddings come into play, transforming text into numerical representations that machines can process. What are word embeddings? Word embeddings are dense vectors representing words. They capture meanings, relationships, and contexts in which words are used, enabling efficient text processing by machines. Why use pre-trained word embeddings? Pre-trained word embeddings leverage large datasets to enhance Natural Language Processing (NLP) models. They save time and computational resources while capturing both the connotative and syntactic meanings of words. This is especially beneficial for tasks with limited training data or extensive vocabulary. Top 5 Pre-trained Word Embeddings: 1: Word2Vec Created by Tomas Mikolov at Google, Word2Vec represents words as vectors based on their contexts. Trained on the Google News dataset, it uses Continuous Bag-of-Words (CBOW) and Skip-Gram models to learn word relationships. 2: GloVe Developed by Stanford, GloVe (Global Vectors) uses global co-occurrence data to build word vectors. It effectively captures semantic similarities between words by analyzing their co-occurrence probabilities. 3: fastText A product of Facebook AI Research, fastText enhances word embeddings by breaking words into character n-grams. This approach improves generalization and requires less training data, supporting multiple languages efficiently. 4:ELMo Created by Allen NLP, ELMo (Embeddings from Language Models) generates context-sensitive embeddings from bidirectional LSTM states. It excels in handling polysemous words by understanding their usage within phrases. 5: BERT Developed by Google, BERT (Bidirectional Encoder Representations from Transformers) uses transformers for bidirectional language modeling. It pre-trains on vast text corpora, then fine-tunes for specific NLP tasks, significantly improving performance. Understanding these pre-trained word embeddings can give you a competitive edge in NLP. They capture the rich semantic and syntactic nuances of language, enhancing the performance of your models. Feel free to connect with me for more insights on NLP, Data Science, Machine Learning, and Deep Learning! #NaturalLanguageProcessing #DataScience #MachineLearning #DeepLearning

٤

Rohit Saha, Benjamin Ye, Kyryl Truskovskyi and I will be hosting a free hands-on workshop on practical fine-tuning and evaluating LLMs at TMLS! Check it out if you're interested in customizing LLMs for your use case and scaling your fine-tuning process 🚀

٢٧

The journey of AI engineering can be humorously captured in one meme: 🎮 Back Then: Countless hours spent manually fine-tuning models, debugging hand-crafted algorithms, and celebrating after days of compute power finally produced a working RNN. It was a tough slog, but those wins? Absolutely exhilarating! 💥 🤖 Now: A quick fine-tuning on a large language model, a few clever prompt tweaks, and boom—you’re an AI trailblazer. 🚀 Prompt engineering has become the new creative frontier. So, what’s the verdict? Have we oversimplified the process, or is this the long-awaited democratization of AI? #artificialintelligence #promptengineering #finetuning

٣٧٢

١٤ تعليق

This is a major development for the open-source AI ecosystem. Apple has gone fully open-source! They have just released DCLM, a 7B open-source LLM, including weights, training code, and dataset. They plan to open-source all their models for deployment across their billion devices. Apple's embrace of open-source marks a significant shift. Here’s a summary: → Apple has released and open-sourced their DCLM 7B model, outperforming Mistral/Qwen2/Gemma on various benchmarks. → It is primarily trained on English data with a 2048 context window, and licensed under the Apple Sample Code License. → The release includes an exceptional explanation of the data curation process, offering valuable insights into LLM training. → DCLM is trained on 2.5T tokens using the DataComp-LM (DCLM) framework, with datasets from Common Crawl. → DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments aimed at improving language models. → The DCLM Framework is designed for future experiments with data curation strategies like deduplication, filtering, and data mixing. Why is this significant? Apple's AI strategy now includes open-sourcing all their models. This shift, alongside their recent on-device AI announcements at WWDC, indicates a move toward smaller models and open AI. The future of AI is looking increasingly open. Resources: Model: https://lnkd.in/grgNegcN Repository: https://lnkd.in/gzcTsxqY Dataset: https://lnkd.in/gqY_U2x5

٤٦

LLMs - Function Calling In one of my previous posts i have talked about function calling with LLMs. I have attached the code snippet to illustrate function calling using a mistral model. The gist of the code is i have asked about the current stock price of a company to a LLM, which it can't answer. Now with the help of function calling leveraging yahoo finance, the LLM can provide the stock price. Invoking mistral model needs an api key which can be obtained by having a mistral subscription.

٢

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

٢٨

🚀 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗮𝘁 𝗪𝗮𝗹𝗺𝗮𝗿𝘁 🚀 I recently came across an insightful paper on Semantic Retrieval at Walmart that showcases a cutting-edge hybrid system combining traditional inverted index retrieval with neural retrieval to enhance search relevance, especially for complex "tail queries." Here’s a quick summary of the key highlights:  🔍 Hybrid Retrieval System:   The system integrates the strengths of traditional inverted indexing (great for rare tokens) with neural retrieval (bridges vocabulary gaps for tail queries). This hybrid approach significantly improves search relevance.  🤖 Neural Model Training:   The authors introduced a novel method for selecting negative examples during training, which is crucial for improving model performance. They also proposed an approximate metric to evaluate the system’s effectiveness.  📊 System Architecture:   The architecture includes a Query Planner, ANN Fetcher, and a re-ranking system. The Query Planner generates both a query plan for inverted-index retrieval and a query embedding for neural retrieval, ensuring a seamless and efficient search process.  🧠 Semantic Model:   The model uses a two-tower BERT-based architecture to generate embeddings for queries and products. This approach outperformed traditional bag-of-words models, especially when incorporating product attributes like category, brand, and color.  📈 Data & Labeling:   The model was trained on Walmart engagement data, with a clever labeling strategy that assigns scores based on user interactions (purchases, clicks, impressions). This helped the model better understand user intent.  🎯 Negative Item Selection:   The paper explored various strategies for selecting negative items during training, including in-batch negatives and hard negatives, which proved essential for boosting model performance.  ⚙️ Model Optimization:   To reduce model size and improve efficiency, the authors experimented with linear projection and smaller transformer architectures. They found that a 256-dimensional embedding struck the right balance between performance and storage costs.  🏆 Re-Ranking Stage:   A Gradient Boosted Decision Tree (GBDT) was used to rank the merged results from both retrieval systems, incorporating features like query attributes, product details, and query-item interactions.  💡 Key Learnings:   - Inner Product vs. Cosine Similarity: Inner product was more stable during training but harder to optimize for ANN indexing.   - Blending Features: Adding product descriptions didn’t improve performance, likely due to noise.   - Model Complexity: Deep models or large embeddings weren’t necessary for top performance, as queries and product titles are relatively simple semantically.  #LLM #RAG #NLP #ML #AI

٤٣

٣ تعليق