Matt McVicar

https://lnkd.in/epNxY4fb I've been thinking about the way that large language models seem to be able to generalise to unseen problems lately and I've been thinking that the spectral bias of neural networks is probably a key component of that. Between this and the double descent problem, it's becoming increasingly clear that massively overparameterized neural networks are biased towards simple and smooth solutions to a problem. Often, the simplest representation of a domain is also the 'correct' one. Consider a dataset that consists exclusively of uniformly sized black dots against a white background. The most compact encoding that contains all the information for this dataset is just the x and y co-ordinates of the dot. These values also happen to be the real generator of the dataset. It’d be reasonable to expect a beta VAE to converge on this representation based on spectral bias and it would be the most generic encoding. Is a similar thing happening in large language models? When presented with conversations and papers about scientific matters where the next word depends on what is true, perhaps the simplest policy is to understand the topic and to model reality first - language second.

3

📢 If you're working in relation extraction / character network domain, then the following post would be relevant. Excited to share the most recent milestone on releasing the ARElight 0.25.0 🎊 https://lnkd.in/exMgcWYb What is ARElight? It represents Granular Viewer of Sentiments Between Entities in Massively Large Documents and Collections of Texts. Shortly speaking, it allows to extract contexts with mentioned object pairs for the related prompting / classification. In the slides below we illsutrate the ARElight appliation for sentiment classification between object pairs in context. 🤖 We exploit DeepPavlov NER modes + GoogleTranslate + BERT-based classifier in the demo. The new update provide a series of new features: ✅ SQlite support for storing all the extracted samples ✅ Support of the enhanced GUI for content investigation. ✅ Switch to external no-string projects for NER and Translator 📜 Paper: https://lnkd.in/eNhYdyg9 🤗 More deatails: https://lnkd.in/e6zaZ7NK ⭐ Powered by AREkit: https://lnkd.in/g6jyWWqb #relationextraction #graph #sampling #nlp #lm #ner #processing #text #collection

1

Seems like a fascinating book. The review itself is written very succinctly. Learnt that laws of conservation ( momentum, energy, ) in physics is mathematically a result of symmetry. https://lnkd.in/gKTH3-2p

1

Three papers I've been thinking about lately. The first paper examines language modeling as compression, showing how large language models (LLMs) can achieve compression ratios far beyond traditional entropy-based methods by leveraging semantic context. LLMs, in essence, are compression engines. The second is a philosophy paper that rethinks understanding as the process of compressing information into compact, generalizable forms. Understanding, in this sense, is compression. The third brings these two ideas together, suggesting that LLMs rely on inductive reasoning to compress data further than classical methods. Understanding, again, is compression. While their titles do form the premises of a complete logical argument, I'm not fully ready to commit to any such bold conclusions here. So instead, I've simply placed these papers side by side as some, admittedly leading, food for thought. https://lnkd.in/eB8fzW8H https://lnkd.in/eKkf2nuv https://lnkd.in/eKkXu9js

3

5 Comments

It's tonight! Please join the PyData Cambridge community to welcome Mayank P. (Astra Zeneca) with his talk "Image Registration: How to align images" and Ehecatl Antonio del Rio Chanona (Imperial College London) with "Data-driven optimization for expensive problems: a chemical reactor case study". This event is free and in-person only. Please signup on Meetup (link below) and reach out if you have any questions or wish to contribute in any form. We are always looking for speakers, attendees, and funding (company sponsorships, personal donations, etc.). Please also follow us and share this event! - MeetUp (event signup): https://lnkd.in/eT3FRzt6 - LinkedIn: PyData Cambridge - Twitter: https://lnkd.in/eCr9BNT5 - Location (Map): https://lnkd.in/evutQgnY #llm #ai #datascience #dataengineering #cambridgeuk #sponsorship

15

Chatbots need tutors; Spot the deepfake; Generative AI threatens translators #generativeai

3

Working with some Large Language Models (LLMs) at Trionic Technologies LLP, Vatsal Patel asks a very interesting question: "Can we really exercise some influence or exploit LLMs to get focused answer from multiple choices?" On digging into it, we found a very interesting research paper that highlights the same! This paper, "Manipulating large language models to increase product visibility," shows how "strategic text sequences" could increase the visibility of the product in LLM-driven searching aids. Think about dropping a purposely designed message into a product description and seeing that one rise in the recommendation listing to the very top. Not only is it possible, but it's also now proven! Their research in optimizing these sequences employs techniques such as the Greedy Coordinate Gradient (GCG) algorithm to ensure that the very products, even if in real spaces, which were previously shrouded on digital catalogs, shine bright on the virtual shelf. This is almost revolutionary in SEO for AI-run search services in the sense of really changing the perception and priority of products by algorithms. The potential of businesses to harness this ethically is not less impactful and innovative in generating competitive edges. Let's see how strategic text sequences can help in innovations of digital marketing and e-commerce. Link: https://lnkd.in/dPE8zFtm #AI #MachineLearning #LLMs #DigitalMarketing #ecommerce

16

This week for Once a Maintainer we spoke with NLP expert Sofie Van Landeghem, core maintainer of the popular open source NLP library spaCy. Sofie shared her thoughts on the evolution of NLP as applied to text mining for industry, how spaCy manages its roadmap, and why dependency management in python is so difficult. https://lnkd.in/eW55VZbe

26

3 Comments

This clearly explains, how AGI cannot be achieved through LLMs, without understanding of enviornement there is no meaning behind the words inferenced by gpt. https://lnkd.in/dvFqU-BE

7

2 Comments

In The New Yorker Weekend Essay, Ted Chiang (who is one of my favourite authors) wrote about "Why A.I. Isn’t Going to Make Art" which is a great read. I have some disagreements with his ideas, he wrote - "When you are writing fiction, you are—consciously or unconsciously—making a choice about almost every word you type; to oversimplify, we can imagine that a ten-thousand-word short story requires something on the order of ten thousand choices. When you give a generative-A.I. program a prompt, you are making very few choices; if you supply a hundred-word prompt, you have made on the order of a hundred choices. If an A.I. generates a ten-thousand-word story based on your prompt, it has to fill in for all of the choices that you are not making. There are various ways it can do this. One is to take an average of the choices that other writers have made, as represented by text found on the Internet; that average is equivalent to the least interesting choices possible, which is why A.I.-generated text is often really bland. Another is to instruct the program to engage in style mimicry, emulating the choices made by a specific writer, which produces a highly derivative story. In neither case is it creating interesting art." When he anchors on the number of decisions made, he is just looking at one decision per word. But this is seldom true; even when we think of something, it is often just a few words which would be an apt choice to follow up with the previous word. When giving a prompt, I don't think we can presume we do a couple of decisions per word; it is not linearly related to the number of words in the prompt. There is richer information which can be embedded in the right prompts. Even more, I feel like a prediction model would probably be doing more decisions within the model, with, for example, the activations in an MLP layer. By this I mean that the model just doesn't take one decision per word predicted; there is much more than that. It is true that in current SOTA LLMs, there is blandness in the output due to the averaging and loss of information, but something like the "temperature" parameter already is a minor mitigation measure for this. It is true that we make a lot of decisions and there is novelty in it, but are all decisions important and are some a waste of time? I feel it is rather too quick to dismiss AI as being able to help create art, especially when augmented with partial human control in decision making. What are your thoughts on this? (Here's the link to the article: https://lnkd.in/ec7Gt8DP)

6

Proprietary datasets and the combination of AI and experimental data are the key to build successful companies.

1

The OP is wrong about the count, I think they are counting all models in HF as LLMs. Because there are only 700K+ total models in HF. But it sure begs the question why ~100K text generation models (if not LLMs). For perspective sentence-transformer library based models are only ~7K Related reads ⭕️ Elon’s idiot index for LLMs: https://lnkd.in/eWsbM45J ⭕️ How open is your LLM: https://lnkd.in/eY6JE7Ae

15

3 Comments

One of the best blogs I've come across on quantization, with outstanding visualizations. Highly recommend a read for anyone looking to understand LLM quantization in more depth! 📐 - https://lnkd.in/e_WTgq6x At Felafax AI (YC S24), we are building Infra to make full precision multi-GPU fine-tuning easier. #AI #Infra #LLM

16

3 Comments

I am pleased to share that our paper on Jaynes machine is now available online via Open Access: https://lnkd.in/e64_Jgja John Hopfield and Geoffrey Hinton, who are receiving the Physics Nobel next month, looked to physics for inspiration to model neural networks. They developed the Hopfield network and the Boltzmann machine by using the Ising model and energy minimization as their guiding principles. I went in a completely different direction, looked to game theory, and identified potential games as the starting point. In our framework, the neurons and the connections compete in an error-minimization game to increase their effective utilities during training and reach an arbitrage equilibrium. The outcome is a lognormal distribution of weights and neuronal iota. Our paper shows that this is empirically observed in real-life deep neural nets such as the LLMs. The Hopfield model and the Boltzmann machine cannot predict the distribution of connection weights as it is beyond their capabilities. We show that the Hopfield network and the Boltzmann machine are special cases of the Jaynes machine. One can think of the former two as the "ideal gas law" version of neural networks, whereas ours is the "van der Waals equation". Just like the real-world gases are not ideal and behave more like van der Waals gases, the real-world deep neural networks such as VGGnet, BERT, and LLAMA are not Boltzmann machines, but are more like Jaynes machines.

132

14 Comments

Some more insights into diffusion models, in particular the seemingly forgiving scaling laws regarding data requirements needed for accurately identifying subspaces of increasing dimensionality: https://lnkd.in/etB79gBU

12

Another great paper on In-Context Learning (ICL) for LLMs: https://lnkd.in/e4SP4K5d More and more, we're hearing from builders that ICL is an improvement over fine-tuning for dialing in domain/task specificity. Fine tuning is slow, expensive, and improves specialization at the cost of generalizability. On the other hand, an efficient ICL pipeline is quick and cheap to set up, boosts performance on specialized tasks without sacrificing on general ones, and can be iterated quickly as the project context/use case shifts. If any builders are still choosing fine-tuning over ICL, let's chat. I'd love to learn more!

15