Bjarne Stroustrup

Nearly a decade ago, I wrote out a complete and rigorous proof of Cantor's theorem that the set of sequences of natural numbers is bigger than the set of natural numbers...using only words of one syllable. https://lnkd.in/g4wWYVYc This is not the only time I revisited the result, much more recently I formally published a number of generalisations of this using different fundamental assumptions (for experts: taking Lawvere's fixed-point theorem and the contrapositive diagonal argument and showing how the proof works in a substructural type of logic). But this one-syllable-word proof was so much fun. I was reminded of this because someone got in touch with me, saying how it is something he periodically re-read for pure interest's sake, and asking if I could recover some material that was on the now-vanished Google+ to which I linked. I had in fact saved that discussion, which had contributions of similar pieces from a number of other mathematicians, including the Fields medal winner Tim Gowers. I will put a pdf of that thread somewhere public, soon, and link to it. But my correspondent was so taken by the exercise that he said he'd go so far as to purchase a physical book containing a collection of such "short words proofs", if one existed. The idea of publishing a slim volume of fun little proofs certainly is appealing, in some respects, because I love producing physical artifacts that reflect mathematics, but the market must surely be tiny!

7

2 Comments

Chaos Report: Iran's Response Matrix: A Game Theory Analysis Iran is in a classic decision-making bind: it has to retaliate but it must also be able to survive. However, there is already a problem: it is thinking too much and acting too late. More here on my Substack: https://lnkd.in/gQzQnAKj

10

1 Comment

In computational linguistics, researchers are focused on enhancing language models' ability to extract precise information from extensive textual data. Recent studies introduce "retrieval heads," specialized attention mechanisms that selectively focus on critical segments of large texts, improving targeted data retrieval. These heads, integrated into transformer-based models like LLaMA and Yi, significantly enhance accuracy and efficiency, particularly in handling long-context scenarios. The methodology involves thorough experiments across various models, including the Needle-in-a-Haystack test, which embeds specific information to measure precision. Results show models with active retrieval heads outperform those without, with accuracy dropping notably when these heads are masked. The empirical evidence underscores the effectiveness of retrieval heads in improving information retrieval precision and reliability. This research deepens our understanding of attention mechanisms in large-scale text processing, paving the way for more efficient and accurate language models. https://lnkd.in/dNNPvMXw https://lnkd.in/ddbHX6EA Innovation Hacks AI #aiinnovation #generativeai

1

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

Thanks Joe Morgan, MD I do take Yann LeCun ‘s words very seriously. He is right about building and deploying effective #generativeai solutions are not a trivial task. These do require more than just a jupyternote book and a Huggingface #LLM call as are being experimented by many startups. We are in the “dotcom” equivalent of bubble, with many naive but good experiments being run. Many of these will fail fast and fail forward. But then those who execute with a clear strategy, focus, scalable resources including a cohesive team will be successful. These are likely to be the next generation of Googles and OpenAIs which will have real commercially deployed solutions generating revenue. We at BrainX AI are focused on this key translational and strategy aspects of #genai deployment in healthcare, with an amazing multidisciplinary team. If you need any help, our team is available for consultation. Follow our work here: https://www.brainxai.com and on #brainxcommunity Frank A. Papay MD Jacek B. Cywinski, M.D.,FASA Kamal Maheshwari MD MPH FASA Ashish K. Khanna MD,MS,FCCP,FCCM,FASA Dr Avneesh Khare Chintan Dave, MD Raghav Awasthi Shreya Mishra Dwarikanath Mahapatra Julia Maslinski Rachel Grasfield Izabella DiRosa Edward Marx Avirup Guha, MD, MPH, RPVI, FICOS, FACC, FAHA Ty Vachon, M.D.

7

1 Comment

Did you know a Liger is a real animal (part lion, part tiger - h/t Animesh Singh)? It's also a killer open source project from some of my colleagues at LinkedIn, and can improve training throughput by 20% and reduce memory usage by 60% with just a single line of code for popular models like LlaMa, Gemma, and Qwen. With over 3.5K stars, and 350K downloads on GitHub since the team's August launch, you've got to check out the team's blog post to learn more. 🔥 🦁 🐅 #opensource #liger

11

1 Comment

https://lnkd.in/gGpqT8V4 - Chicago business media says the Illinois government is pressing hard to build a $20 billion, 150-acre quantum computing campus in Chicago. - The campus would be anchored by PsiQuantum. - The $20 billion figure refers to the potential for future business and government investments, as well as job creation. ‘Illinois may be on the verge of securing the largest technology project in its history—what is being labeled a “$20 billion, 150-acre quantum computing campus,” potentially anchored by Silicon Valley startup PsiQuantum, according to Crain’s Chicago Business. PsiQuantum, hot off an announcement that its receiving $600 million to build a manufacturing site in Australia, is reportedly considering two Chicago-area locations for the project, the business journal reports. ‘The proposed sites, the former U.S. Steel plant on the South Side and the former Texaco refinery in Lockport, are both under final review, with a decision expected soon. This initiative is part of a broader vision by Governor J.B. Pritzker’s administration, which pundits are referring to a modern-day Manhattan Project, to position Illinois as a leader quantum computing. ‘The initiative would initially create about 1,000 construction jobs and up to 500 direct jobs, including positions for hardware and software engineers. The project is designed to be accessible, with many positions requiring only associate’s degrees & specialized training. Additionally, the site would feature a cryogenic facility, essential for stabilizing subatomic particles used in quantum computing, which could cost between $300 million to $500 million. 'The selection of the former industrial sites is strategic, providing the necessary infrastructure for industrial-scale helium production, and the substantial water & power needed for the cryogenic process. The campus plans also include an R&D center to support the development of quantum [technology]. ‘The state’s investment in quantum technology is already considerable. Governor Pritzker’s administration previously allocated $200 million from Illinois’s capital budget to quantum initiatives, aiding in securing an additional $275 million in federal funding for local research projects. With another $500 million requested from the state legislature for further investment, Illinois is trying to make a push to become a federally recognized quantum hub, a designation that could bring up to $70 million in additional funding. 'The bet on quantum is not guaranteed. The Colorado quantum ecosystem is also making a play for that federal designation. Quantum computing, itself, is making profound progress toward practical uses, but it is uncertain as to exactly when that quantum practicality is achieved... There is also stiff competition within the quantum computing field with several quantum approaches being explored and exploited..'

15

2 Comments

This paper https://lnkd.in/eGaCaD7s shows that calibrating LLMs to the distribution of their training set always leads to hallucination, at a probability lower bounded by Good-Turing frequency estimation. Good-Turing frequency estimation https://lnkd.in/ewrh3RZC was developed to estimate the frequency of discovering a new species. Assuming the discovery of new species follows i.i.d., then its frequency can be reasonably estimated by the historical discovery rate, which is N_1 / N​, where N_1​ is the number of species observed only once (a.k.a., new discoveries), and N is the total number of observations. Now, calibrating LLMs forces them to emit only in-distribution facts (known species) and zeros out the probability of out-of-distribution facts (unknown species). Therefore, calibrated LLMs will hallucinate when the question is OOD, at a rate of N_1 / N​​. And there is no cure, as non-calibrated LLMs can be even worse. #artificialintelligence #machinelearning #deeplearning

208

18 Comments

#NVIDIA is effectively building AI infrastructure-as-a-service for the entire drug discovery industry. A new article in Genetic Engineering & Biotechnology News discusses Nvidia's launch of AI-based NIM™ Agent Blueprints, which are customizable workflows designed for enterprise-level users, including drug discovery teams, to create AI applications. These Blueprints aim to accelerate virtual screening of small molecules by incorporating generative AI models. Key features include the integration of cloud-native "microservices" (NIMs) that optimize the drug discovery process by shifting from traditional screening of fixed databases to generative AI-driven molecule design. While generative AI design of molecules is not a new topic, there are companies doing that, NVIDIA is offering it as a "plug-and-play" solution for corporate clients. This is "democratization" of AI in its pure form. And this will level the playing field to some extant. Obviously, plug-and-play mode won't replace full-scale "AI-first" platform R&D approach, but it can enable many interesting use cases for these off-the-shelf AI capabilities. Testing, piloting, and even building entire processes. The Blueprint integrates three key AI models: ✅ AlphaFold2 – predicts 3D protein structures from amino acid sequences. ✅ DiffDock – predicts molecular binding to proteins while optimizing properties like solubility and toxicity. ✅ MolMIM – generates drug candidates based on user-defined properties and optimizes molecule design. These tools help identify and optimize drug candidates ("hits") more efficiently, making the drug discovery process faster and less costly. Nvidia is also working on Blueprints that cover other stages of the drug development pipeline. What is interesting, is that global system integrators and technology providers like Accenture, Cisco, Dell, and AWS are working with NVIDIA to facilitate access to these Blueprints. So, this increasingly looks like NVIDIA is effectively building a new tech paradigm for AI adoption at scale. Now, you may wonder about molecular data privacy, now will pharma use that for billion-dollar projects? From what I found, the architecture of these Blueprints allows enterprises to host and deploy AI models locally, giving control over the infrastructure and data. This local or hybrid deployment model ensures that proprietary molecular data remains within the enterprise's own infrastructure, reducing exposure to external risks. In addition to localized hosting, Nvidia incorporates security frameworks like Nvidia Morpheus, which is a cybersecurity AI tool designed to identify vulnerabilities in large-scale data environments. Could be great to hear from someone at NVIDIA providing more context. #drugdiscovery #artificialintelligence #digitaltransformation Image credit: NVIDIA

161

20 Comments

Much has been said about many companies’ desire for more compute (and data) to train large foundation models. I think it’s under-appreciated that we also have nowhere near enough compute available for inference on foundation models. Years ago, when I was leading teams at Google, Baidu, and Stanford that focused on scaling up deep learning, many semiconductor makers, data center operators, and researchers asked me if AI would continue to make good use of more compute if they kept delivering it. For many desktop workloads, like running a web browser, a faster CPU doesn’t help much beyond a certain point. So do we really need faster and faster AI processors? Each time, I confidently replied “yes!” and encouraged them to keep scaling up compute. (Sometimes, I added half-jokingly that I had never met a machine learning engineer who felt like they had enough compute. 😀) Fortunately, this prediction has been right so far. However, beyond training, we are also far from exhausting the benefits of faster and higher volumes of inference. Today, a lot of LLM output is for human consumption. A human might read around 250 words per minute, which is around 6 tokens per second (250 words/min / (0.75 words/token) / (60 secs/min)). So it might seem there’s little value to generating tokens much faster than this. But in an agentic workflow, an LLM might be prompted repeatedly to reflect on and improve its output, use tools, plan and execute multiple steps, or implement multiple agents that collaborate. So, we might generate hundreds of thousands of tokens or more before showing any output to a user. This makes fast token generation very desirable and makes slower generation a bottleneck to taking better advantage of existing models. That’s why I’m excited about the work of companies like Groq, which can generate hundreds of tokens per second. Recently, SambaNova also showed it can hit hundreds of tokens per second. Incidentally, faster, cheaper token generation will also help make running evaluations (evals), which can be slow and expensive today since it involves iterating over many examples, more palatable. Fortunately, both training and inference are rapidly becoming cheaper. I spoke with Cathie Wood and Charles Roberts of the investment firm ARK, which is famous for its bullish predictions on tech. They estimate that AI training costs are falling 75% a year. If they are right, a foundation model that costs $100M to train this year might cost $25M to train next year. Further, they report that for “enterprise scale use cases, inference costs seem to be falling at an annual rate of ~86%, even faster than training costs.” I don’t know how accurate these specific predictions will turn out to be, but with progress in both semiconductors and algorithms, I do see training and inference costs falling rapidly. This will be good for application builders and help AI agentic workflows lift off! [Original text: https://lnkd.in/dJ9tVGh7 ]

2,894

80 Comments

💡 ByteDance's Latest Work on Neural Network-Based Molecular Force Fields 💡 I recently read ByteDance’s work on developing a new force field, and it’s pretty good! 👍 ByteFF is a molecular mechanics force field trained on a massive dataset: 2.4 million geometries and 3.2 million torsion profiles. It leverages GNN to predict bonded and non-bonded parameters, showing strong performance in predicting molecular geometry, torsion energy, and conformational energy. This model has great potential for drug discovery and molecular simulation. I do have some concerns about using torsion scan + hessian methods to train non-bonded parameters in this framework. Non-bonded interactions often involve long-range effects, which are challenging for GNNs to capture accurately. That said, the overall results presented in this work are still quite promising. It's exciting to see tech giants like ByteDance stepping into AI for Science, and their vast resources enable them to support DFT calculations on this massive scale. (🤞 Hopefully, they will release the dataset for open use, LOL!) 👉 You can find the paper here: https://lnkd.in/gjmGV4fd 👉 And check out their open-source framework on GitHub, which is well-engineered: https://lnkd.in/gkxcTi6t #AIforScience #DrugDiscovery #MolecularModeling #ForceField #ByteDance

49

1 Comment