Gleb Bahmutov

My bets where AI won't replace humans: 1) Human connection: Like the one that many of us just had at Config. 2) Shared human experience: When you look at another person's eye and you know you've been through a similar joyful and painful journey. 3) Real stories behind art and culture: I listen to music, and I like paintings not just because they feel good, but because there's a real human story behind it that speaks back to 2. 4) Ingenious net new creative fields: AI can use past things to create something new by mixing them, but it's mostly always can't really create something completely new and unseen. 5) What's your bet?

34

4 Comments

Opinion piece by me: What wearables are getting wrong about sleep tracking. - The big wearable feature that's lacking accuracy - Real-world examples of when data goes wrong - Why data presentation is never neutral – and it's always "loaded" with meaning for users - James Hewitt - Where sleep tracking can improve #sleep #sleeptracking #wearables #health #wellness

33

5 Comments

Anthropic's latest Claude AI model is taking human-AI interaction to a whole new level! 🖥️🤖 The new Claude 3.5 Sonnet can now use computers just like a human would - by looking at the screen, moving the cursor, clicking buttons, and typing text. This is a major leap forward in AI capabilities. Key features: Operates based on screenshots, emulating human computer use Can perform complex multi-step tasks across different applications Self-corrects and retries when encountering obstacles2 Accessible via API for developers to build with Why it's game-changing: This opens up incredible possibilities for AI assistance in data entry, form filling, software testing, and automating repetitive computer tasks13. Imagine having an AI assistant that can seamlessly work alongside you, handling tedious computer work while you focus on higher-level tasks. While still in beta, this technology demonstrates how AI is rapidly advancing toward more human-like interaction with digital interfaces. As it develops further, it could revolutionize how we work with computers and boost productivity across industries. The future of human-AI collaboration just got a lot more exciting! What do you think about this new capability? #AIInnovation #FutureOfWork Here is a video about it: https://lnkd.in/gh8aNYEu

6

Notes upon reading the licence for what Meta promotes as its "open source" Llama 3.1 model. Licence - https://lnkd.in/eQ6Kxi4X Open source PR - https://lnkd.in/ebmsZ5ui 1. Background The widely acknowledged definition of "open source" software is OSI's - https://opensource.org/osd In outline, this requires liberty for anyone to use, study, modify and redistribute source code for any purpose. To illustrate what it means in practice, see the Apache 2.0 licence - https://lnkd.in/ex-AAs4j Apache 2.0 does have two minor requirements: (1) you must put a notice about it in your code (2) your licence terminates if you sue the licensor alleging the work constitutes patent infringement. These are accepted by OSI as consistent with open source. 2. Open source AI There's an ongoing debate about modifying the OSI definition for AI models. There are some issues, but the freedoms for anyone to use, study, modify and share for any purpose are still emphasised - https://lnkd.in/emH_A-rm 3. Open-washing A pejorative term for claiming something is open source when it's not 4. Llama 3.1 licence (23 July 2024) Section 6 allows Meta to terminate for any breach. No materiality requirement, no notice period. There are lots of ways you can be in breach - - Section 1(b)(i) and (iii) - attribution requirements more stringent than those in Apache 2.0 - Section 1(b)(iv) - you must comply with "applicable laws and regulations" including trade ones. Presumably will be argued to mean applicable to Meta or Llama 3, even if you are outside the US. Go figure what might happen. - Section 1(b)(iv) - you must comply with Meta's use policy - https://lnkd.in/epFcwj9m -- Much of it prohibits things no decent human would do (though some are things certain social media companies have been accused of) -- But it's important to read closely as any breach of any part lays you open to termination. For instance, a duty to disclose "appropriately" any known dangers of your use to end users. Confident on that? -- A reference to "the most recent" version of the policy seems to suggest Meta seeks to reserve the right to change it - Section 2 - you can't use it in products for which you have 700m MAUs on the Llama model release date Draw your own conclusions about - whether that's "open source" - the relevant risks. Interestingly, the licence itself doesn't use the word "open source." Indeed, section 1 describes it as "limited." But the PR is very different. Note that it's all subject to the laws and exclusive court jurisdiction of Meta's home state. Edits: p.s. The image is from v3 (Apr 2024) but I've updated the text and links to refer to v3.1 dated 23 Jul 2024. See Michael Bommarito's and my comments below. The main difference is that v3.1 drops v3's ban on using Llama to improve another LLM. p.p.s. Also see the potentially expensive indemnity in the final sentence of section 5(c) p.p.p.s. On attribution, see the comments of Gayle Gorvett and responses

26

25 Comments

Also, we released a new version of the public beta, which is available in Creative Cloud in the "Beta Apps" section under Applications. No new features this time--we are still iterating on the Assets Library Panel and "Search by Shape" feature, as well as the "Smart Upres" feature. If you haven't tried either of these, please do and let us know what you think! Here is how you can access the beta: https://lnkd.in/gq3uqEdm

3

I once stayed up for 72 hours straight fixing a “small bug” in our webhook code that had shut down SuperCopy's entire signup process. 🐛💻 (Narrator, in the voice of Werner Herzog): "In the vast digital wilderness, where lines of code stretch endlessly like the Amazon rainforest, our protagonist embarks on a Sisyphean task. The irony is not lost - a mere punctuation mark, so insignificant in the grand tapestry of human communication, becomes the linchpin of technological chaos." I was living the high life – hunched over my laptop, eyes bloodshot, muttering incoherently about JSON syntax. Here's how it went down: Day 1: "It's just a small bug. I'll have it fixed in an hour." (Narrator): "Ah, the naive optimism of youth. Little did he know that this 'small bug' would become his white whale, his Everest, his Aguirre's quest for El Dorado." Day 2: "I'm close. I can feel it!" (Narrator): "The madness begins to set in. The line between reality and code blurs. Is he chasing the bug, or is the bug chasing him?" Day 3: "Is that the sun rising? Again? Wait, what day is it?" (Narrator): "Time loses all meaning in this digital purgatory. The sun rises and sets, oblivious to the cosmic struggle playing out on a 15-inch screen." Finally, after what felt like a lifetime of staring at code, I found it… A single, mischievous period hiding where it shouldn't be. One tiny dot causing chaos in our entire system. I deleted it, ran the code, and... it worked. 🎉 (Narrator): "In this moment of triumph, we are reminded of the fragility of our digital existence. A single character, no larger than a speck of dust, holds the power to bring empires crumbling down or to raise them to unimaginable heights." I'd love to say I celebrated with champagne, but I think I passed out on my keyboard. Sometimes the biggest problems have the smallest solutions. ALWAYS double-check your punctuation. Has anyone else had a seemingly trivial issue spiral into a multi-day debugging marathon? Share your war stories in the comments! 👇 #StartupLife #CodeDebugging #TinyBugBigProblem #SuperCopyAI

24

3 Comments

If you're wondering how to use AI and events to enable real world workflows like podcast production, my colleague Sean Falconer has written a great blog post showing you how to do this.

4

I've always believed that LLMs represent more of a UX revolution than just a shift in computing platform.    As more LLMs move from PoC to production, it's critical for companies to determine whether real users are getting value from the LLMs.   This means that a key part of the infra stack will be understanding user behavior at scale and tuning models to user preferences, rather than just using LLMs-as-a judge to evaluate performance.   I wrote a blog post about a user-centric framework that has successfully helped companies launch, evaluate, and improve LLMs at scale. Please check it out and let me know what you think. Link in the comments.

59

3 Comments

This is the final part of the 3 part series of posts that shows how to Evaluate your LLM based RAG application. In the last part what I would like to show us how to Evaluate the Answer (Actual Response) faired against the Contexts Provided after Retrieving it from a Vector Database. The only metric that is available for this is Faithfulness. Where the Claims made in the Answer (Actual Response from the LLM) are checked against the Claims present in the Contexts Provided to the LLM to answer the user query. The way the Claims are obtained is by asking the same LLM or another LLM to present all the Claims made in the Answer, and then get all the Claims made in the Contexts that was provided to the actual LLM to generate the Answer. Then calculate the ratio between the two. #llm #evaluation #metric #ai

4

I really enjoyed playing with the AutoGen framework! There are so many interesting Agent to Agent communication design patterns. It's quite uncanny how much better you can get LLMs to perform when you create multiple focused prompts; assign each prompt to a different agent and then let the agents pass messages around. AutoGen has quite a few capabilities that make it stand out as a library. Including: 1) Reflection strategies, including LLM summarization 2) Human in the loop user proxy agents 3) Group Chat Manager agents 4) Nested Chat (metaphor: agents having a private chat; inner monologue of personas) 5) Code generation agents (with support for docker containers) 6) The ability to supply user defined functions into the code generation prompt 7) The ability to specify the edges between agents to force specific conversation paths 8) ... My favorite pattern is the Critic pattern, where you pit two LLM agents against each other: one is trying to perform a task, the other is trying to raise the bar and spot errors. I loved the example of using nested chat to make a super Critic. i.e. an LLM agent summarizing a conversation with a team of Critic agents assigned a specific focus. e.g. for a blog post. SEO critic, Legal critic, Ethics critic, etc. I don't want to give it all away. Take a look for yourself and if you are building something interesting with agents. Let's chat and compare notes. https://lnkd.in/gDe_S7ES

7

1 Comment

We have our own powerful domain specific language in Motif, but users do need to spend some time ramping up before they see the productivity gains. In the meantime, Motif users can now rely on our new Query Copilot, built right into our editor. If you've been waiting to try to Motif, there's never been a better time to get started. You can use plain-English descriptions of what you'd like to find and tag with our pattern-matching engine, and Query Copilot will write the SOL (Sequence Operation Language) code for you. Under the hood, we then compile this into a super-efficient state machine that executes on our incremental query engine. It's been exciting to watch as Alex Chin and the team gradually improved the model performance to the point where we can rely the LLM to accurately translate natural language into a new language that no GenAI model has ever seen before. Huge props to the team!

103

7 Comments

Show, don't tell. Nothing gets your point across to your audience like a visual explaining whatever it is you're trying to get across. My tool of choice for quick and easy diagrams is https://excalidraw.com/ You could explain something ten different ways, and still have it not be received as intended. I've found that drawing out concepts ensures they land more quickly and concretely with both technical and non-technical audiences. Here are some tips based on my experience communicating technical vision and strategy internally and externally: Step 1: Start Simple. Begin with high-level components. This isn't about dumbing it down; it's about clarity. Your goal is to illuminate, not overwhelm. Step 2: Use Analogies. Relate components of your system or product to everyday items or concepts. For instance, explaining a router's function by comparing it to a post office sorting and directing mail. Step 3: Visual Cues are Key. Colors, shapes, and icons convey information faster than words. Use them to differentiate between types of components or highlight the flow of information. Step 4: Iterate. Feedback is your best friend. Use it to refine your diagram. The best system diagrams evolve over time, improving with each iteration based on input from both technical and non-technical audiences. Step 5: Keep it Accessible. Store and share your diagrams where team members can easily access and review them. This encourages ongoing dialogue and understanding. Whether explaining how systems work at a high level or detailing how different products interact, I've seen clients, legal teams, and stakeholders benefit immensely from this clarity. A well-crafted diagram becomes an artifact for future conversations, a point of reference, and a tool for iteration.

5

AI is getting particularly good at generating code and images, key elements of video game development. Video game development isn't unionized. Result: Possibly the most AI job disruption in any single industry. How much disruption? "Video games—and the people who make them—are in trouble. An estimated 10,500 people in the industry were laid off in 2023 alone. This year, layoffs in the nearly $200 billion sector have only gotten worse, with studios axing what is believed to be 11,000 more, and counting." How much AI? "A recent report from the consulting firm CVL Economics, commissioned by entertainment industry trade groups, found the gaming industry already relegated tasks to generative AI more than its peers in TV, film, or music. According to its survey of 300 CEOs, executives, and managers, nearly 90 percent of video game companies had already implemented generative AI programs." This is one more data point to show that, yes, AI is reducing jobs. We also have data showing that it's improving productivity. Those are two sides of the same coin. There will likely be new jobs, but whether they come fast enough, or in enough quantity, for all the people getting displaced is an open question. Article here: https://lnkd.in/gUAdAtjm PS: I'm obviously still pro-AI. But we shouldn't pretend that it's not affecting people's jobs and livelihoods already. PPS: Why has Wired become so anti-AI? It's interesting to see this pivot from what I always perceived as a very pro-tech publication.

6

Let's talk about the constant race against libraries and framework version updates. A friend contacted me this week to ask for a bit of advice: Google forced him to update his Android app's SDK version to something more modern. However, since the newer SDK is not backward compatible he ran into a hell of an update needing to update React Native, gradle, and probably most of the RN libraries. Although we were able to find a simple workaround in this specific case, the point remains, that he would have to update to newer versions eventually. My suggestions on this issue are simple: 1) Like internal technical debt you already manage, put aside some time every quarter to update your base libraries and frameworks to the (almost) recent versions. The more frequently you do this, the easier it is. 2) Don't overuse external libraries. If you need something simple prefer simple local development. You never know when the maintainer of 'simple-sheker-kolshehu' will not have the time to upgrade to the latest Android SDK that you need to get into the app store.

10

2 Comments

Deep in the Llama 3 paper, Meta tested how common prompt tactics affect results: Not at all. On page 31, they told the model: 👨‍⚕ You are an expert ✅ Choose the best answer And found ❌no❌ measurable difference in MMLU scores, across all model sizes. You don't need to tell the bot to be "helpful" either! Only give it a persona if you need it to act other than a smart expert. And then, just ask the question. This makes sense, because these models are post-trained on how they hope humans will use it - no "gotchyas" and secret hacks required!

5

3 Comments

The revoultion in Storytelling coming to us via Generative AI is two-fold. One, we are soon to be able to assemble small, and very talented creative teams (writers, artists, directors, and actors) and create vast universes and intricate stories that while still driven from their center by human hearts, minds, and hands, have the ability to explode into being and even evolve over time at the highest fidelity and in myriad or even blended styles. Two, these narratives will become increasingly personalized, immersive, but still social shared experiences that sustain engagement in ways that give each of us what we love most without compromising the integrity of the storytelling universe, succumbing to the weight of insurmountable costs, or being hindered by the production timeline realities of today. It will be a disruptive shift, but one that can be ultimately democratizing in ways that will eventually encourage and empower artists again and give rise to new talents and new stories of all kinds because the gap between the audience and the creator will close to the point of intimacy. #media #generativeai #storytelling #ai #artists #streaming

11

Curious and slightly unsettling experience with chatGPT during a novel review processl (it has been a long time since the publication of my last fiction book!). I uploaded 4 chapters using the premium version and requested summaries of the main characters' names, surnames, and descriptions. One character intentionally had no surname, but chatGPT provided one. After pointing out the error, chatGPT apologized, acknowledged the mistake, and offered a different surname. When I inquired about the repeated incorrect surnames, chatGPT explained its reasoning: Certainly! The reason for the earlier errors with the characters' surnames is a mix of factors: Initial confusion in searching the document: When I tried to retrieve the names from the original document, the search didn’t extract the correct information, leading me to create plausible but incorrect surnames instead of relying on the actual text. Autogenerated names: In cases where I couldn’t locate the correct surname, I defaulted to generating surnames that seemed fitting for the context or the character’s first name. This was a mistake, as I should have used the accurate details from the text. Imprecise searches: My early attempts at searching the document didn’t yield the right results, so I made assumptions instead of double-checking the content properly. #chatGPT #AI

22

2 Comments

GPT 4o is same base model as 4 *but* much faster and better results for audio and visual input. So improvements on the base model, plus now 4 is free (used to be free users were stuck with 3.5)

4