Yiwei Song

Meet AGUVIS: A pure vision-based framework for autonomous Computer/Mobile-use GUI agents, operating seamlessly across web, desktop, and mobile platforms without UI code. Key Features & Contributions - Pure Vision Framework: First fully autonomous pure vision GUI agent capable of performing tasks independently without relying on closed-source models - Cross-Platform Unification: Unified action space and plugin system that works consistently across different GUI environments - Comprehensive Dataset: Large-scale dataset of GUI agent trajectories with multimodal grounding and reasoning - Two-Stage Training: Novel training pipeline focusing on GUI grounding followed by planning and reasoning - Inner Monologue: Explicit planning and reasoning capabilities integrated into the model training - Generalization to OOD (Out-of-Distribution): AGUVIS demonstrates impressive generalization on OSWorld benchmark even without desktop trajectory data during training. Project Page: https://lnkd.in/guzkwpbB Paper: https://lnkd.in/gs7wXyJS GitHub: https://lnkd.in/gWeSkFmK Collaboratiion among Yiheng Xu Zekun Wang Junli Wang Dunjie Lu Tianbao Xie Amrita Saha Doyen Sahoo Tao Yu Caiming Xiong

155

6 Comments

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

16

3 Comments

We launched structured data retrieval in Bedrock Knowledge Bases last week at re:Invent. While embeddings-based semantic search has worked wonders for unstructured data, we felt we need a new approach for structured data. We needed a way to translate customer natural language queries to an expressive query language that traditional databases already support. https://lnkd.in/eTPStNif #rag #text2sql #aws #bedrock

29

Check out the latest episode of our Tech Teams Today podcast with Yongsheng Wu, VP of Engineering at Granica - now available on all major podcast platforms. #AI #EngineeringLeadership #DataScience #MachineLearning #TechTeamsToday #Revelo

4

You want to do Machine Learning on large scale Marketing solutions ? Check below 🚀

3

Welcome to the Gemini 2.0 era ✨ Today we announced our most capable AI model yet, designed for the future of agents. “If Gemini 1.0 was about organizing and understanding information, Gemini 2.0 is about making it much more useful.”[https://lnkd.in/gzWTHFXq]

455

3 Comments

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

So excited to see our team's work featured here! These are fundamentals directions for recommender systems, essential for advancing the field. Excited to see how others will build on this and drive deeper insights. https://lnkd.in/gkqpq7HS

134

4 Comments

GPUs are expensive and in short supply. Learn how to automagically consume just-in-time rightsized NVIDIA GPUs for your Ray clusters from Anne Holler and Justin Willoughby Anyscale #mlops #gpu #ray #nodelesskubernetes

20

Chip Huyen on Twitter today below. I 100% agree. Metric hygiene is system hygiene. LLM-Generated metrics require a higher standard of care than they're currently receiving, and much more scrutiny. Mind the compasses. 🐦  A big issue I see with AI systems is that people aren't spending enough time evaluating their evaluation pipeline. 1. Most teams use more than one metrics (3-7 metrics in general) to evaluate their applications, which is a good practice. However, very few are measuring the correlation between these metrics.If two metrics are perfectly correlated, you probably don't need both of them. If two metrics strongly disagree with each other, either this reveals something important about your system, or your metrics just aren't trustworthy.2. Many (I estimate 60 - 70%?) use AI to evaluate AI responses, with common criteria being conciseness, relevance, coherence, faithfulness, etc. I find AI-as-a-judge very promising, and expect to see more of this approach in the future.AI-as-a-judge scores aren’t deterministic the way classification F1 scores or accuracy are. They depend on the judge's model, the judge's prompt, and the use case. Many AI judges are good, but many are bad.Yet, very few are doing experiments to evaluate their AI judges. Are good responses given better scores? How reproducible the scores are -- if you ask the judge twice, do you get the same score? Is the judge's prompt optimal? Some aren’t even aware of the prompts their applications are using, because they use prompts created by eval tools or by other teams.Also fun fact I learned from a (small) poll yesterday: some teams are spending more money on evaluating responses than on generating responses 🤯 https://lnkd.in/gYnTRZW3

2

Effortless Integration for Smarter Insights: CloudEagle.ai Meets Bitbucket Integrating CloudEagle.ai with Bitbucket takes just minutes and gives you: ✔ Simplified App Tracking: No more manual reports, get the app usage data you need, when you need it. ✔ Centralized Control: Manage your Bitbucket security from the CloudEagle dashboard. ✔ Proactive Monitoring: Catch unusual login activity before it becomes a problem. Ready for a smarter, more secure development workflow? Book a demo today - https://lnkd.in/gXAhChrt #saas #saasmanagement #toolstuesday Atlassian #jira #toolstuesday

5

Congrats the team on ICML acceptance of the Wukong paper: https://lnkd.in/g-MyahNH, which represents a breakthrough of recommendation model design and efficiency. Recommendation models traditionally suffer from not able to utilize the full computational power of GPU (only 5-10% of peak GPU FLOPS while LLMs can reach >50%), due to their sparsity. Wukong is first of its kind by pushing toward 40% GPU peak FLOPS though model-infra codesign. One of my teams consisting of GPU experts is building a high performance model component libary where we’ll provide highly optimized model components (eg sequence summarization, x-sequence interaction, x-GPU layout scheme) that could be composed - with the goal that models built from this library have high performance out of box to scale the model-infra codesign effort. We are also scaling training beyond the limits that are typical of recommendation models - from hundreds of GPUs to 10s of thousands. If this is the kind of work that excites you, come join us!

265

4 Comments

Exciting and pleasantly surprising news in the AI world: US & China join forces to develop first international AI standard. A promising step towards rapid, scalable, and safer AI development. Collaboration over confrontation paves the way forward! #airevolution #artificialintelligence #largelanguagemodels #AICollaboration

36

Using generative AI, NVIDIA operations built an AI Planner agent, developed on NVIDIA Inference Microservices (NIM). The agent leverages LLM, NeMo Retriever and CuOpt NIM to reduce re-planning time from hours to just seconds. https://nvda.ws/4bIfS2T

3