Reza Shiftehfar

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

Hey everyone, I am an investor and advisor to Katanemo.... They are reinventing the infrastructure primitives for GenAI and launched Arch - https://lnkd.in/gCrbbkEV - an intelligent prompt gateway. Give the project a look on Github, and star the project if it resonates

88

3 Comments

When you need to play with your data in Gen Ai, you need RAG and Chunking is one of the main component in the RAG framework while consuming or processing your data. Excellent post below describing how in different ways chunking in RAG can be implemented..

23

1 Comment

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

So excited to see our team's work featured here! These are fundamentals directions for recommender systems, essential for advancing the field. Excited to see how others will build on this and drive deeper insights. https://lnkd.in/gkqpq7HS

134

4 Comments

How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out. https://lnkd.in/dwczVutW

42

5 Comments

We launched structured data retrieval in Bedrock Knowledge Bases last week at re:Invent. While embeddings-based semantic search has worked wonders for unstructured data, we felt we need a new approach for structured data. We needed a way to translate customer natural language queries to an expressive query language that traditional databases already support. https://lnkd.in/eTPStNif #rag #text2sql #aws #bedrock

29

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

16

3 Comments

Evaluating Large Language Models: Key Metrics and Their Application in Real-World Scenarios This article delves into the essential metrics used to assess LLMs, exploring their significance and practical applications, particularly in developing a Question Answering system. Join us as we unpack these evaluation criteria and illustrate their impact on real-world AI implementations. #AI #LLM #Evaluation #AIDevelopment #AIApplication For part 2 of the article, click👇 https://lnkd.in/gd8gDGPM For part 1 of the article, click👇 https://lnkd.in/gdSew4uV For a quick summary - Swipe 👉

14

1 Comment

My recent blog post with Iris Fang on how NeuroPace ML workloads is leveraging Google Cloud is now live.

31

1 Comment

It is time to look forward to the summer of 2025! https://smrtr.io/npvQ- Staff Machine Learning Scientist, Intern - Summer 2025 at Visa, inc.   Unleash the world scientist within you this summer! Inspired by Richard P. Feynman's maxim, “You do any problem that you can, regardless of field,” we're inviting you to contribute your innovative skills in cross-industry advancement of science and humanity at VISA, the technology giant at the forefront of billions of global credit payments. We’re calling all physical scientists to venture into our world of ML/AI. The mission? To uncover fundamental truths hidden in the vast and largely untapped domain of Payment transactions, a realm as complex and intriguing as Physical interactions. Just as scientists construct effective models to make sense of real-world data, we strive to unearth elemental factors driving billions of transactions per day. With the rise of neural network-based deep learning in Physics, we're pioneering the use of these techniques to revolutionize global commerce.

120

1 Comment

In Retrieval-Augmented Generation (RAG) and other tasks like Knowledge Base Gap Analysis & Discovery, the vector embedding of article chunks is crucial within the landscape of LLMs. Implementing better chunking strategies (e.g., dynamic chunking) and retrieval strategies (e.g., reranking models) significantly boosts RAG performance. To further improve RAG systems, Raghav Garg discusses how to generate effective vector embeddings for real-world, noisy Knowledge Base articles in this insightful blog post: https://lnkd.in/dyGQgi25 How do you optimize your RAG systems?

69

1 Comment

At Kargo, we use AI/ML across a wide variety of products and processes. While I understand how our regression models are trained, tuned, and deployed, I realized I was clueless about how LLMs were helping power the technology behind our contextual advertising tools. A while back, a Kargo engineer shared a short video by Ishan Anand explaining LLMs in a novel new way. Using only Excel spreadsheets, he re-built GPT-2, the 2nd transformer model released by OpenAI back in 2019. This implementation is the basis for his course: "How AI & LLMs Work: A Fast-Track Crash Course for Busy Professionals". The interactivity of the spreadsheets enabled the hands-on learning style that I needed to grasp the more complicated concepts (like how Positional Embeddings help the model understand that the placement of a word in a sentence can alter its meaning). In 3 relatively short sessions, Ishan seamlessly breaks down the various components of a basic LLM, giving you a better understanding of what's going on behind the scenes of these models. I'd highly recommend his course to any technical person that uses LLMs in any form, even if it's just to decode some of the magic behind ChatGPT. And to my Kargo co-workers, you could consider putting your Lifelong Learning budget towards the course fee!

99

1 Comment

Interested in learning more about what powers #AppleIntelligence? This paper describes in detail the Language Models that power these features. Please reach out if you want to join us in building incredible features powered by generative models that bring surprise and delight to millions of users! https://lnkd.in/ghZxHD47

49

1 Comment

I first joined Amazon in December of 2005. Even though I left for two years, my "Amaversary" still coincides with the end of the year and a natural moment for reflection. As I reached my 17 year mark this December, I also had the privilege of working with Amazon Careers to share some of my journey and thoughts about individual contributor technical leader and people manager career paths. In my ruminating, I put together far more content than could be used for a single article. So, I've decided to post here as a series of Q&As - in case others might find my observations, experiences, and perspectives interesting or useful. Below is the first of 6. 1. 𝘽𝙧𝙞𝙚𝙛𝙡𝙮 𝙨𝙝𝙖𝙧𝙚 𝙮𝙤𝙪𝙧 𝙘𝙖𝙧𝙚𝙚𝙧 𝙟𝙤𝙪𝙧𝙣𝙚𝙮; 𝙬𝙝𝙖𝙩 𝙢𝙤𝙩𝙞𝙫𝙖𝙩𝙚𝙙 𝙮𝙤𝙪 𝙩𝙤 𝙩𝙖𝙠𝙚 𝙤𝙣 𝙙𝙞𝙛𝙛𝙚𝙧𝙞𝙣𝙜 𝙘𝙖𝙧𝙚𝙚𝙧 𝙥𝙖𝙩𝙝𝙨, 𝙄𝘾 𝙫𝙨 𝙢𝙖𝙣𝙖𝙜𝙚𝙧? I started my career as a software engineer 25 years ago. I've worked for startups, enterprise companies, and 7 different organizations over 17 years and two tours at Amazon. I've written firmware for robotic camera mounts; built highly scalable distributed systems; designed restaurant marketing, ordering, logistics, and operations software; created automated test harnesses for mobile games; tested HD DVD encoding; helped launch grocery delivery to customers around the globe - just to name a few of the many cool things I've had the privilege to work on. I've been an entry-level software engineer, a technical lead of teams, a software manager on a couple of occasions, chief software architect reporting to a CTO, and (for the last few years) Amazon Principal Engineer directly supporting a VP with about 200 engineers within my remit. I first became a manager because I mistakenly thought it was the only way to continue growing as a leader, and in many ways I failed. I learned some very humbling lessons about leadership through that experience, which I used to be a better individual contributor technical leader on my path to Senior Engineer. My second time as manager was because my organization had a need that I could fill. I had grown enough as a leader that this time I was much more successful. I said yes to an opportunity that started with managing a small team through delivery of a complex and critical project. Once I grew a high performing team that delivered that initial project, I took on more scope and expanded into multiple teams. After a couple of years of success growing and investing in my teams so that we could deliver greater value to our customers and business, I decided to prioritize being close to my family. For me, at that time, it meant giving up my people management role and returning to the role of an individual contributor technical leader that supports a globally distributed organization. Even though I really enjoyed my time as a people manager, there is no better job for me than that of a Principal Engineer. #amazon #careerdevelopment #technicalleadership #peoplemanagement

91

3 Comments

According to Eric Schmidt, in the next year or so, the union of the large context window, agents, and text-to-action, when delivered at scale, is going to have an impact on the world that no one understands yet. Listen to the fascinating discussion in the "The Age of AI, Eric Schmidt" youtube video - https://lnkd.in/dFTKgBVY

53

1 Comment

First story on what we have been up to at Augment. Much more to come! https://lnkd.in/gC6gbADA

370

64 Comments

🚀 Thrilled to see the launch of the next-gen OpenSearch Dashboards! As an engineering leader, you often work in multi-year timeframes, where the seeds you plant take years to fully bloom. This project was one of those—one I poured my heart into from the early days, writing and collaborating on countless documents with my team and cross-functional partners and laying the foundation for a new era in OpenSearch Dashboards. Although I left Amazon, it’s incredibly rewarding to see this vision come to life and to see my team’s hard work bear fruit. I’d like to give a special shoutout to Huan Jiang. When he first reported to me, he was one of the strongest ICs I’d ever worked with, with a natural inclination for leadership. I had the privilege of watching him grow into an amazing manager who relentlessly supported his team. Huan took strong ownership of this initiative, collaborating with other engineering groups, and was crucial to the success of this release, guiding his team through to the finish line with excellence. His dedication to the project and his team makes me immensely proud. Congratulations to everyone involved—seeing this reach users is a testament to the hard work, resilience, and collaboration of the entire team. It’s a reminder that, as leaders, we may not always witness the full lifecycle of every project we start, but the impact lives on through the incredible people who carry it forward. Our wake is bigger than our boat. Here’s to many more achievements in the world of the OpenSearch Project! #engineering #leadership #teamwork #OpenSearch #AWS #analytics #proudmoment

15

I'm excited to share two papers from the Retrieval team at Walmart that were presented at CIKM last week. They're both about improving the relevance of Embedding-Based Retrieval. 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗙𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗳𝗼𝗿 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹. https://lnkd.in/gNQe7QhA The first is about how to intelligently choose a query-dependent cutoff on the cosine score to remove irrelevant items from the retrieved set. Since cosine scores aren't comparable across queries, we introduce a "Cosine Adapter" to map the scores to interpretable scores. 𝗘𝗻𝗵𝗮𝗻𝗰𝗶𝗻𝗴 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗼𝗳 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗮𝘁 𝗪𝗮𝗹𝗺𝗮𝗿𝘁. https://lnkd.in/gdqkc2hR The second is about model training techniques we learned to improve the relevance of embedding models. In particular, we describe how to use a teacher model to provide relevance labels for training Juexin (Joy) Lin Nicholas Rossi, PhD Feng LIU Zhen Yang Sachin Yadav Praveen Reddy Suram Satya Chembolu Prijith Chandran Hrushikesh Mohapatra Alessandro Magnani Ciya Liao

148

2 Comments