Vish Agashe

Building Text-to-SQL from Scratch using DAGs 🛠️🧱 Most existing pre-packaged text-to-SQL modules are not suitable for production - inevitably you’ll have to go in and customize the prompts, and define custom orchestration. An underrated feature of LlamaIndex for advanced LLM developers is query pipelines: the ability to define very custom LLM pipelines as DAGs, letting you mix both prepackaged modules you don’t want to write (e.g. retrievers, SQL integration) as well as custom functions. Here Aruna Withanage shows you how to compose an advanced text-to-SQL pipeline:  1. Index the table schemas via a vector database 2. Retrieve relevant table schemas given a user query 3. Given schemas, convert query to a valid SQL statement 4. Retrieve relevant rows from the SQL database 5. Define a custom response synthesis prompt https://lnkd.in/gkhTJqjt

575

21 Comments

dbt Mesh is officially GA, changing the game for advanced data teams managing data at scale. 🚀 Our updated Best Practices guide is packed with insights from top orgs on making dbt Mesh work for you. Learn how to: • Optimize team setups for mesh architecture • Manage shared macros across projects • Map repositories to teams effectively Transform your data workflow with dbt Mesh. Learn how with our guide: https://lnkd.in/gRxFedGY

228

3 Comments

Worth reading: How the Apache Arrow Format Accelerates Query Result Transfer by Ian Cook One of the hidden problems of intersystem communication is transcription of data formats. Ian does a wonderful job of explaining the "why" of Apache Arrow as well as the "how" of its advantages. Read on...

8

🚨 Is your data engineering workflow holding you back? 🚨 MotherDuck and Dagster Labs are teaming up to discuss the secrets of modern data engineering. We'll cover: - Crafting an end-to-end data platform with Bluesky data, MotherDuck, and Dagster - Proactively solving problems with "shift-left" testing to catch issues early - Practical automation strategies to streamline your workflow and boost productivity 📅 January 14th ⏰ 9:00 AM PT ☑️ Registration link in the comments

13

1 Comment

ChaosSearch, as a Databricks Technology Partner, brings ELK use cases to Databricks ecosystem enabling centralized log and event analytics for observability and security. 🔒 Deliver on the best of Databricks (open Spark-based data lakehouse) and ELK (efficient search, flexible live ingestion, API/UI) via ChaosSearch on Databricks. Data ingested into Delta Lake format, all compute in Spark via Databricks and with the best of ChaosSearch’s hunting and Databricks’ AI/ML. Explore the power of ChaosSearch in your Databricks environment: https://lnkd.in/dSAXRgEF

4

Kumo helps data scientists build highly accurate predictive AI models that predict user and customer behavior by combining graph learning over enterprise data and generative AI models trained on public data—all entirely within Snowflake. If you want to create better machine learning models faster with Kumo, try our Snowflake Native App now. 🔗 You can request a free trial at the following link: https://hubs.ly/Q02HgQ1P0. If you need additional support and more details, contact us. We'd love to get in touch with you and help you query the future: https://hubs.ly/Q02HgGvG0. #AI #MachineLearning #PredictiveModels #DataScience #GraphLearning #GenerativeAI #Snowflake #TechInnovation #KumoAI #FutureOfData

9

"The new Columnar capabilities enable rapid real-time intelligence for data analysis and quick insights, along with further opportunities for traditional ML and AI" ⚡ Check out this write up on Capella columnar from private preview user Will Thayer, MBA Technologist 📝 https://bit.ly/3JGo8UR

24

1 Comment

Observe natively integrates with Snowflake Trail and continuously collects telemetry data. Observe for Snowflake provides complete visibility into the performance of Snowflake Data Cloud, UDFs, and Snowpark Container Services apps. In episode ten of of our #observability trends webinar series, Kyle Champlin and Daniel Odievich will explore how you can maximize performance and achieve compliance readiness in Snowflake with Observe. Register now: https://lnkd.in/gftXxx_4 (P.S. - If you can’t attend on the day, register anyway and we’ll send you the recording.)

11

So, what's new in dbt Cloud for June? Quick answer: a lot 🚀 From support for staging environments across various platform experiences, to our Snowflake native app, to the GA of dbt Explorer, dbt Mesh, and the Cloud CLI (oh my!)… there’s something for everyone. We've consolidated an overview of all of the latest features in one easy-to-read blog post. Dive in, below👇 https://lnkd.in/g_mqSgwn

155

"LLMs don’t reason like humans—understanding that is key to unlocking their true potential in the real world." — Drew Banin. In this insightful conversation, Drew explores how dbt Labs transformed cloud data workflows, the evolving role of semantic layers in AI, and how vector embeddings are changing the game. He also dives into the limitations of LLMs' reasoning and why prompt engineering might become less crucial as AI continues to evolve. This is a must-listen for anyone interested in how data and AI are shaping the future. To listen to Drew, click here :

10

Step into the lakehouse by tuning into our latest interview: The Databricks Data Lakehouse with Josh Miramant 📼 ☑ Discover how Lakehouse architecture merges the best of data lakes and warehouses to offer enhanced governance, scalability, and accessibility: ☑ Understand Lakehouse basics and its transformative benefits. ☑ Explore essential tools like Delta Lake, Apache Iceberg, and Apache Hudi. ☑ Learn best practices for effective implementation in your business. 🔗 https://lnkd.in/e9PQNEVr #DataManagement #Data #LakehouseArchitecture #Innovation #Databricks #DataScience

5

Confluent's Field CTO, Kai Waehner, shares insights on the future of streaming analytics and data management, highlighting the importance of reliable data flow, collaboration, and security. Discover how Confluent's cutting-edge platform revolutionizes streaming analytics. Read the full interview: https://lnkd.in/gSMVCmvy #StreamingAnalytics #DataManagement #FutureOfData #BusinessIntelligence

80

We are LIVE at PASS Data Community Summit! Join us for Day 1 of Postgres Conference Seattle for a "Journey through Postgres" w/ Zak Tedder in Rm 420 or "Tips and Tricks for developers moving to PostgreSQL" w/ Eric Lendvai in Rm 422. Check out the full schedule here https://lnkd.in/estNqRHm #Community #Postgres #Data #peoplepostgresdata

1

💡 Snowflake and Databricks are great for what they do—but what about workflows that span beyond them? Kestra bridges the gap, offering orchestration that integrates Snowflake, Databricks, and any other systems in your stack. Whether it's connecting APIs, automating pipelines, or scaling across platforms, Kestra provides flexibility and visibility that lakehouse tools alone can't deliver. Explore how Kestra enhances both platforms capabilities: ➡️ Kestra with Snowflake: https://lnkd.in/extwQjpy ➡️ Kestra with Databricks: https://lnkd.in/dhjyGKYY

28

3 Comments

Analyzing streams shouldn’t break the bank. 💰 But as volume balloons, so does the cost of observability tools like Datadog, ELK, CloudWatch or Loki; SIEMs like Splunk or Sumo Logic; and data warehouses like Snowflake or BigQuery. That’s because the architectures of these solutions were not built to efficiently ingest and analyze streams at scale. But ChaosSearch is, which allows our customers to realize 50-80% savings with unlimited retention. 💸 ChaosSearch delivered 90% cost savings to Equifax vs. their legacy tools. Try our pricing calculator for your guaranteed savings: https://bit.ly/427wXOq

1

Analyzing streams shouldn’t break the bank. 🏦 But as volume balloons, so does the cost of observability tools like Datadog, ELK, CloudWatch or Loki; SIEMs like Splunk or Sumo Logic; and data warehouses like Snowflake or BigQuery. That’s because the architectures of these solutions were not built to efficiently ingest and analyze streams at scale. But ChaosSearch is, which allows our customers to realize 50-80% savings with unlimited retention. 💸 Achieve an industry-leading price with guaranteed savings at scale: https://lnkd.in/dC26BrTX

5

Analyzing streams shouldn’t break the bank. 🏦 But as volume balloons, so does the cost of observability tools like Datadog, ELK, CloudWatch or Loki; SIEMs like Splunk or Sumo Logic; and data warehouses like Snowflake or BigQuery. That’s because the architectures of these solutions were not built to efficiently ingest and analyze streams at scale. But ChaosSearch is, which allows our customers to realize 50-80% savings with unlimited retention. 💸 Achieve an industry-leading price with guaranteed savings at scale: https://lnkd.in/dC26BrTX

4