Dan Sullivan

New blog post by Alex Monahan: Benchmarking Ourselves over Time at DuckDB The DuckDB team's philosophy is to ensure correctness first, then iterate and optimize to improve performance. This blog explores how this happened over the last three years when DuckDB became approximately 3-25x faster and 10x more scalable. Read more at https://lnkd.in/dnQ7Nya8

196

16 Comments

This week on the blog, Ami Ikanovic discusses the "DIY is better" mentality for #datamasking and how it might seem like a quick win initially, but the long-term costs – time, complexity, and risk – are too high. https://lnkd.in/eUXyXKQf #datasecurity #datasecops #datagovernance #Snowflake

18

A very informational article discussing database news across 2024, including the politics of the database market, acquisitions, funding, and other random happenings. https://buff.ly/4hgB3LZ #opensource #technews #postgresql #oracle #databricks #snowflakedb

2

Our new research report, "Key Value Stores: Adoption Trends Through a Valkey Lens," reveals industry-first insights into customer sentiment regarding Redis's decision to revoke the open source status of its key-value data store software. Explore key findings and takeaways below. https://bit.ly/3Y7Havn via TFiR #Valkey #OpenSource #database

9

Deepa Ganiger's blog offers beneficial insights into implementing schema change detection and evolution in a streaming scenario with Snowflake. Gain a high-level understanding of the process and learn how to handle complex JSON data. phData is here to help! Reach out to our team of experts today. 😊 https://lnkd.in/g9KntHZV #DataScience #Snowflake #Kafka #DataProcesses #DigitalTransformation

11

💡 "Onehouse isn't about competition (between Apache Hudi, Iceberg, and Delta Lake) but rather about interoperability" 👐 Yes! We're building the most open data lakehouse, which also means we support all three formats ➡ it's all about user choice, freedom to choose the right format for the right task, and avoiding lock-in. Great article from VentureBeat highlighting our Series B and two latest product launches LakeView and Table Optimizer. #dataengineering #bigdata #data #datawarehouse #datalake #datalakehouse #nolockin

27

We are LIVE at PASS Data Community Summit! Join us for Day 1 of Postgres Conference Seattle for a "Journey through Postgres" w/ Zak Tedder in Rm 420 or "Tips and Tricks for developers moving to PostgreSQL" w/ Eric Lendvai in Rm 422. Check out the full schedule here https://lnkd.in/estNqRHm #Community #Postgres #Data #peoplepostgresdata

1

🔍 Just stumbled upon an interesting article! Snowflake Cortex LLM functions are now generally available, offering powerful machine learning capabilities within Snowflake. Dive in: https://lnkd.in/gehtvatQ #Snowflake #MachineLearning #TechNews

3

✨ Master AI in Snowflake: Enhance Your DBT Workflow ✨ Our Senior Data Engineer, Jaime Andres Salas, guides you in integrating LLMs with Snowflake Cortex into your DBT transformations. Key points: 🔹 Sentiment analysis with AI 🔹 Join sentiment data with reviews 🔹 Create recommendations for feedback https://hubs.ly/Q02BX1sW0 👉 Read the full article #AI #Snowflake #DBT #DataScience #LLM

8

Reasons to monitor dbt in Orchestra 🕊️ Ease of use Orchestra is the easiest way to set-up dbt core orchestration and monitoring. The Data Catalog even gets updated without any additional integration 💸 Cost Orchestra provides an execution environment for dbt core with lightweight pricing, which can save on existing dbt compute costs. We also support monitoring dbt core projects in AWS EC2, Virtual Machines, etc. so you can run dbt core your way without compromising on visibility 🔀 End to End Orchestration There is sometimes more to life than synchronously pushing down SQL Queries to your data warehouse. Orchestra provides a feature-complete, end-to-end orchestration framework so you can trigger and monitor any job from a single place What other problems do you want us to solve? See it in action below. https://lnkd.in/eTzuG7Ty #dataengineering #analytics #dbt

2

dbt Cloud 🤝 Datavault 🤝 AutomateDV (formerly known as dbtvault) How do dbt Cloud and Data Vault work together? Enhanced Visibility and Governance: Combining dbt and AutomateDV is smart—they work seamlessly together. dbt features included in the dbt Mesh offering include data security, lineage and contracts as well as model versioning support many of the paradigms also built-in to Data Vault. All of this is defined using files, so it is all tracked in version control as well. Scalability and Efficiency: AutomateDV on dbt Cloud supports scalable Data Vault components and has been tried and tested in larger companies and production environments. Combined with enterprise features in dbt Cloud, it provides effective management of extensive Data Vault projects. Higher Standards and Reliability: The dbt ecosystem of packages and features provide a solid foundation for doing Data Vault projects consistently and to a high standard with less maintenance overhead. dbt Cloud streamlines development, ensuring speed without sacrificing quality. Paired with Data Vault's pattern-based approach, it lets developers focus on business needs, not just technical tasks. Learn how to streamline your data warehouse development with this quickstart guide by Alex Higgs, AutomateDV Product Manager at Datavault: https://lnkd.in/e7KYMuVP

80

If you're tired of expensive data lakehouse/analytics solutions and are considering open-source technologies, you definitely would like to read this: Dien, Tran Thanh shares his firsthand experience, proven best practices, and insightful suggestions for migrating from BigQuery to Apache Doris. They are saving $4,500 per month. https://lnkd.in/ggVXgRaB #datalakehouse #BigQuery #dataanalytics #dataengineering #dataplatform #opensource

50

2 Comments

⚖️ Tech Comparison: Redshift vs Druid https://lnkd.in/e3ZCD3xb Like {hashtag|\#|Snowflake}, {hashtag|\#|Redshift} was built for traditional BI and infrequent reports. Not having multi-level indexing and time-based partitioning means inefficient compute with more data processed than necessary. When the analytics use case requires high performance on large data sets or concurrency requirements and streaming data, then Apache Druid is ideal.

10

Apache XTable provides users with the ability to translate metadata from one #lakehouse table format to another omni-directionally. What exactly happens after the XTable "Sync" Process is run? The sync process provides users with the following: ✅ Syncs the data files along with their column-level statistics and partition metadata information ✅ All the schema-level updates in the source table are reflected on to the target format metadata ✅ Metadata maintenance for the target table format. - If the target format is Apache Hudi, unreferenced files will be marked as ‘cleaned’ to control metadata table size - If the target format is Apache Iceberg, snapshots will be expired after a configured amount of time - If the target format is Delta Lake, the transaction log will be retained for a configured amount of time ⭐️ Want to try out XTable? - Here is a link to the getting started page: https://lnkd.in/gHMBQeqV #dataengineering #softwareengineering

49

1 Comment