Alistair Garfoot

Best data news? It's right here 🎺 https://lnkd.in/ecYCuxYD Check it out: Medium 🧠 🧠 The BigQuery Qualify clause @matteoBennio 🧠 How Notion handles 200bn data entities Vu Trinh 🧠 A scalable ingestion framework in MS Fabric Piethein Strengholt 🧠 Migrating and testing out pyairbyte Felix Gutierrez 🧠 Snowflake vs Databricks; a useful comparison for 2024 Hugo Lu 🧠 People do not like this latest article “The rise of the notebook engineer” (sorry Daniel Beach!) **Editors Pick** 🧠 How Pinterest do Analytics Kapil Bajaj LinkedIn🕴 🕴 BigQuery release continuous Queries and real-time rELT (also see Richard Shenghua (升华) He's excellent post on this recently) 🕴 Difference between Junior and Senior engineers 🕴 Trio of Leaders leave Open AI #dataengineering #roundup #dataroundup #bigquery #streaming

14

3 Comments

🦆 What makes a Great Data Leader? 🦆 Data Engineers London recently had the pleasure of hearing from Nik Walker, Head of Data Engineering at Co-op, who delivered an absolutely quack-tastic talk on the power of human-centric leadership for data teams. Nik certainly brought his learned own views, but not just that; he guided us through a thoughtful exploration of how empathy, trust, and open communication are the real engines driving successful teams. And yes, there were a few quacks along the way to keep things lively! 🦆😅 The best part? Nik’s insights aren’t just for those leading teams - there’s something valuable here for anyone working in data. Trust us, it’s worth a watch! Some of the highlights! 🎥 Human-Centric Leadership 🤝  People over technology. Understanding your team’s emotions and needs is key to tackling any challenge. Trust & Vulnerability 🚀  Effective leaders build trust by being open and showing vulnerability—this creates a space where teams feel empowered to innovate and take risks. Psychological Safety 💬  Safe spaces aren’t just buzzwords. They’re essential for fostering honest conversations, admitting mistakes, and driving innovation. Inclusivity 🌍  Valuing every voice on the team leads to richer ideas and solutions—diversity truly drives creativity. Continuous Learning 🧠  Never stop learning! Encouraging ongoing development keeps the team adaptable and engaged in a fast-changing field. And perhaps the most resonant takeaway? Servant Leadership 🏅 Prioritising the team’s needs above your own cultivates a loyal, motivated, and highly effective team. Check it out now! - https://lnkd.in/eN8d4rye Thanks to our sponsors Kyndryl for your support, and Zego for hosting us! And thanks from us to you all in the community! Make sure you join our community page for the next event! Anna Aleshko, Jamie Marney & Harry Duddridge #DataEngineering #Leadership #EmpathyInTech #Data #TeamCulture #ContinuousLearning #Humancentricleadership

8

3 Comments

AI isn’t the end for data practitioners. Very much the opposite. Barry McCardel, the co-founder and CEO of Hex, is bullish on a new golden era of data teams, because, “There's an infinite demand for insight and an infinite demand for data work.” Barry joins Tristan Handy on the Analytics Engineering Podcast to discuss the hype cycle of data science, how AI is different, the experience of building AI products, and how AI will impact data practitioners. https://lnkd.in/d-hfmisr

53

Metadata is a key asset in your data estate, but the term is broadly used. So let’s break it down… In the latest episode of our “Data Engineering in Five Minutes" series, Harry Cummings—our Principal Technical Consultant—discusses the different categories of “data about data.” #DataManagement #MetaData #DataEngineering

31

𝗛𝗼𝘄 𝗛𝗶𝗿𝗶𝗻𝗴 𝗬𝗼𝘂𝗿 𝗙𝗶𝗿𝘀𝘁 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗖𝗮𝗻 𝗨𝗻𝗹𝗼𝗰𝗸 𝗖𝗼𝗺𝗺𝗲𝗿𝗰𝗶𝗮𝗹 𝗣𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹👇 At Burns Sheehan, we offer tried and tested advice to our partners and network on the best time to make that critical first Data hire. Opinions vary widely—some start-ups wait too long, burdening their Software Engineers with Data Engineering tasks. This delay can make the eventual transition more challenging. Drawing from recent conversations with Andy Done, we explore the ideal timing and profile for a Data Engineering specialist in a startup. If you're a leader considering this, it's probably time to act! Would love to hear your thoughts on this topic. 📊 Check out our full blog here below. #datainsights #datateams #datahiring #firstdataengineer #datablog #dataengineer

2

What makes a ‘Good Data Engineer’? 🤯 Data Engineers London recently had the pleasure of hearing from the Director, Data Architecture at gravity9, Benjamin, do a brilliant talk on the differing views of how to be a 'Good Data Engineer' in a big organisation. Ben did not bring just his own views, he took us through a journey of thoughts from industry leaders around the Data scene, and left the audience with a lot to think about! 🧠 The best news? Anyone in Data can take some lessons from this talk. (It’s honestly worth the watch!) Some of the highlights! 🎥 📊 The importance of understanding corporate goals (as well as the Tech): chasing revenue, reducing costs, and mitigating risks. 🤝 That communication is key in complex environments with multiple perspectives. 📚 Treating data as a product helps develop valuable domain knowledge. 🔄 Ensuring you adapt to variety of data engineering roles and perspectives of data in large organisations. And maybe the most interesting take? 🧩 Soft skills are essential for navigating large, complex organisations. Check it out now! https://lnkd.in/dsquWQG3 Make sure you join our community page for the next event! Thanks to our sponsors Kyndryl for your support, and Zego for hosting us!

19

3 Comments

Catch-up on all the Data and AI news you missed! Google in talks to buy Wiz for $23bn and some more fundraising in the data space 🙌 🧠 Step-by-step guide to the Snowflake Solutions Centre Cameron Wasilewsky 🧠 Deliver Data as a Product - not as an application @bernd wessly 🧠 Go from staging to prod with one SQL line (link) Very reminiscent of this but for bigquery Zach Quinn 🧠 MLOps Data Drift monitoring architecture Ling Huang 🧠 Tearing down the monolith - The Rise of Microservices in Data Engineering Modern Data 101 Vivek Dubey LinkedIn🕴 🕴You don’t need another Data Quality Tool Phil Ballentine, MSc 🕴Propose a Talk for the Forward Data Conference Christophe Blefari 🕴Don’t auto delete or merge my PRs please! Ryanne Dolan And some great video content form Matthew Mullins and Joe Reis 🤓 SUBSCRIBE ✋ https://lnkd.in/eR42VRXK #dataengineering #dataleadership #technology

21

3 Comments

It's Day 2 in our Snowflake mini-series. Since being founded in 2012, it's fair to say Snowflake has been pretty transformative to data teams' lives worldwide. Here are some of the ways how Snowflake revolutionised Big Data: 1️⃣ Separation of Storage and Compute: - Before Snowflake, businesses had to deal with monolithic architectures where storage and compute were tightly coupled, leading to inefficiencies and high costs - Snowflake’s architecture decoupled storage from compute resources, allowing businesses to scale independently. This means you only pay for what you use, optimising costs and performance. 2️⃣ Data Sharing and Collaboration: - Sharing data across organisations used to mean cumbersome data transfers and security risks. - Snowflake’s secure data-sharing features allows for instant, controlled access to live data without the need to move or copy it. - This has enabled real-time collaboration across organisations and created a dynamic ecosystem where partners, vendors, and customers can seamlessly interact. 3️⃣ True Multi-Cloud Capability: - With Snowflake, you’re not tied to a single cloud provider. Businesses can deploy across AWS, Google Cloud, and Microsoft Azure, leveraging the strengths of each. - This multi-cloud strategy ensures high availability, disaster recovery, and vendor independence, offering unparalleled flexibility and global reach. - Previously, organisations were often locked into a single cloud provider, limiting flexibility and increasing risk. If you want to find out how Orchestra can further revolutionise life for Snowflake users, check out for yourself down below! ➡ https://lnkd.in/eM6uqgCU #BigData #CloudComputing #Snowflake #Innovation

6

2 Comments

We're coming to London 🇬🇧 Join Guy Fighel at Big Data LDN on Sept 18-19 for four #dataengineering events you cannot miss: ⚙ Guy's Keynote: The Inevitable Shift to Modern Data Lake Architecture (12:40pm Sept 18) ⚙ Great British Data Founders Panel (5:40pm Sept 18) ⚙ Data Engineering Things Meetup (6pm Sept 18) ⚙ Drinks with Hetz at Big Data LDN (2pm Sept 19) Featuring Hetz data portfolio Expanso + Superduper + Upriver Sign up info in the comments 👇 #data #engineering #startups

17

2 Comments

“Data is important for everyone, and it should be approachable for everyone, meaning that you don't need to be a data scientist or some sort of data expert to be able to use data.” - William Bett Listen to Ryan and William discuss the importance of data in an approachable way on the Making Better Decisions podcast. https://bit.ly/3BbIUuu #CanopyAnalytic #MakingBetterDecisions #Data #Podcast #DataAnalytics

2

In context of debate if large context #LLM models will eventually negate the need for #RAG, an article from Databricks has some good insights - • Longer contexts boost RAG accuracy... to a point • Performance often peaks at 32k-64k tokens, then declines • Top models (GPT-4, Claude 3.5) maintain consistency at scale • Others show unique failure modes (e.g., copyright concerns, summarization instead of Q&A) • Optimal context size varies by model and task • Lack of long-context training data may explain some issues Key takeaway: Long context and RAG are synergistic, but LLMs need refinement to fully leverage extended contexts. https://lnkd.in/eehdKtCm

17

How Long Does it Take to Implement Data Governance Framework? In this insightful article from Nicola Askham, Data Governance Coach at Nicola Askham Ltd T/A The Data Governance Coach, she dived into the realities of data governance - how it’s not just a one-time task, but an ongoing practice that builds a healthier data environment to support organizational goals. Read the whole at https://hubs.li/Q02RllBm0 #hyperight #hyperightread #datagovernance

10

1 Comment

In analytics, change is the only constant—so what’s next? Recorded live at MeasureCamp London, Bhavik Patel and Daniel Perry-Reed dive into the future of data, tackling myths around 100% accuracy, data ethics, and the evolving role of analysts. 🔗 Full episode in the comments! 👇 ------- #data #MeasureCamp #livepodcast #dataethics

9

2 Comments

Are technical book clubs still relevant in 2024? Our Data Platform Principal Engineer, Thomas Rushton, thinks they might be. Thomas's latest blog details his experience running a technical book club for SQL engineers, including what worked, what didn't and why he's now stuck trying to find suitable books that aren't a decade old! The ideal book needs to work for drop in sessions, be current enough to be relevant, and be available to purchase in quantity. Turns out that's harder than it sounds. Have technical books had their day? If not, what would you recommend for a modern technical book club? Read more here - https://lnkd.in/esFasd6N #SQL #Data #DataAnalytics #CloudComputing

9

What should come first; the ML algorithm or the data prep? 🤔 Andrew Jones says that with experience, his answer is always data preparation. His checklist is comprehensive and demonstrates a deep understanding of the importance of data preparation in machine learning. Here are a few additional steps and considerations we thought of that could even further enhance the process: ✅ Data privacy and compliance - are we adhering to regulations? Ensure all data processing adheres to privacy laws and regulations, anonymising or encrypting sensitive data as necessary. ✅ Class imbalance handling - how do we address uneven distributions? Address imbalanced classes using techniques like resampling, synthetic data generation or adjusting class weights to improve model performance. ✅ Pipeline automation and monitoring - how do we ensure long-term performance? Automate data pipelines and set up continuous monitoring systems to ensure efficient workflows and maintain model performance post-deployment. Incorporating these steps will further ensure that ML models are not only robust and performant but also ethical and reproducible. #data #dataprocessing #dataprivacy #compliance #datasystems #DAMAUK

5

Apache Spark is a secret weapon that data engineers can use to revolutionise any company's data strategy. In this article, we discuss how Databricks, with Apache Spark under the hood, allows a company to utilise their big data infrastructure. We focus specifically on versatility, scalability, and speed. Read the full article here: https://lnkd.in/dMU56Z2u #BigData #ApacheSpark #Databricks #DataEngineering #ReallyAwesomeDataPeople #ExpectMore #StepAhead #DataConsulting 

21