Alex DeBrie’s Post

Instead of subscribing directly to a DynamoDB stream with a Lambda function, it is worth taking a closer look at the DynamoDB stream to SQS via EventBridge pipe pattern. See: https://lnkd.in/eexq_a3e It offers several advantages, such as the use of filters, enrichments or the use of a so-called dead letter queue with SQS. In addition to DynamoDB streams, EventBridge offers several other sources and targets.

Recently I have been reading 📖 /watching 🔍 a lot of stuff on Event Driven Architecture (EDA) on AWS . This week I learnt about Amazon DynamoDB Streams and how it can be used to notify about the changes in DynamoDB tables, which is critical in EDA implementation. One such implementation i have written in my Hashnode blog. https://lnkd.in/gAHPzkz8 PS: I am sharing this post and would like to thank and greatful 💐 to Savinder Puri for motivating me to learn in public. #AWS #DynamoDBStreams #SNS #EDA

This is a compelling example of the realistic throughput and performance you can get from #serverless while also highlighting the clear benefits of a well-designed cache. Momento in this case, which also happens to be built in #rust (and not coincidentally has a great team behind it). Benjamen Pyle continues to share a wealth of great insight in this space. I highly recommend following his blog. This post is a particular favorite, especially as I talk with more small and early stage companies struggling to find the right tech stack: https://lnkd.in/gRyF_RAz.

More Data? No Problem! We Can Handle It with No Issue: 6.8 Billion Records Now Want to learn how we manage and deep-dive into these massive tables? Read our blog: "How We Used Bucket Index with Apache Hudi and Right Partitioning for ~660GB Tables with Nearly 4.4B+ Records Using AWS Glue 4.0". Discover the strategies and tools that make it possible to handle enormous datasets efficiently. https://lnkd.in/ezz3ha7J

If you’re using DynamoDB and still making individual writes for each item, you’re probably leaving cost savings on the table. The BatchWriteItem operation lets you write up to 25 items in a single request. It helps reduce: ✅ API call overhead – fewer network round trips ✅ Request charges – especially in on-demand capacity mode ✅ Latency – faster and more efficient operations Here are the main differences: - 10 PutItem requests → 10 API requests, 10x the overhead - 1 BatchWriteItem request (10 items) → 1 API request, same WCUs, but less overhead Why does this matter? If you’re scaling your app or working with large datasets, this optimization can make a big difference in reducing DynamoDB costs while improving performance. The second video in my series "Dollars & DynamoDB" goes into more detail with a live demo of writing multiple items to DynamoDB with BatchWriteItem. Check it out here 👉 https://lnkd.in/eANcHJzZ --- Like this post? 🛎️ Follow Uriel Bitton to learn more about building with DynamoDB ♻️ Repost to share it with your network ☕️ Book me now for DynamoDB help: https://lnkd.in/ezS-ykFD

How to turn your AWS data lake into a lakehouse with Apache Iceberg? If you’re curious about open table formats and dynamic workflows, this guide is for you. Iceberg brings schema evolution, reliable transactions, and metadata management to your S3 data. With tools like Athena and Kestra, you can query and manage your data with ease. This post dives into creating and querying Iceberg tables, running updates and deletes, and even optimizing file storage. Whether it’s scheduled or event-driven data ingestion, Iceberg with Kestra makes it simple to build robust pipelines. 👉 Explore the full guide and stay tuned for upcoming tutorials on S3 tables and Iceberg! https://lnkd.in/ebjbaYHP

There are a number of ways to upload data to an S3 bucket on AWS. You can use pre-signed URLs, Cloudfront or other approaches but one simple method involves using API Gateway as a proxy. If you’re using smallish files (under 10MB) then this API Gateway proxy approach is likely your best option. With this approach, you setup an API endpoint that will result in files going to S3. You can also integrate auth with Cognito and have built in logging with this approach. The article below from Helio Tarnowski shows how to set this S3 upload option up via API Gateway. https://lnkd.in/edvdWBNm

Mastering DynamoDB Scan Pagination Recently, I encountered a critical challenge while working on a client's AWS infrastructure involving DynamoDB's ScanCommand. Many developers might not realize the nuanced behavior of DynamoDB scans, so I wanted to share my experience. The Problem 🔍: When using the ScanCommand, I discovered that DynamoDB does NOT guarantee the return of all data in a single query. This can lead to incomplete result sets if not handled correctly, potentially causing significant data retrieval issues. The Solution 💡:  To ensure complete data retrieval, I implemented a pagination strategy using two key parameters: ExclusiveStartKey: Specifying the starting point for the scan LastEvaluatedKey: Capturing the pagination token from each response By iteratively using these parameters, I created a mechanism that guarantees:  ✅ Complete data retrieval  ✅ Efficient pagination  ✅ Minimal performance overhead Tip 🚀: Always implement pagination logic when working with DynamoDB scans to prevent partial data retrieval and ensure data consistency. #AWS #CloudComputing #DatabaseManagement #SolutionsArchitecture #DynamoDB

Storage-first patterns using Serverless: ▫️ SQS-based queueing - $1.4 / million requests ▫️ DynamoDB - $2.25 / million requests ▫️ S3 - $6 / million requests 🔗 https://lnkd.in/dqdy6Cg6

🚀 Day 5 of AWS re:Invent 2024. Chalk Talk. [NEW LAUNCH] Accelerate data discovery with Amazon S3 Metadata STG353-NEW. Great discussion around this new service. My lessons learned: 1️⃣ In preview. 2️⃣ It uses new S3 Tables and Apache Iceberg under the hood. So, the same service quotas will apply. 3️⃣ Creates a table per bucket. 4️⃣ It’s a read-only table for customers, with several default metadata fields and an option to add custom metadata. 5️⃣ $0.45 / million requests + S3 Tables storage. 6️⃣ No data lifecycle/retention yet, so it is indefinitely stored until you turn the feature off and delete the table. #AWSreInvent #AWSS3 #ApacheIceberg