Streaming Data Best Practices

Newsletter highlights.

This post is a periodically updated collection of publications from this newsletter highlighting the best practices for building data streaming pipelines and platforms.

• Start with Streaming-First Architecture. Even if you’re not going to follow it, it offers a lot of food for thought.

• Design your messages with Approaches for Defining Message Metadata.

• Decide on using Changelog vs Append-Only Data Streams.

• Understand how to ingest operational data with Change Data Capture Is Still an Anti-pattern. And You Still Should Use It.

• Understand how to ingest event data with Building Event Ingestion HTTP API for Kafka.

• Decide what data streams to materialize after reading Considerations for Data Stream Materialization.

• Determine if exactly-once delivery is a must with Do you really need exactly-once delivery?

• Think about stateful stream processing after reading State and Timers.

• Consider how to deploy your stateful applications with Blackhole Sink Pattern for Blue-Green Deployments.