For a lot of smaller companies, streaming data usually comes from JavaScript tracking on websites or webhooks from SaaS tools, and tools like Segment or RudderStack tend to cover those needs out of the box. They handle web events and system events directly from an app—something that would often run through Kafka at larger companies. With features like HTTP web servers, JS tracking, debugging UIs, and batch exports to data warehouses, these tools work pretty well without requiring a full data platform setup.
I’m surprised Confluent or Redpanda haven’t targeted smaller analytics teams more. If they offered a higher-level option that reveals complexity as organizations grow, I could see Kafka working for smaller teams, with advanced features available as they grow.
Maybe there’s just not enough money in that market, so building a product mainly as a lead-gen tool rather than a direct revenue driver might be too much of a distraction. Still, I wouldn’t be surprised if Confluent eventually looks to capture this segment by buying RudderStack.
> Maybe there’s just not enough money in that market
Yeah, that would be my first guess.
Many vendors offer HTTP interface for writing to Kafka, but you need to build the client and tackle a few very tricky problems (batching, retries, delivery guarantees, etc.)
I think it'll require quite a bit of engineering effort, but my main question is who would be incentivized to make it an open source project? :) Maybe a BSL license will do the trick? But this also means it won't be an Apache project.
For a lot of smaller companies, streaming data usually comes from JavaScript tracking on websites or webhooks from SaaS tools, and tools like Segment or RudderStack tend to cover those needs out of the box. They handle web events and system events directly from an app—something that would often run through Kafka at larger companies. With features like HTTP web servers, JS tracking, debugging UIs, and batch exports to data warehouses, these tools work pretty well without requiring a full data platform setup.
I’m surprised Confluent or Redpanda haven’t targeted smaller analytics teams more. If they offered a higher-level option that reveals complexity as organizations grow, I could see Kafka working for smaller teams, with advanced features available as they grow.
Maybe there’s just not enough money in that market, so building a product mainly as a lead-gen tool rather than a direct revenue driver might be too much of a distraction. Still, I wouldn’t be surprised if Confluent eventually looks to capture this segment by buying RudderStack.
> Maybe there’s just not enough money in that market
Yeah, that would be my first guess.
Many vendors offer HTTP interface for writing to Kafka, but you need to build the client and tackle a few very tricky problems (batching, retries, delivery guarantees, etc.)
Nicu read! Would you you use a Kafka like solution for event sourcing to get the replayability?
It really depends! Is topic compaction enough to perform snapshotting? If not, it could be very challenging.
I think it's a great medium for writing events, but you probably need something more optimized for reads on the other side.
What I believe is missing is an open source WarpStream that scales down to zero effortlessly (perhaps also embeddable into applications directly)
Just a very lightweight Kafka that supports the API and lets you produce/consume from a distributed log.
Shouldn't be that hard to pull off?
I think it'll require quite a bit of engineering effort, but my main question is who would be incentivized to make it an open source project? :) Maybe a BSL license will do the trick? But this also means it won't be an Apache project.