As discussed in this post on reducing Kinesis costs, Kinesis is one of the more expensive AWS services. Sticker price for a 1 million records per second data stream with 1kb records and 7-day retention costs over $800k annually (provisioned-mode) or over $4.2 million annually (on-demand mode).
The cost is due to a few factors. In on-demand mode, you pay for bandwidth in and out, over $3.6 million per year for the above example. In provisioned-mode, you pay for "PUT payload units", over $400k per year. You pay for shards which are capped at only 1000 records per second. You pay for storage beyond 24 hours. All this adds up.
Using S3 to back a data stream thus wins in a few ways:
- There are no bandwidth charges or "PUT payload units" in and out of S3 within region (making that $3.6 million or $400k annually become $0).
- Shards can write large pages of records to S3 at higher throughput; with batching, per-shard throughput can even exceed 200k writes per second, reducing the number of shards needed.
- There are no extra costs when retaining data beyond 24 hours, as the data goes directly into S3, the cheapest place to store data in AWS.
We offer a drop-in Kinesis replacement using this approach, and cost savings can be substantial (you may have discounts in place with AWS, so get in touch if you'd like a more realistic estimate for your workload). The product runs on your infrastructure, requires no code changes, and it is compatible with existing Kinesis client libraries on both the producer and consumer side. The endpoints look like Kinesis, just with different URLs.
Using S3 means we can lift restrictions on batch size (from 500 records to unlimited), records per second (from 1k to 10k, 100k, or more), and individual record size (from 1MB to unlimited). We also add a new feature: the ability to define incremental map-reduce jobs atop your data streams, without needing to operate a separate stream processing engine like Flink.
The product runs on your infrastructure anywhere you can launch containers. It is simple to operate, using a single container image that doesn’t rely on local disk and uses only S3 and an in-memory cache. The implementation is powered by a small source-available library in the Unison distributed systems programming language.
The tradeoff to using S3 is somewhat higher latency. Whereas AWS Kinesis might be 10-20ms latency to publish, writing a page of events to S3 will be 200-500ms. Many stream processing applications are fine with this additional latency and using S3 for storage can thus be a good tradeoff.