Kinesis is one of the more expensive AWS services, much of it due to bandwidth costs. Sticker price for on-demand usage is $0.08/GB to ingest and $0.04/GB to consume. A 200k records / s data stream with 1kb records costs over $750k/yr just in bandwidth, before including enhanced fan-out, data storage, and monthly per-shard costs:
This $756k in bandwidth becomes $0 using our drop-in replacement, which runs on your infrastructure, requires no code changes, and is compatible with existing Kinesis client libraries on both the producer and consumer side. The endpoints look like Kinesis, just with different URLs.
The cost savings comes from using an S3-based representation of data streams, completely avoiding bandwidth costs on both ingest and consumption, since in-region bandwidth to and from S3 is free. (You may have different discounts in place with AWS, or be using provisioned-mode, so get in touch below if you'd like a more realistic estimate for your workload.)
We also lift a few restrictions and add new features:
- On ingest,
PutRecordsbatch sizes aren't limited to 500 records or 5MB. - Shards are not limited to 1,000 record writes per second and 2,000 record reads per second. Depending on batching and record size, ingesting upwards of 200,000 records per second per shard is possible.
- Individual records are allowed to exceed 1MB.
- New feature: for any Kinesis data stream, define incrementally-maintained map-reduce jobs in a few lines of code, without needing to operate a separate framework like Flink or mess with AWS Lambdas.
The product runs on your infrastructure anywhere you can launch containers. It is simple to operate, just a single container image that doesn’t rely on local disk and uses only S3 and an in-memory cache. The implementation is powered by a small source-available library in the Unison distributed systems programming language.
The tradeoff to using S3 is somewhat higher latency. Whereas AWS Kinesis might be 10-20ms latency to publish, writing a page of events to S3 will be 200-500ms. Many stream processing applications are fine with this additional latency and using S3 for storage can thus be a good tradeoff.