We make a service that is API-compatible with AWS Kinesis but 75-85% cheaper. How? By using S3 for storage, which saves costs in several ways:
- There are no bandwidth charges or "PUT payload units" in and out of S3 within-region. This is a huge savings, anywhere from 50-85%.
- Shards can write large pages of records to S3 at higher throughput; with batching, per-shard throughput can exceed 100k writes per second (as opposed to AWS Kinesis' 1k/s shard limit), reducing the number of shards needed, sometimes by a factor of 10, 100, or more.
- There are no extra costs when retaining data beyond 24 hours, as the data goes directly into S3, the cheapest place to store data in AWS.
The service is simple to operate: it's packaged as a single container image, runnable at any scale on your infrastructure. It writes directly to your S3 buckets. And it's powered by a small open-source library in the Unison distributed systems programming language.
You'll use the exact same Kinesis clients you're using now. No code changes are needed. If you use AWS Lambdas to consume Kinesis, you can continue doing so. And migration is easy, requiring no downtime or global coordination across your org.
Aside from saving on direct Kinesis costs, with this solution, data starts out in S3 and you may find it is no longer be necessary to set up a separate process (say, with AWS Firehose or with a costly AWS Lambda) just to archive to S3 for longer-term storage. This can save further on costs while simplifying operations.
The tradeoff to using S3 is somewhat higher latency. Whereas AWS Kinesis might be 10-20ms latency to publish, writing a page of events to S3 will be 200-500ms. Many stream processing applications are fine with this latency and using S3 for storage can thus be a good tradeoff.
Sign up below to get a detailed cost comparison and a whitepaper on the system architecture.