Note: This blog post has been done as an independent pricing evaluation for the benefit of Vantage users and the broader infrastructure community. This is not a sponsored post and was not commissioned by either party.
Streaming is eating the world. In particular event streaming, where event producers like websites and apps generate user events 24/7 and feed them to event consumers like recommendation engines, fraud detection systems, and advertising platforms. Apache Kafka is the go-to open-source event streaming project and modern businesses are leveraging its capabilities to respond to customers in real-time. Two options to deploy Kafka at scale are Confluent and Amazon MSK. Each offers advantages and in those post we will explore how they price and examples where one may be favorable to the other.
Amazon MSK Serverless Pricing
This year Amazon released Amazon MSK Serverless which is a much more comparable competitor to Confluent than classic Amazon MSK which requires users to provision their own servers and manage scaling themselves. That said, MSK is the cheapest option in terms of pure server costs for experienced teams with well-understood workloads and limited feature needs. More on that below.
|Compute-hours||$0.75 per hour|
|Partition-hours||$0.0015 per partition per hour|
|Storage||$0.10 per GiB-month|
|Data In||$0.10 per GiB|
|Data Out||$0.05 per GiB|
|Connectors||$0.11 per hour|
|Schema Registry||Free via AWS Glue|
Kafka messages are organized in Topics which are distributed across Partitions which are replicated across Brokers which run within Clusters. All this enables Kafka’s legendary high availability. When Kafka clusters are running, charges accumulate for both CPU hours and partition hours. The number of partitions you create is a function of how much throughput you need for your brokers (basically the speed at which messages should be available). Finally, you can set message retention policies which will eat up storage costs. On MSK messages can be stored for up to one day. Data In and Data Out is exactly what you would expect.
|Compute-hours||$1.50 per hour|
|Partition-hours||$0.0015 per hour|
|Storage||$0.10 per GiB-month|
|Data In||$0.13 per GiB|
|Data Out||$0.06 per GiB|
|Connectors||$0.03 - $0.30 per hour|
|Schema Registry||$0.002 per schema per hour|
Confluent prices along exactly the same dimensions as MSK, although we should give them credit for establishing these first. The first thing to note is that Confluent costs more than MSK on a very basic level, but this comes with numerous caveats. First, Confluent offers discounts for longer term commitments, second Confluent has a “complete data streaming platform” and in our view that is an accurate description. Their feature set is much more comprehensive than MSK, and there is no need to glue (literally in the case of AWS Glue) multiple services together. In particular here are some major differentiating features from Confluent:
- Comprehensive admin panel (query topics from a web browser)
- Local development clusters
- Public access to cluster endpoints
- Managed connectors which need minimal configuration
- Deployment options beyond AWS including GCP and Azure
Unlike many AWS services, there is no free tier for MSK. Confluent does have a free tier which gives you access to most of the features of their platform to try. You must still pay for Storage, Data In, and Data Out at the rates above and there are hard caps of 1 GB/s and 5 TB of storage. You can create up to 10 partitions for free and Compute-hours are also free.
MSK vs Confluent Scenarios
Normally in our pricing comparisons we design hypothetical scenarios to demonstrate where one service or the other is cheaper. With the publicly stated pricing here, MSK is always cheaper than Confluent. In fact, if cost is your only concern you can consider using the non-serverless option for MSK and save even more money.
But at what total cost? Kafka is a complicated piece of technology and most businesses choose to offload the scaling, upgrading, and partitioning of it to a managed service built by Kafka experts. The cost to your business if the message broker which connects your applications goes down is likely far greater than the savings you realize by running the more bare-bones offering from Amazon. In the end, the best approach is to evaluate the level of Kafka expertise in-house and the consistency of your workload. On one side of the spectrum with less expertise and spikier workloads, Confluent’s breadth of features and support comes out ahead. On the other side, with experienced Java and Kafka engineers and a very predictable workload, you can save money with MSK.