What should an Architect with Kafka experience recommend?

Universal Containers (UC) uses Apache Kafka on Heroku tostream shipment inventory data in real time throughout the world. A Kafka topic is used to send messages with updates on the shipping container GPS coordinates as they are in transit. UC is using a Heroku Kafka basic-0 plan. The topic was provisioned with8 partitions, 1 week of retention, and no compaction. The keys for the events are being assigned by Heroku Kafka, which means that they will be randomly distributed between the partitions.

UC has a single-dyno consumer application that persists the data totheir Enterprise Data Warehouse (EDW). Recently, they’ve been noticing data loss in the EDW.

What should an Architect with Kafka experience recommend?
A . Enable compaction on the topic to drop older messages, which will drop older messages with the samekey.
B . Upgrade to a larger Apache Kafka for Heroku plan, which has greater data capacity.
C . Use Heroku Redis to store message receipt information to account for "at-least" once delivery, which will guarantee that messages are never processed more than once. Scale up the consumer dynos to match the number of partitions so that there is one process for each partition.

Answer: C

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments