Apache Kafka

A distributed event streaming platform used for:

• Distributed, high-throughput, real-time data streaming
  • It runs on many computers at once, so can handle huge amounts of data reliably.
  • Writes to disk efficiently in batches.
  • Spreads data across partitions and brokers.
  • Consumers can read in parallel.
• Decoupling services
  • Producers and consumers don’t need to know about each other.
• Durability & Fault tolerance
  • Messages are stored safely until read, so nothing is lost.
  • In a durable log on disk.
  • Consumers can re-read messages anytime.
    • Can recover from failures.
    • Can reprocess old data for analytics or debugging.
• Data pipelines (logs, metrics, user events)

Pull-based

• Consumers request messages from Kafka when they’re ready, Kafka NOT forcing messages on them.
  • More control to consumers
  • Able to handle high volumes without being overwhelmed.
  • Keeps messages in the log even after consumers read them

Delivery Semantics

Components

1. Producer

• Sends records (messages) to Kafka