Last night’s Data Professionals meetup reminded me why I keep showing up to Memphis tech events. Joey Brown didn’t just explain Kafka, but he brought it to life with clarity, energy, and a knack for making the complex feel approachable. Out of the 100+ events I’ve attended this year, this one stands out as one of the most compelling.
💡 What Is Apache Kafka?
Think of Kafka as a post office for data streams. Instead of letters, a service (producers) sends events. Kafka stores them as records and makes them available for other services (consumers) to pick up. Originally built by LinkedIn, Kafka shines when you need real-time, high-throughput messaging. The creators later launched Confluent, a company that supercharges Kafka with advanced features.
Use cases include:
Real-time analytics
System integrations
Event sourcing
Stream processing
🧠 Joey’s Big Lessons
Joey walked us through how Kafka brokers events between systems, helping multiple consumers access the same stream of data without creating bottlenecks. Some key nuggets:
Kafka ≠ database. You can’t query or edit data directly.
Data is split into partitions for fault tolerance.
Consumers use offsets to keep track of their place in the stream. Kafka guarantees events stay in order within a partition, but not across the entire topic.
Topics group related data streams. For example, all patient activity could go into one topic, with partition keys ensuring records for the same patient stay in order.
⚠️ Challenges to Watch For
Kafka is powerful but not plug-and-play. Joey cautioned us on:
Needing a dedicated owner (it’s not just another microservice).
No alerts for changed records.
No built-in tracking of unread data.
Difficulty finding specific records.
Tricky retention policies.
🚫 When Kafka Might Not Be Right
Internal job processing
Non-real-time workflows
Small-scale operations
🛠️ How to Get Started
Heroku offers a managed Kafka service that lowers the barrier to entry if you want to experiment without standing up your own infrastructure.
🎉 Why This Matters for Memphis
Joey Brown’s talk was a celebration of Memphis brainpower. Events like these remind us that innovation lives right here in our city. Let’s keep showing up, supporting local talent, and building a tech scene that reflects the brilliance of our community.
Check out the slides to the talk here: https://github.com/joeybrown/kafka/releases/latest/download/kafka-introduction.pdf
