Using Event-Driven Design with Apache Kafka Streaming Applications ft. Bobby Calderwood

Streaming Audio: Apache Kafka® & Real-Time Data

What is event modeling and how does it differ from standard data modeling?
In this episode of Streaming Audio, Bobby Calderwood, founder of Evident Systems and creator of oNote observes that at the dawn of the computer age, due to the fact that memory and computing power were expensive, people began to move away from time-and-narrative-oriented record-keeping systems (in the manner of a ship's log or a financial ledger) to systems based on aggregation. Such data-model systems, still dominant today, only retain the current state generated from their inputs, with the inputs themselves going lost. A converse approach to the reductive data-model system is the event-model system, which is enabled by tools like Apache Kafka®, and which effectively saves every bit of activity that the system generates. The event model actually marks a return, in a sense, to the earlier, narrative-like recording methods.

To further illustrate, Bobby uses a chess example to show the distinction between the data model and the event model. In a chess context, the event modeling system would retain each move in the game from beginning to end, such that any moment in the game could be derived by replaying the sequence of moves. Conversely, chess based on the data model would save only the current state of the game, destructively mutating the data structure to reflect it.

The event model maintains an immutable log of all of a system's activity, which means that teams downstream from the transactions team have access to all of the system's data, not just the end transactions, and they can analyze the data as they wish in order to make their own conclusions. Thus there can be several read models over the same body of events. Bobby has found that non-programming stakeholding teams tend to intuitively comprehend the event model better than other data paradigms, given its natural narrative form.

Transitioning from the data model to the event model, however, can be challenging. Bobby’s oNote—event modeling platform aims to help by providing a digital canvas that allows a system to be visually redesigned according to the event model. oNote generates Avro schema based on its models, and also uses Avro to generate runtime code.

EPISODE LINKS

• Event Sourcing and Event Storage with Apache Kafka
• oNote
• Event Modeling
• Toward a Functional Programming Analogy for Microservices
• Event-Driven Architecture - Common Mistakes and Valuable Lessons ft. Simon Aubury
• Watch the video version of this podcast
• Coding in Motion Workshop: Build a Streaming App
• Kris Jenkins’ Twitter
• Join the Confluent Community
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Intro to Event-Driven Microservices with Confluent
• Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

פרקים

1. Intro (00:00:00)

2. Data Modeling vs Event Modeling (00:01:55)

3. Data-thinking mindset (00:10:15)

4. From records of states to data model (00:15:44)

5. Conway's Law (00:20:41)

6. Use case (00:22:47)

7. Test-driven development (00:25:58)

8. Event modeling is so visual (00:28:25)

9. Career experiences and stories (00:29:49)

10. Recording data as events (00:34:12)

11. Event modeling yields benefits (00:35:38)

12. Creating oNote (00:37:14)

13. Avro code generation (00:43:14)

14. Solving different mindsets in one web tool (00:47:03)

15. It's a wrap (00:49:05)