Why Event Driven Platforms give you back control of your data.

Real time messaging, batch processing, stream processing — just a few descriptions of frameworks and platforms that have been thrown around without much thought behind from the leaders that be. At Enfuse we’ve been through the laundry cycle of data processing platforms time and time again. Rarely, if ever (and by that I mean never), does a single pattern or paradigm fit an entire enterprise. We can wrap all of these patterns up in a platform we call the Event Driven Platform (EDP). Developers who write code and implement solutions within this platform use Event Driven Design (EDD) coupled with Test Driven Development (TDD). Whoa, acronym city!

Depending and where you’re company is at in any particular hype cycle graph (remember Big Data and Hadoop?), you’ve no doubt heard the terms above given more credence than they probably deserve. Typically it happens like this,

We need real time processing guys. The only way to stay a head of the curve is if we implement an entirely new platform utilizing this so we can stay ahead of the curve. Just think about how easy real time Machine Learning will be once we have move our entire company towards this pattern.

Thinking like this is categorically false. Large enterprise corporations are never typically made up of a single language or stack. Typically they are made of a number different acquisitions, complete with employees and technology stacks that don’t align with their core strategy in mind. The enterprise might be a completely Java driven stack incorporating something like Spring Cloud Stream on Cloud Foundry or Kubernetes. Company acquisition A may be a C# stack with only MS SQL Server. Herein lies a dilemma.

The enterprise is clearly going to want to consume the data that Company A produces continuously. But how do we do that? Are we to task our Enterprise developers with one-off application development stories that connect and consume from a series of esoteric legacy data repositories (that we know will be phased out sooner rather than later)? Just take a step back at this point and think about the implications of that. This requires your developers to be domain experts with respect to Company A, that takes time. This requires your developers to figure out how to keep state so they are continuously grabbing correct data, that takes time. This requires your developers to figure out how to write code to follow whatever pattern has been mandated (real time, batch, stream processing) and notify downstream that a job is finished, this takes time. What if Company A is completely batch driven but the Enterprise has mandated real time message processing? The complexity easily becomes overwhelming for the developers, and clearly, time is not on their side.

So how do we mitigate situations like the above? Introduce the Event Driven Platform. The authority of the business unit to dictate exactly what they feed into the platform is the basis for Event Driven Platforms. This pattern put’s control of the data, what is processed, and how much data back into the business unit’s control. Whether it be batch, real time, or streaming — series of events are emitted to the platform which recognize these events and kick off corresponding data pipelines.

If the business unit is sending device created IoT data, then each payload pushed would be considered an event. Further, if there is a batch of data that is ready to be transmitted, then instead of the developers creating applications to grab that data, an event would be created by the actual push of the data into corresponding cloud data repositories or on-prem repositories. Each file or piece of data may then trigger of an individual event letting the platform know it’s time to do work.

The premise is simple, the business unit that owns the data feeds the data to the platform in whatever mechanism they are comfortable with. They’re now fulling in control of what data is being processed or given to the platform. The idea seems obvious, but in practice, rarely if ever is this pattern effectively deployed.

At Enfuse the benefits of Event Driven Platforms are endless. Our data pipelines end up being stateless by using Event Driven Design coupled with Test Driven Development. This allows cleaner more generic code that multiple business units can take advantage of. Instead of writing one-off applications, we create EDD applications that can be scaled out in a stateless manner to handle whatever load capacity we’re anticipating, either in advance, or dynamically.

This is but one in a series of postings where we’ll be talking about Event Driven Platforms and Event Driven Design (EDD). So look forward and keep an eye out for the next posting that will contain an example EDD application!