'Event driven' is used to describe a wide range of application and system architectures so it is important to define some context.

: Event driven distributed systems that use asynchronous exchange of events to deliver business functions.

When integrating with an other system I like to keep the exposed integration surface area as low as possible. What I mean by this is that only the required functions are available to the application and that the API is not directly accessed by the software that I am writing. In short - facade the provided API.

integrating event systems 1

Adding an adapter is arguably

This has several advantages;

There are many ways to integrate systems. The seminal work in this space is Gregor Hope book - enterprise integration patterns.

Ingegrating systems that have completely different architectures requires a mapping layer that addresses these compatibilty issues. Asynchronous event driven systems typically encounter these challenges.

integrating event systems 2

Batch data exchange has been the mainstay of system integration for decades. To be successful event driven systems need a way to work with systems that rely on bulk, infrequent exchanges of data i.e. Batch.

When we exchange data each systems has to reconcile the received data against its current view.

System A

System B

System B has no representation of a record received from System A. System B creates a representation of the data based on the record in the file.

System A

System B

System A has no representation of a record received form System B. System B creates a representation of the data based on the record in the file.

System A

⇐⇒

System B

System A and B had the same representation so no action required.

System A

⇐/⇒

System B

System A and B both have a version of the data record but disagree in the values.

When the two systems have different data representations we need to do more work to reconcile the differences. Simple reconciliation might involve time-stamp for when the data changed or to choose one system as the system of record.

Time-stamp reconciliation

uses a time-stamp generated by the source system placing the change on a time-line. Using the latest time-stamp means that the last change wins.

System of record reconciliation

One system is chosen as the system of record. This could be system A or B. Whichever one is chosen as the system of record is taken as truth and the other system honors the state.

System of record reconciliation becomes more complex when we control the system of record but the system we are integrating with does not honor that decision. If that’s the case then the system of record would need to update the system directly using an API.

Quite often batch data exchanges can be delayed because they rely on everything being in place infrequently. File formats change, data problems occur during preparation, data transfer and availability changes. Typical batch cycles of 24 hours can often end up being days.

Batch data comes in two flavors. Point in time state and logs. State data is typically the systems view at the point the file was generated. Be cautious about using the generation time for time based reconciliation because batch data takes time to prepare.

Log data files are a record of changes since the last file was generated. Log data is very similar to events - a point in time record of state. While attractive (from an event point of view) it is unlikely that event data boundaries between systems will match up. Some time ago working with a change log from an external system the system listed field changes first name, second name, date of birth. Our system had events for name change. This meant that field level changes had to be combined with other field changes or existing state so we could generate appropriate events.

Typical Integration patterns

integrating event systems 3

Direct Call integration

integrating event systems 4

In this pattern each service that needs to integrate with the external service or system does so independent of the other services. Changes in the service only affect a subset of the interaction. Changes to the integrated service require changes to all the services that are affected.

It is challenging to maintain a complete separation between event driven behavior and the integrated system. Concepts and data items naturally leak out of the API and start to invade the event service data model.

comments powered by Disqus