Semprini

Ya know, I love me a question mark. The elegant sickle like curve and dangling dot, those little paired words of how? and why? It's much nicer than a full stop (or period for my American colleagues). Why do architects seem to prefer the latter?

There is a Proof of Concept below the rant - I beg your indulgence dear reader. The PoC uses Kafka and python to transfer files based on the argument laid out below. The GitHub project is here: mft-kafka

As an architect it's my job to talk and write a lot of bollocks. One must constantly be on the forefront of buzzword lore to maintain ones architectural standing and feeling of general superiority over lesser IT peons.

However, in this obviously correct and proper feudal type system, what sometimes gets lost or confused is the practicality of implementation and shared understanding. The same architectural buzzwords can be interpreted in so many ways so it becomes difficult for people implementing to realise the actual benefit. This means uninspired doers either take no notice or go off on a tangent which looses any architectural benefit intended.

This is why I'm not a fan of ivory tower architects and IT leaders who shoot Google arrows and Gartner bolts from the parapet, and I prefer to wield a bloody big sword in the melee of IT implementation. This blog is about a quick PoC I had a bash at for a pipeline which streams from a simulated on-prem source to a data lake-house.

The hint is in the name people. Proof of Concept. Why is this so difficult to understand? The point of a PoC is to show that a thing can be done, not how a thing can be done. I.e. we don't know something but we have an idea - awesome, I'm here to help. Where I seem to part ways is when the PoC plans to deliver best practice architecture, continuous delivery and push the code & pipelines into production.

Where is the truth of an organisation? Many programmers would say that the truth is in the code as this is what is actually running the organisation. The code is in effect a realization of business demand and therefore the truth of what an organisation actually is can be found within.

I think that while somewhat accurate, it is a little 'ambulance at the bottom of the cliff' thinking and we should rather strive to be declarative in what the organisation is. Also, assuming we buy for commodity then the view is obscured and incomplete. The declaration of a business is best done via modelling as the related views can inform and align everyone from board to developers.

The famous dig from Jamie Zawinski:

Some people, when confronted with a problem, think "I know, I'll use versioning." Now they have 2.1.0 problems.

My assertion is that integration versioning should be a last resort, used in specific situations. If you own both sides of the integration contract then do not version.

Versioning is in effect kicking the can down the road for your future self, or worse - leaving it for someone with no knowledge of the interface.

By following Postel's Law, using abstraction layers, micro-service boilerplate and not being afraid of regression testing (I.e. modern IT practices) we have simple to change components which we can track and upgrade.