Hello, people of the internet, browsing random IT blogs - and this one with a Gilbert and Sullivan reference! We are indeed esoteric and interesting.
Confused? Excellent. That's a great place to start.
But first, hats.
As the Very Big Corporation of America research shows, we're not wearing enough hats...
Hello, people of the internet, browsing blogs to do with IT and Data. You've found the 2nd part of a data blog which looks at using patterns to form data products. Welcome, dear reader.
In the previous blog, I made some bold claims - promised some patterns to unlock all the goodness, save hundreds of millions, speed everybody up and unify classical physics and quantum mechanics. Ok, not that …
Sounds like I'm selling a silver bullet doesn't it? Actually no. What I'm selling is a butt-load of hard work, arguments and pain. And I'm trying to do this before vendors subvert the meaning of the Data Product and Data Mesh terms to the point where it becomes more meaningless buzzword bullshit.
Now, I'm not quite sure if that's a click batey title or not. I guess I see it as subverting expectations, because you see dear reader, I'll be arguing for the proposition that anarchy is a good model of operating IT in a large enterprise.
Bollocks! I hear you roar, or possibly chuckle to yourself if I'm not engaging in hyperbole. But none the less, I expect that you think it's a case of another architect, up in their ivory tower smelling their own farts.
Now, lets see if I can extricate myself from this hole that I've dug shall we?
Ya know, if your Data, AI and Tech Governance is not making your organisation faster, and less risky, then you're doing it wrong.
Standards are a governance superpower, open to all - and they benefit everybody.
If your organisation finds itself chasing its tail, with those expected "quick wins" or "low hanging fruit" not turning out to be either quick or low hanging, then I'll bet you're not taking advantage …
Ya know, I love me a question mark. The elegant sickle like curve and dangling dot, those little paired words of how? and why? It's much nicer than a full stop (or period for my American colleagues). Why do IT professionals seem to prefer the latter?
There is a Proof of Concept below the rant - I beg your indulgence dear reader. The PoC uses Kafka and python to transfer files based on the argument laid out below. The GitHub project is here: mft-kafka
As an architect it's my job to talk and write a lot of bollocks. One must constantly be on the forefront of buzzword lore to maintain ones architectural standing and feeling of general superiority over lesser IT peons.
However, in this obviously correct and proper feudal type system, what sometimes gets lost or confused is the practicality of implementation and shared understanding. The same architectural buzzwords can be interpreted in so many ways so it becomes difficult for people implementing to realise the actual benefit. This means uninspired doers either take no notice or go off on a tangent which looses any architectural benefit intended.
This is why I'm not a fan of ivory tower architects and IT leaders who shoot Google arrows and Gartner bolts from the parapet, and I prefer to wield a bloody big sword in the melee of IT implementation. This blog is about a quick PoC I had a bash at for a pipeline which streams from a simulated on-prem source to a data lake-house.