A large enough company, which can have thousands of systems running in prod, needs someone to be the gardener pulling out the weeds (systems which duplicate each other's functionality, systems which try to introduce obscure technology for no benefit etc.) and also design the company's systems landscape (start initiatives, kill existing ones, move data and functionalities between systems etc.) so that it fits company's goals. Typically, you have to come up with some strategic vision and then convince all the high-level managers affected to commit to it - and then steer the execution across all teams, so that all the pieces come together. It's basically big picture job, and very difficult to do well, since it doesn't come with any managerial power or budget - EAs are the Cardinal Richelieus of corporate world.
I'm watching what happens when enterprise architecture is done badly, right now. I'm currently assisting a huge agency that is a mish-mash of smaller agencies. Multiple clouds, several data centres, hundreds (if not thousands!) of "apps" and servers. Every combination of bare metal, VMs, and cloud IaaS, PaaS, and SaaS. Much of it overlapping or redundant functionality.
It's an unmitigated clusterfuck: A nightmare tangle of interdependencies, band-aids, duplicated systems, inconsistent DR, non-existent DR, partial HA, and on and on.
If you've ever seen a "spaghetti base" in Factorio, it's like that, but as if a hundred players had been building one for a decade. Not just spaghetti, but multiple interwoven styles of spaghetti.
It's hard to explain to someone who hasn't seen it. Recently, someone upgraded a Layer 7 load balancer in a legacy data centre and broke an application in an unrelated Public Cloud because someone hard coded a HOSTS entry for the internal certificate CRL distribution point. Why was a cloud service using legacy internal PKI? Reasons. Why did it need a HOSTS entry? More stupid reasons.
The technical details don't matter, of course. The moral of the story is that because they don't have a good EA at the helm, they're still adding to the spaghetti.
I'm watching this unfold in slow motion. Layers of new stuff just draped over the existing garbage. No uplift. No clean-up. Nothing ever turned off. Just buried under new strata, slowly fossilising.
It's fantastic yet horrifying to watch, like a slow motion shipwreck.
The irony is that with enough experience, you get a gut feeling for these kind of scenarios and the only thing one can do is to fasten the seatbelt and brace for impact, because of the politics involved no one really wants to listen.
The way my Silicon Valley company manages this is basically by having an infrastructure org. There is a catalog of about 3 managed storage services. One stateless compute platform. One HTTP ingress platform. One message broker. One workflow orchestrator. One RPC fabric. Two supported languages with central CI infrastructures.
There are hundreds of teams serving specific business goals, and thousands of services between them, but everything gets made out of substantially the same parts. No one person has the 10,000 foot view, but the platform is sufficiently comprehensive and it's sufficiently hard to deviate, that no one really needs to.
Now if you were hoping to have each team build from raw materials while still having coherence between projects, I can see why you might think you need this role. But I submit that it's probably simpler and more effective in the long run to just give people the building blocks you want them to use.
Ah reasons... we've all had to approve horrible kludges to get something delivered.
The truth is you never get completely clean beautiful architecture or code, there are always trade-offs to make along the way.
Most change takes place in individual work streams, which are most concerned with delivering their scope on time and budget. There is typically no budget to consider changes that have wider scope or which might derail delivery. So kludges.
But without the longer term vision and commitment to paying down technical debt it becomes spaghetti as you say, whether you're a developer or an architect.
Working at a big consulting company where we have enterprise and senior technical architect paths. The EA reaches salary wise one level higher. I deliberately stay on the technical one as doing the EA from the outside as consultant i.e. from the outside is really hard even if positioned correctly. The nature of the EA job is political and long term. Often product/service follows organization so to affect change one needs to reshape the org in selective places requiring really high level sponsorship.
