> The problem I find is that many data teams get thrown into having to design data infrastructure with little experience actually setting up one in the past. Don’t get me wrong, we all start with no experience. But it can be difficult to assess what all the nuances of different tooling and designs can be.
I've been there. Companies can be cheap about training and I was given none before building some sophisticated data stuff that surprisingly worked, but probably could have been simpler.
I got a much better job soon after, and hopefully my replacement got some training.
I think that one of the biggest issues is that a lot of training is flat out horrible. The author preaches simplicity and working directly with the business, but mean while you have entire teams of developers being taught something like Clean Architecture, SOLID, DRY and all sorts of fancy things that when used poorly leads to extreme over-abstractions.
So even the most well meaning engineers with good training can go about building data structures which won’t last very long once the key people leave. Simply because what they were taught doesn’t work.
Maybe better recognition of data architecture as a specific domain would help. I'm a data architect, I have a specific set of skills to build and maintain data platforms, but when applying for jobs I'm very often finding myself in 1 of 2 situations.
1) I'm interviewed by software developers / architects, who expect me to have strong knowledge of software development patterns (like OOP patterns or microservices design in webapps). Well I know the basics but not at senior level. That's also not what's actually needed to design and implement a data platform.
2) I'm interviewed by data engineers, and I'm expected to show expert knowledge in data pipelines implementation. That's a lot closer to what I can do, but it's still not the core of my skill set, and it's a bit frustrating not to be judged on that.
I've been there. Companies can be cheap about training and I was given none before building some sophisticated data stuff that surprisingly worked, but probably could have been simpler.
I got a much better job soon after, and hopefully my replacement got some training.