Honestly, calling validation a trick isn't helping.
Understanding the motivation behind validation is an absolutely fundamental concept, and lack of coherence on the topic shows an inherent lack of understanding of the goal of building the model in the first place; GENERALIZATION.
This is synonymous with one checking in code that has no issues locally, without testing in the stack or a production environment.
I work and hire in this space and it's actually a bit shocking how widespread this lack of understanding is. Asking a candidate how to evaluate a model, even at a basic level, is this field's version of FizzBuzz. Just like Fizzbuzz, a lot of candidates I've encountered who are "trained" in machine learning or statistics fail miserably, and my peers seem to have similar experiences.
These issues are expected, given how popular data science is these days. We all win when more people are getting their hands dirty with data, but it's extraordinarily easy to misuse the techniques and reach misleading conclusions. This can potentially lead to people pointing fingers at the field and it's decline. The only thing we can do is correct the wrongs and do our best to limit incompetence that only serves to tarnish the field.
Count me among those who thought validation was a thing you just had to do when training ML algorithms. After all, the most beautiful theoretical model in the world is of no use if the predictions it delivers are terrible.
The real trick (for most algorithms) is to select the correct features to train against. This really is more of a black art than an exact science, so I think labeling it a trick is justified.
Understanding the motivation behind validation is an absolutely fundamental concept, and lack of coherence on the topic shows an inherent lack of understanding of the goal of building the model in the first place; GENERALIZATION.
This is synonymous with one checking in code that has no issues locally, without testing in the stack or a production environment.
I work and hire in this space and it's actually a bit shocking how widespread this lack of understanding is. Asking a candidate how to evaluate a model, even at a basic level, is this field's version of FizzBuzz. Just like Fizzbuzz, a lot of candidates I've encountered who are "trained" in machine learning or statistics fail miserably, and my peers seem to have similar experiences.
These issues are expected, given how popular data science is these days. We all win when more people are getting their hands dirty with data, but it's extraordinarily easy to misuse the techniques and reach misleading conclusions. This can potentially lead to people pointing fingers at the field and it's decline. The only thing we can do is correct the wrongs and do our best to limit incompetence that only serves to tarnish the field.