Recommend the paper "A Pendulum Swung Too Far"[0] as a nice take on Rationalism vs Empiricism. Interestingly, the paper was written in 2007 when the current crop of deep learning methods were not around (the provocative Hinton/Sutsekever/Krizhevsky paper was in 2012) ... so the pendulum has swung even farther! To the point that, as the post points out, we don't even have good statistical justifications for network architecture design choices. We have some answers, for some choices, but mostly a compendium of techniques empirically validated to work very well.

[0] http://languagelog.ldc.upenn.edu/myl/ldc/swung-too-far.pdf