Another factor is that what one wants from a neural network is the best-generalizing approximation. What is this best generalizing thing almost seems like a property of the world rather than a mathematical curve but still, to various degrees one would guess a rough approximation might turn out to more generalizing than an absolutely exact approximation (since you talking about approxing a set of data with with curve rather than, say, getting the closest approximation to known-correct equation)
Yup. Soon after I heard about curve fitting, I thought, maybe we can always fit a polynomial! So, for 2 points, we can use a polynomial of degree 1, 3 points, degree 2, etc. Think a little and see how to fit n points with a degree n-1 polynomial. Can just write it down or just program it. So, I rushed to do that.
I typed in a few points and got a nice fit; the polynomial went exactly through the points. Hmm ... I couldn't be the first to think of that! So, I typed in points something like a square wave and thought, "Let's see a polynomial fit that!". Well, it did fit -- a polynomial fits a square wave -- gotta be kidding.
So, I plotted out the whole thing. Below, spoiler!!!!
Sure, the polynomial went through the points but between the points it shot off toward either positive infinity or negative infinity and made a lot of progress before turning around.
Then there are spline fits, least squares spline fits, multivariate spline fits, and, ..., ironically, neural network fits.
It's important to recognize that the space of all programs that take x,y as input and then output 1 or 0 is equivalent to the mathematical space of all possible functions. We interpet the programs as indicator functions.
Neural networks are usually smooth because they must be differentiable. Stochastic binary neurons and other techniques take us into a larger space of possible mappings, discontinuous ones.
But yeah, ultimately the model we solve for, will affect what kind of interpolation the solution ends up having.
The biggest question is what kind of interpolating techniques the brain uses. The search for the magic model....
