A linear SVM can in turn be expressed as a very shallow neutral network. The main difference is that with SVMs you put all your effort into transforming inputs for the model (e.g. all the popular kernels) while with neural networks usually most of the effort goes into clever model architectures.

There is probably a ton of isomorphism between different models. It may come down to what is easiest to understand and fastest to implement in code.

See also:

A visual proof that neural networks can approximate any function

https://news.ycombinator.com/item?id=19708620

So a "neural network" is actually a type of parameterized mathematical function that can be fit to any curve including higher dimensional surfaces, etc.?

That's correct. See https://en.wikipedia.org/wiki/Universal_approximation_theore... for more details

Yes.