We've developed a LaTeX-based platform (https://www.overleaf.com) exactly for this type of audience :)

Yes, it supports inlined math formulas, through Mathjax (and KaTeX). It the upcoming 2.0.0 release, we actually render mathjax formulas to SVGs & PNGs so that they are visible in PDFs/EPUBs/Mobi.

Here's a very math intensive book for example: https://www.gitbook.com/book/jandeleeuw/bras/details

FYI: Citations \cite{} don't work in this example book

I suppose that's another target audience. Check out https://www.sharelatex.com/

There are MathJAX and KaTeX plugins:

http://plugins.gitbook.com/