How does this work with both? Is there a defined band plan standard that cable operators know to avoid? Do you need to install a notch filter on the line coming into the house?
It's more the other way around. There's a defined band plan, because historically cable TV was just transmitting radio signals over coaxial cable instead of radio, so it inherited the broadcast TV band plan. Then digital TV inherited the analogue cable band plan and added higher-frequency channels.
To answer your question, cable TV uses frequencies between about 55MHz to 1GHz, but mostly starting at about 500MHz for digital cable. So if I wanted to transmit Ethernet over cable, I'd use a 2GHz band or something to avoid interference.
> historically cable TV was just transmitting radio signals over coaxial cable instead of radio
Well, to be pedantic, that is still how it works. And it's how cable Internet works. And it's also how a lot of techniques over twisted pair work, too.
The conductor is used as a waveguide and a very-much-analogue signal encoding a digital signal is what is sent over the line. DOCSIS 3 uses up to QAM-4096, which can encode 12 bits in a single symbol on the line, by using multiple steps of amplitude shift and phase shift, to encode bits. Quite similar to how FM radio works, just digital steps, rather than an analogue continuum between 0 and 100 amplitude and 0 and 100 phase, at the decoder.
This has even started showing up for links within a single computer, now. The latest revision of PCIe uses modulated RF (4-level pulse amplitude modulation) rather than simple binary voltage levels.
MoCa was designed specifically to work over existing coax used for TV and behave with existing signals. The board of the consortium that designed it includes Comcast, Cox, and Verizon. It was basically a solution for "we're gonna need internet in our customer's homes but we already ran coax in them"
I think it just sits in the mutli-ghz bands. It shouldn’t conflict with cable, which uses lower frequency bands. You just gotta make sure to install a low pass filter at your demarcation point so you aren’t broadcasting your MoCA stuff beyond your home.
You also have to use appropriate splitters that are rated for the top ends of the spectrum.