I had a similar experience with a monthly recurring billing, and the helpful lady at the credit card company suggested getting my credit card reissued with a new number (effectively by marking it as lost / stolen).
Unfortunately pre-authorised recurring payments are not interrupted by this method. They will continue to show up on your statement against the old card number until you either pursued them to stop, or cancel your credit card entirely.
I would definitely charge back every unauthorized charge they make, and charge it back again if they fight it and win (yes, you can do this, and yes, it does work). It's not worth their time to fight small chargebacks at all[1], and the damage multiple chargebacks can do for even relatively large payments often makes it not worth going through multiple rounds.
Although at that point, I'd consider simply reporting the card as stolen. In effect, at that point it is. Once the PAN is marked as invalid by the issuing bank, those recurring charges should definitely not be able to go through (exception: some sort of wacky bill-pay system that bypasses the credit networks entirely; yes, these exist, although normally go the other way)
[1] As a merchant, your chargeback rate is unaffected by winning chargebacks, and the fee (typically starting at $15, and often marked up) is per-incident. Meaning if the customer fights the charge a second time after you win the chargeback, you're out $30 and now have two chargebacks in your history, not just one. Between the hard costs and whatever the human factor is in fighting the charges, it quickly becomes non-economical to fight them.
Unfortunately pre-authorised recurring payments are not interrupted by this method. They will continue to show up on your statement against the old card number until you either pursued them to stop, or cancel your credit card entirely.