Probably. There's a membrane past the hole that you can destroy and I doubt the mic could function well enough to worry about if you punctured it.
But I usually just crush the whole mic package with pliers when I need to remove one. They pry off the board pretty easily in my experience.