My honest knee-jerk reaction to this is that it's not morally wrong for the exact reasons you gave. But then again it is a hypothetical that is all but impossible to occur in real life, as even the fact that the man is being judged would require that we (and thus the woman as well) know it happened.
I prefer your kind of bullet-biting consequentialism to the kind which tries to approximate the intuitively correct (imo) verdict by watering down the concept of consequence (e.g. by going for some version of rule-utilitarianism).
But I disagree with the second point. Consequentialism is a philosophical claim to the effect that an act is morally wrong if and only if such-and-such conditions obtain. If it's possible to imagine a situation, however recherche, involving a morally wrong act but in which "such-and-such conditions" do not obtain, that automatically refutes consequentialism and it either has to be revised or given up altogether.
The counter-intuitive finding here -- which is in favor of rule-utilitarianism -- is that everyone trying to optimize every difficult decision does reliably lead to unpredictable consequences that are worse than the "follow a rule that would lead to optimific outcomes if performed by 99% of the population" set of consequences.
That is, you can hold a belief that the optimal "decision criteria" for an act-consequentialist is actually a rule-consequentialist one, and this belief is fairly common.
Restated: being an act-consequentialist doesn't absolve you from having to determine act-consequentialism's best decision criteria, because if it did then you wouldn't be being act-consequentialist about it. It's recursive like that.
