I think there's a little more to it than that. If the exception in question can be seen to have specific properties which are clearly the cause of its exceptionality, and if those properties can be seen to be rare, I think the exception supports the conclusion that the rule is a pretty good one, by showing what an unusual combination of circumstances is required for it to be violated.

That's pretty much what the saying is about. The existence of an exception proves the existence of a rule. The reliability of the rule can be estimated by considering the exceptionality of the exception.

I like the alternate interpretation that understands "prove" to mean "to test" in this situation, so that "the exception tests the rule".