I think the assumption is that given that other successful conferences have had speakers that were good (technically), that were non white/male, that if such speakers had been considered for this conference, the organisers wouldn't have ended up with a lineup of 100% white & male speakers.

You're right though that we have no real evidence for this, this is just a natural assumption to make given the evidence we do have (the lineup).

As a poster said above me, this could have possibly been mitigated by publishing who they considered and possibly even selected that were non-white/male and couldn't make it. (this would give us some of the evidence you mention).