I think a fair bit of his non-cryptography security advice of the past 10+ years has been...different than a lot of people I know better and have direct evidence of their competence would give. Increasingly so recently (the past year or two). As a cryptographer, particularly on the symmetric side, he does a good job (at least, the other people who I know who are good at that also think he does a good job; I understand number theoretic cryptology better than I understand the more complex details in designing symmetric stuff). He also went way far over to the "high level policy/politics" side post-BT acquisition vs. actual implementation work, other than crypto competition entries, as far as I can tell.

So, it's not so much "not trust" as "critically evaluate what he says each time".