It's sexist whenever you have a big disparity that people exploit as an excuse for sexual harassment, whether a small number of women in a crowd of men or a small number of men in a big crowd of women. I purposely omitted any mention of gender in the grandparent comment.