I see where you're coming from, and they distinct, but I would not say 'completely irrelevant'. The number of toddlers on the planet relates to possible variations (in environment and genetics) and therefore possible discrepancies to the behaviour studied.

Yes, what matters is indeed those variations, not the amount of toddlers. If you randomly removed 99% of the toddler population the variations wouldn't be significantly reduced. Hence the sample size you'd need in that situation would still be roughly the same as now. Yes sure, if you removed 99.999999% of the toddler population it would get so small that you'd have appreciably fewer variations in the remaining population, so the population size is not completely irrelevant, but it's still overwhelmingly irrelevant in practice.

You do need a representative sample obviously, but that's a different issue.