I think the right phrasing ought to be that anyone can learn physics if they want to. The problem is that learning anything deeply is really hard, much harder than you think it'll be upfront. So, unless you're really motivated and really want to see it through, you're likely going to give up.

That being said, I believe that if you're a reasonably intelligent person (say at least above the median) then I don't see any major issue with you learning physics to some decent extent.