That a joke is funny because it plays on the racial stereotypes in an unexpected way. For example:

"What is a game called where a bunch of white guys are chasing a black guy? Golf."

I think the joke would be less funny if the races were reversed (and a different, but appropriate game was used).