The joke is that they probably have the same father and the emperor's mother wasn't faithful, basically the reverse of what the emperor was implying.

The joke goes further than that.

This implies that the emperor may not be heir to the throne. Or, may not be 'high born'. This was 2000-2500 years ago, perhaps.

Legitimacy came from the male in these times.

So, having a shared father means nothing bad for the emperor. Having a shared mother means his title, his lands, his bloodline may be entirely false, and thus, his legitimacy.

(Even if the emperor rose and claimed his place by blood/conflict, the above all still counts in many ways.)

The provincial man is basically saying that his father did Augustus' mother.

I mean the contemporary idiom is Moor concise, but less eloquent

a veiled reference to “Your mother” joke.