Who, exactly, is it supposed to be "nearly" an actual slur against?

Because I'm sure that actual landlords don't have a problem being referred to as landlords, so you must be referring to some other group.