If I'm reading this correctly, the Kafka topic only had 5 partitions, but they had 10 workers.

With the Spark direct stream, kafka partitions are 1:1 with spark partitions, which means at most half of the workers would be doing work without a shuffle.

Seems like a pretty basic oversight that should be addressed.

there were a few before but not many.