Neat idea, be curious to see it implemented! You can probably get away with just stopping at the bounds of the dirty area. Sure, the error would accumulate, but they'd be localized to a single pixel. I doubt it would be visible.

One issue I suspect you'll encounter is that redrawing a restricted area during movement will cause that area to flicker weirdly while the rest of the image stays relatively stable.