What kind of storage was that library on? We have users with 100, 150k files who are largely happy.

If you can replicate this on a new release (4.4.0 was out recently), please file a bug! [1]

[1] https://github.com/quodlibet/quodlibet/issues