If you wait for the temperature to drop, and the universe is expanding, distant galaxies will recede to the point that they are now moving away faster than the speed of light and their matter is no longer available to contribute.
Thank it is expanding now really means that it will expand forever? Are there not other physics at play that could stop that at some point, yeah I don't know much about any of this just wondering out loud.
This is generally unknown, of course. However it currently appears that the expansion of the universe is accelerating, i.e. it is expanding faster and faster.
Obviously it can't be ruled at that at some point this would stop and/or reverse. But there's no reason to think so, and if we're considering arbitrary future changes then we may as well consider that the universe might suddenly start heating up again in the future, or more mass will start appearing out of nowhere. Or god appears and hands out free decryption keys to everyone.
Gathering matter into one place heats it up, and prevents it from cooling (more or less - the outside can cool but thanks to the square-cube law that's basically negligible on universe-sized masses.)
