This is something I've been thinking about for awhile now that I'm pursuing my own startup. Coming from a background in HPC, my papers were published using prototypes and proof-of-concepts. On the other hand, a product needs to work at all times. Hence, the idea of an MVP from a systems standpoint is rather fanciful -- if an MVP works for 1TB of data (or worse, 5TB of data), but doesn't work for 10TB or 100TB, then re-normalizing after the fact is a huge issue.
As a question to other startups based on novel systems, did you feel that an MVP was the right way to go? If not, what did you do differently so as to not labor in "almost, but not quite ready" limbo.
As someone who is building a startup that deals with slightly-more-than-usual data and who has been involved with various startups that similarly pushed the data-boundaries in later phases: Yes.
Make it work. Make it right. Make it fast. In that order.
Restructuring data is, in most cases, a smaller issue than it may seem. Running systems don't normally crash and burn over night. More often you'll see it coming and have plenty time to apply bandaids while transitioning to a new system.
Doing so is much easier when you can actually see and measure the bottlenecks, rather than trying to predict them beforehand.
I think MVP does not mean "crappy product". MVP means just a version of product (which is working, tested, etc.) which can collect the maximum amount of validated learning with the least effort.
In other words, my understanding is that MVP should cover main use case in order to validate if that is what your potential customers want (as my mentor was saying "it must not suck"). So, in your example, if scaling up to 100TB is the key value proposition then you have to have it. On the other hand, CLI, nice GUI, nice installation, etc. are not probably part of your MVP.
As a question to other startups based on novel systems, did you feel that an MVP was the right way to go? If not, what did you do differently so as to not labor in "almost, but not quite ready" limbo.