> Scientist appears to be largely limited to situations where the code has no "side effects".
That's one of the things I was initially thinking too, but then as I thought about where I could have used it in the past, I can think of only a few cases where it wouldn't be possible to keep it isolated.
For example, have your new code running against a non-live data store. Example: When a user changes permissions, the old code changes the live DB, while the new code changes the temporary DB. Later (or continuously), you could also compare the databases to ensure they're identical (easier if the datastore remains constant, a bit harder if you are changing to a different storage schema or product).
Where it would be in the difficult-to-impossible range is when touching on external services that don't have the ability to copy state and set up a secondary test service, but even in that case, you could record the requests made (or that would have been made) and ensure they both would have done the same thing.
That's one of the things I was initially thinking too, but then as I thought about where I could have used it in the past, I can think of only a few cases where it wouldn't be possible to keep it isolated.
For example, have your new code running against a non-live data store. Example: When a user changes permissions, the old code changes the live DB, while the new code changes the temporary DB. Later (or continuously), you could also compare the databases to ensure they're identical (easier if the datastore remains constant, a bit harder if you are changing to a different storage schema or product).
Where it would be in the difficult-to-impossible range is when touching on external services that don't have the ability to copy state and set up a secondary test service, but even in that case, you could record the requests made (or that would have been made) and ensure they both would have done the same thing.
Definitely an interesting concept overall.