How robust do you imagine it would be to just record the call / response pairs of the mutable objects in the new code and then replay them when running the experiment on the old code?
For example, suppose you have a db object and two versions of the code new_code and old_code. You call something that looks like:
experiment.run(new_code, old_code, mutables=[db])
Then the infrastructure runs new_code normally, but records the arguments and return value of every call to db (and any other object defined as mutable). Then, the infrastructure runs old_code, but whenever a method of db is called, it tries to match it with a call made by new_code and just directly returns the return value that call returned. If it can't match the call, it signals an error, but it never actually tries to call db, thus negating the risk of side effects.
Obviously this would fail when the two versions perform different operations in the database, even semantically equivalent, but non-identical operations (say one retrieves a value and increments it inside a transaction, while the other uses a stored procedure in db to increment without fetching). But it still relaxes the constraint, now you can do this for code that has no side effects and code that has the exact same side effects as represented by exact call/return pairs to mutable objects.
> How robust do you imagine it would be to just record the call / response pairs of the mutable objects in the new code and then replay them when running the experiment on the old code? […] Obviously this would fail when the two versions perform different operations in the database, even semantically equivalent
I'd think the latter would be the common expectation by virtue of a different implementation.
For example, suppose you have a db object and two versions of the code new_code and old_code. You call something that looks like:
experiment.run(new_code, old_code, mutables=[db])
Then the infrastructure runs new_code normally, but records the arguments and return value of every call to db (and any other object defined as mutable). Then, the infrastructure runs old_code, but whenever a method of db is called, it tries to match it with a call made by new_code and just directly returns the return value that call returned. If it can't match the call, it signals an error, but it never actually tries to call db, thus negating the risk of side effects.
Obviously this would fail when the two versions perform different operations in the database, even semantically equivalent, but non-identical operations (say one retrieves a value and increments it inside a transaction, while the other uses a stored procedure in db to increment without fetching). But it still relaxes the constraint, now you can do this for code that has no side effects and code that has the exact same side effects as represented by exact call/return pairs to mutable objects.