There are two basic responses to the inconsistent/volatile issue that you bring up.
The first is mitigation of risk. We generally A/B test versions that are similar enough that we'd be comfortable rolling out the winner to the whole site without warning. As a result they tend to be fairly small changes, which goes a long way to solving the inconsistent/volatile issue. Furthermore you associate users with an A/B test slice so while different users get different experiences, a single user's experience should be consistent across their experience with the site. As a result of these two factors the impact of A/B testing is very small.
The second is magnitude of reward. Most websites that do not currently do A/B testing can find site improvements which will give business improvements of 20% or more. That's a pretty clear financial incentive to put up with unhappy users. But in fact many of those improvements made your users happier. So the "construction in progress" is far, far outweighed by the rewards.
This is why companies like Microsoft, Google and Amazon use and strongly advocate A/B testing. Google has created free tools to help people A/B test websites. I know that Microsoft has an internal version of that, and suspect that they intend to open access at some point as well.
If you want to learn more about how to build an A/B testing framework you could do worse than to go through my OSCON tutorial I did last year. My slides are at http://elem.com/~btilly/effective-ab-testing/. Or you can go to http://www.google.co/websiteoptimizer and use theirs. Theirs has some drawbacks over building your own. Their approach can't A/B test email programs, and is somewhat tricky to do with dynamic content. (Not impossible, but you have to rewrite pages in JavaScript.) When I last looked at it they also don't allow you to track multiple statistics in an A/B test, and can't let you analyze combinations of tests after the fact. OTOH theirs is already built and is easy to use. :-)
The first is mitigation of risk. We generally A/B test versions that are similar enough that we'd be comfortable rolling out the winner to the whole site without warning. As a result they tend to be fairly small changes, which goes a long way to solving the inconsistent/volatile issue. Furthermore you associate users with an A/B test slice so while different users get different experiences, a single user's experience should be consistent across their experience with the site. As a result of these two factors the impact of A/B testing is very small.
The second is magnitude of reward. Most websites that do not currently do A/B testing can find site improvements which will give business improvements of 20% or more. That's a pretty clear financial incentive to put up with unhappy users. But in fact many of those improvements made your users happier. So the "construction in progress" is far, far outweighed by the rewards.
This is why companies like Microsoft, Google and Amazon use and strongly advocate A/B testing. Google has created free tools to help people A/B test websites. I know that Microsoft has an internal version of that, and suspect that they intend to open access at some point as well.
If you want to learn more about how to build an A/B testing framework you could do worse than to go through my OSCON tutorial I did last year. My slides are at http://elem.com/~btilly/effective-ab-testing/. Or you can go to http://www.google.co/websiteoptimizer and use theirs. Theirs has some drawbacks over building your own. Their approach can't A/B test email programs, and is somewhat tricky to do with dynamic content. (Not impossible, but you have to rewrite pages in JavaScript.) When I last looked at it they also don't allow you to track multiple statistics in an A/B test, and can't let you analyze combinations of tests after the fact. OTOH theirs is already built and is easy to use. :-)