Now we deploy dozens of times a day and I never get called on a Friday night because someone did something stupid.

Edit: I do get called when I did something stupid and it broke the deployment system. But that's gotten much rarer lately.

Behind the scenes it was a decentralized continuous delivery system. Very cool stuff, highly automated. Reduced a lot of work and sped up development cycles from months to minutes. Served quite a large software development organization (1000+). I think we had 1000 servers in 5 datacenters around the globe.

Nowdays I'm working on an open source version of that system, it's still missing few critical features but hopefully I'll get the first release out next spring.

btw, I'm looking for projects/contacts that would be interested in trying out how the system would fit their needs.

Do you have any sort of github/project page?

Cool, sysadmin at a research university sounds like a nice position to be at.

Yes, it's already on github. Unfortunately, since it's missing those critical features it's not easy to see how the whole system is going to work. If you to talk just drop me an email at gmail. mikko.apo is the account.

Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

Or when the bug is only triggered in specific user profiles.

Or when all the devs went on a retreat in the mountains with no cell service.

Or when a dev makes a mistake (which we know never happens to even the best devs)

Or when the only developer that knows which one of the 1000 changes that were pushed could be the one breaking, turned his phone off.

Or when a flaw is discovered in the process for the first time (which we know never happens because everyone's process is perfect, until it isn't)

Or how change management's requirement that the fix be tested and verified by all affected teams might have people staying a few hours after 5pm on a Friday when they just want to get their weekend started.

Or how 10 different people from 10 different teams might need to be called and kept to work until 2am because the change can't be pulled because the database was already modified and the old client data is already expired from cache and a refresh would destroy the frontend servers.

Or another reason.

It drives me nuts when people tell me off for saying 'yeah yeah, no, automating our entire infrastructure of 5 servers isn't really worth it right now', like I'm some unprofessional bozo.

I pretty much have experience with all but one or two of your suggested scenarios, and by now I have no patience for annoying software developers who think that using chef or puppet somehow sufficiently embiggens them to run ops on their own (of course dev ops is almost a political assault on existsing ops guys, not merely a nice new solution to existing problems).

Sigh. This is why I don't work on teams these days (if I can help it).

EDIT: Though I also agree with the sub-parent, that deploying on 5pm is fine in certain teams and certain projects, the most important thing is are the guys pushing to do the deploy going to own the deployment? Are they going to hang around for another 60 minutes to check everything is OK? Are they going to be available at 10pm or on Saturday if something goes wrong and are they going to own it? If the answer is no, then nope, don't do it.

Even if it only happens once in your career, once you've had a dev push out code at 5pm friday night, jet out the door and hit the bar, meanwhile you (the sysadmin/ops on call) get woken up at 1am by site down alert, and have to debug/rollback the changes while the dev who pushed them is unreachable, you learn to really avoid friday evening pushes. Fool me once...

Its sad that most places dont have a proper technical copy (with a full copy of live data) to do full tests on TDD is all very well but you need to test the entire system.

No matter how great your processes and your code are, no test can catch everything that can go wrong in a live environment, and doubly not if your system interfaces with anything third party.

But the real reason we deploy weekday mornings is so everyone is on deck and we can get outside help if required. When I was doing system integration, the problem was never in my code, it was the vendor's. Testing can only get you so close to the real world.

How did you locate it? Measuring ping latency from other machines?

this is why I refuse to do "View Source" on the HealthCare.gov website. I'm afraid of what I might see.

        if ('en' === 'en') {
            $('#desktop-nav .middle').append('<a class="liteac-login topnav myprofile" href="/marketplace/auth/global/en_US/myProfile#landingPage">My Profile</a>');
            $('.mobile-nav-right').append('<a class="mobile-right-bottom liteac-login" href="/marketplace/auth/global/en_US/myProfile#landingPage">My Profile</a>');
        } else {
            $('#desktop-nav .middle').append('<a class="liteac-login topnav myprofile" href="/marketplace/auth/global/es_MX/myProfile#landingPage">My Profile</a>');
            $('.mobile-nav-right').append('<a class="mobile-right-bottom liteac-login" href="/marketplace/auth/global/es_MX/myProfile#landingPage">My Profile</a>');            
        }

because, you know, static HTML and some CRUD/lookup logic behind-the-scenes is just that hard