I had a campus job where I was making a tiny update to the url of a spreadsheet on one page. The sysadmin showed me how to merge my code, test in staging and then promote to live he didnt show me how to rollback or ssh into the host or restart the host…

So I go in and I make my little spreadsheet update to seed the SQL table with my new data and then I add my one line of code to the git repo to point to the new spreadsheet. Perfect I see it’s working fine in dev, I merge into mainline and promote to staging. I sit there and wait for the deployment, perfect all my new data is there the site is working perfectly this is great, my Saturday sure is going well!

I promote from staging to prod, I go to the site, hey this looks great its working but wait the data doesn’t seem to be updated. Wait a minute why is the page getting slower? Oh god the whole website is down… yea that was a great saturday for the sysadmin where he specifically told me not to merge to prod on saturday but I thought it was just a simple config update…

Since no code is completely bug free, we are always testing in prod to some degree, it’s an important lesson to learn and be prepared for.