>Once again, the world of security is complex and surprise features can often be fatal
This. And for me, as an application developer, who is now (willingly) ushered into being a DevOps engineer on the side, it is scary that the tools in the DevOps world are so easily exposing me to security risks, whereas a seemingly much smaller set of best practices on the application side seemed to have been fine when that was all there was to my world.
In my book, the biggest difference between a sysop/devops guy, and an application developer is a broad scope vs a deep scope.
So, as a developer with an application with a simple internal security model, yeah. There's going to be a couple of guidelines, and those will setup a pretty hard surface for that application. For our internal application, well. That security model is hell - I don't understand it and try to avoid it. And we got some picky enterprise customers, that's where the simple guidelines don't work anymore. :)
On the operations side, you can impact each application less, but you have so much more stuff to individually deal with in a quantitative sense. For our SaaS clusters we have to secure 15 - 20 different applications. And for a lot of these applications, we're the one and only line of security - you just have to operate a mysql/postgresql database properly, these systems are pretty secure on their own.
And that's just data security. How many developers do think about backups or disaster recovery for all of this mess. Not saying this is bad - this isn't a job for developers, because it's a lot of work. It's mostly a point against the notion of NoOps.
Just remember that NoOps isn't actually no operations. That's the cargo cult version of NoOps. Instead it's outsourcing operations and relying that the outsource vendors operations team will do a better and more accountable job than any internal team you could build.
So, a database service instead of a locally managed database and a reliance on the service vendor to back things up, keep the code up to date and manage reliability. That's not a bad choice but you still want someone with the judgement to evaluate vendor claims, performance, exceptions and the like.
There's a wider question about who should really be responsbile for all these systems? Would you be happy to have an extra layer of seniority above you that looks after complex systems which you just consume? Would you be happy for them to be paid more than you? Do you think you can actually find these people?
There is an increasing demand for people (and practices) to secure the entire DevOps space. Not only in tools, but throughout the very culture.
It is hard. I recommend looking into the ~newly named concept of SecDevOps - at its core it is about applying the engineering mentality into securing, hardening and enhancing the entire development-operations cycle. Lot of it is obvious. Even more of it is flat out boring. Practically none of it is new.
In resilient systems, the gold standard people seem to aim for is chaos engineering: systems that behave correctly in the face of random failures, with the resilience exercised frequently. Well done security adds an additional aspect - that systems also behave correctly in the face of maliciously introduced inputs and/or failures.
In this case what I would recommend is to put the entire server off the internet, on a separate network segment where it can only talk to a proxy for VNC & web sockets or whatever is needed to make the functionality work.
That way even if the machine does get rooted, it's very unlikely that any damage can be done (it would have to then try and compromise the proxy - all over VNC because you can't even get a reverse shell yet - just to be able to gain unrestricted outbound network access).
This. And for me, as an application developer, who is now (willingly) ushered into being a DevOps engineer on the side, it is scary that the tools in the DevOps world are so easily exposing me to security risks, whereas a seemingly much smaller set of best practices on the application side seemed to have been fine when that was all there was to my world.