There are many ways to harden a Linux system but I like to approach the problem as though you have already been compromised. There are two main technologies that can be leveraged in this situation but they implement the same general concept which is compartmentalization. The technologies are virtualization and containers. There are two main categories of virtualization, one has an advantage in terms of performance over another because it runs on a different layer (in an abstract sense) than the other. However, the purpose of this post is not to distinguish between the two. The main takeaway with virtualization is that it translates ISA commands from a guest OS to a host OS or from a guest OS to the hypervisor. Containers are an alternative that can be used in conjunction with virtualization. But the main takeaway for container technology is that it shares the host kernel.
I think it's important to stress that security is not about making something unhackable - that's impossible. Security is about putting so many barriers against an attack that the payoff isn't worth the work. That's what we're doing here. We're assuming that we've already been compromised and now we're taking measures to put barriers around the attacker. If you wish to harden each environment then you can implement more traditional measures inside of each container/virtualized environment.
There are distros of Linux that operate based on the concept of compartmentalization, like Qubes OS. As cool as what they are doing is there are a few problems surrounding it. One, you're dealing with a one-size-fits-all situation and if someone has managed to compromise one system running that OS, why couldn't they do it again? Two, they have removed networking from Dom0 (you can think of Dom0 as being the host). I understand that they're doing this to prevent DomU's from communicating but it makes the OS difficult to use. I believe it's sufficient enough to compartmentalize the OS by running virtual machines around the software installed. If anyone is seeking to compromise your system to the point where they have or need resources at their disposal that would allow them to jump out of the VM and into the host without networking enabled I would say you have bigger problems. Additionally, the technologies used with an OS like Qubes are not that unique, it's good that life has come back to the Xen project but any new security vulnerabilities discovered from the Qubes development team are shared with the developers of Xen. So, you can have the benefit of active development on the project while not being locked down to the OS. Besides, Qubes's Dom0 runs with Fedora aka the most annoying OS in the world. Jk, but not really. With what I will be describing below you get the benefit of having a totally custom solution, with unique hardware, software, and configurations. Additionally, I'm going to show you tips and tricks such as spoofing ports to slow down an attacker as they try to compromise your system, automating retaliation against an attacker, logging and refurbishing your OS after an attack, implementing a correct restricted shell and so on. This post may come in several parts because this is such a big topic. So please check back occasionally to see if there are any updates.
There are a few reasons why you may not want to run virtualization. The best reason though is that it consumes too much memory (which I guess attests to the fact that you don't have much reason not to run virtualization) and bogs down the system (maybe a better reason, but processors are sufficiently fast today). If this is the case you may want to consider container technology such as Docker.
One subject which I'm not well versed on is the differentiation between systems that have exclusively hard outer shells and systems which exclusively have hard inner shells. I've never understood it because in my opinion implementing security measures at just one point seems lazy and ineffective. What we will be doing is making a total compromise of a system an extremely obnoxious and difficult task through every layer in the OS.
Lastly, while using Docker and Xen is an option, you could instead use LXD. There is nothing wrong with using this software but I simply don't have experience with it and therefore will not write a post about it. Additionally, an argument could be made that because the technologies are centralized that it could be easier to compromise a system. Another option you have is to mix and match all of these different software packages together but that, honestly, seems like overkill.
Please check back regularly as I will be adding more parts to the tutorial, like here.