I started this three-part blog series with the statement that immutability is not so much a state as it is a spectrum; some data protection systems are more immutable than others. I started the series by describing all of the bad things that can happen to your backups and how they might happen. Now, I’d like to talk about the things that backup products and services do to make them more or less immutable.
Simple login-based authentication
This is not a feature you want your backup product to have if you care about immutability;
I’m frankly amazed it is still used. Simple login-based authentication is where you are authenticated by the backup product simply because you are logged into the operating system as a given user – and it does not require multi-factor authentication. For example, I use ssh to log in to the Linux backup server as Curtis (without using MFA), and Curtis is listed in the backup system as a valid administrator.
It doesn’t matter how secure your UI or API is; if you can log into your backup system as a Linux or Windows user with no additional authentication (e.g. OKTA, Google Authenticator, etc), and then run backup commands that manipulate your backups, that is not a secure situation. All it takes is one stolen password or hacked account and your backup system is toast.
What’s even scarier is that some backup products that operate like this have commands designed to reset your entire backup system and delete all your backups! If a hacker was able to log into your system via a stolen password or another method, and successfully escalate their privileges, they could run such a command and wipe out all your backups. Even if such a command doesn’t exist, commands like newfs, dd, and format can all accomplish the same thing with little to no effort. This is why simple login-based administration (without MFA) must not be allowed on any backup server.
Multi-factor authentication
Multi-factor authentication is perhaps the single most important feature that your backup system should have, and it can and should be built into every part of your system where you can authenticate yourself (e.g. CLI, API, UI). There are too many ways to gain the username and password of a privileged person. System administrators – even well-trained ones – are not impervious to social engineering or other methods of electronic attack. Multi-factor authentication ensures that the hacker will not gain the access they are looking for because they will not have the secondary authentication method. I will also add that hopefully your MFA is based on something other than SMS, as SMS is also very hackable. My personal favorite is to use an app on your phone, such as OKTA, Google Authenticator, or Authy. That requires a hacker to steal your phone and hack into it to be able to successfully pretend to be you.
Role-based administration
This is a great security enhancement that started to appear in backup products in the last 10 or 15 years. Its goal is to limit the blast radius of something bad happening in an account, either because the admin made a mistake, or because the account has been compromised through social engineering or a disgruntled employee. It uses the security best practice of least privilege to give each person the level of access they need to complete their job – and no more. This doesn’t stop ransomware, but it limits its impact on your backup server if it happens to spread to it.
Immutable filesystems
A few vendors are starting to tout that their backups are written to an immutable file system, and some of them say they are therefore the only ones with “true immutability.” (It was actually this claim by a competitor that inspired this whole series, because it’s nonsense.)
Immutable filesystems are a useful feature because they can prevent ransomware from spreading from a protected system to a backup copy. This is a very valuable feature, but as I mentioned in both of my previous articles, ransomware is not the only thing you need to protect from.
For example, an immutable file system will not stop a hacker who has used an operating system exploit to escalate privileges and gain root or administrator access on the backup server. Once those privileges are gained, any “immutable” file system can be easily destroyed with a few keystrokes. In a typical UNIX or Windows computing environment, a single compromised system can also be used to crawl and damage other important systems such as backup systems. Obviously, anyone with physical access to your backup server can also easily override any immutability it offers. Depending on how the immutable filesystem is built, it will also not protect you from a system administrator accidentally or maliciously deleting important backups. Immutable filesystems built on a standard Windows or UNIX file system will also not stop bit rot. Data can be silently corrupted over time in your backup system and you will not know until trying to restore it.
So while immutable filesystems are a great feature, they are far from the silver bullet some vendors are making them out to be. The real vulnerability is the operating system of the backup system itself.
Cloud-based object storage
Sending your backup data to object storage in the cloud offers many advantages. Since I just mentioned bit rot, I will say that object-based storage can easily detect and repair bit rot using one of multiple replicated copies. Object-based storage provides natural protection against ransomware, because ransomware only knows how to write to file system data.
If the backups are written to object storage in the cloud (vs. on-premises object storage), the backups are protected even more, because even if the malware in question did know how to write to object storage, it would most likely not be able to make the leap from data center to cloud.
Object-based storage also offers an additional feature for those that would like to use it, which is the immutability option. Unlike the on-prem varieties mentioned earlier in this blog post, cloud-based storage is truly immutable. Even a fully authenticated high-ranking person at the appropriate customer would not be able to delete those objects. Even more secure would be objects in a SaaS-vendor’s account that no one at the customer has direct access to.
Backup deletion protection (i.e. backup immutability)
Backup products that offer this feature do not allow backups to be prematurely deleted or expired sooner than their defined expiration date – if the customer chooses that option up front. Some call this backup immutability, to distinguish it from the immutable filesystems of other products. This protects against an administrator accidentally or maliciously deleting backups via the backup interface. I really like this feature, as it removes another risk. However, it’s important to point out that if those backups are stored on a Linux or Windows system that the customer administers, those backups could still be attacked via the same methods mentioned in the previous section on immutable filesystems.
Backup recycle bin
Some backup products and services offer a recycle bin for backups, meaning that deleted or expired backups aren’t really expired for some user-configurable amount of time. This can offer additional protection against a rogue administrator that might seek to do the company damage just before he or she walks out the door, or an over-ambitious admin trying to do some house cleaning. This is usually done as a setting in the product where the company agrees not to really delete backups until a set number of days, giving a company with a rogue administrator a little bit of time to notice and correct the damage.
Data protection-as-a-service
A data protection-as-a-service (DPaaS) stores customer backup data in their cloud account behind multiple levels of security, none of which are managed by the customer. In addition, data is separated from metadata so even if anyone was able to bypass all of that security, they would simply see millions of tiny encrypted objects that make no sense. They also wouldn’t be able to either steal them or encrypt them because of how object storage works in the cloud.
Unlike customer-managed backup systems, where a single compromised system could be used to launch attacks against the backup system and thus damage it as well, this is impossible with a data protection-as-a-service offering. There are no shared IT systems or accounts that can be compromised; it is a completely separate system with different technology, different storage, a different security model – different everything.
Only DPaaS can protect against all of the risks mentioned in my previous two posts. A ransomware infection cannot spread to your backups stored in the service provider, and the immutability features built into the product cannot be circumvented via physical access, hacked accounts, privilege escalation, etc.
Returning to the original premise of my first blog post in this series, immutability is a spectrum. Some data protection systems are less immutable than others – even if they have a feature called immutability. Unlike our competitor who claims to be the only one with true immutability, I would never make that claim about Druva. But I will say that I think our backups are far more immutable than any backup system based on Linux or Windows systems managed by any customer – for all the reasons listed above.
Next steps
Learn more about immutability and Druva’s advantages for keeping customer data safe and secure in our new solution brief, and register for our upcoming Cyber Resilience Summit to explore how best to defend your data in the age of ransomware.