Two experts who have built hundreds of on-premises backup systems talk about how difficult it is to build them correctly – especially the first time out. W. Curtis Preston (Mr. Backup) and Stephen Manley (Druva CTO) have helped hundreds of customers navigate these challenges and can speak to them firsthand. The two biggest challenges are capacity planning and cyber security. They say it is impossible to correctly provision an on-premises backup system before explaining why. Finally, they discuss how important cyber security is in the current world where the bad actors are coming directly for your backups. They do all this while being entertaining as well. Enjoy!
Podcasts
Challenges of an on-premises backup system
[00:00:00] W. Curtis Preston: This week on No Hardware Required, we’ll be talking about challenges of on premises data protection. With me as always is my co-host Stephen Manley. Thanks for joining.
Hi and welcome to Druva’s no Hardware Required podcast. I’m your host W. Curtis Preston, AKA Mr. Backup, and I have with me, our CTO, Stephen Manley. How’s it going, Stephen?
[00:00:26] Stephen Manley: I’m feeling a little competitive today. Curtis, I’m ready to run through a wall. I’m ready to, just get out on the court and start a fight. Let’s do this. let’s destroy something.
[00:00:36] W. Curtis Preston: When you have such a different architecture that we have -that being a SaaS-based product, it is still new enough that there are still people that say, why would I want to do this in a SaaS way?
And so I, I usually turn that on its head. And I say if you could do all of this and get all of the functionality that you need, and you could meet your recovery time, objective, and recovery point objective, why wouldn’t you wanna do it this way?
[00:01:07] Stephen Manley: Yeah. it reminds me of the early days of VMware, where we used to have those conversations of should I virtualize this? And then there was a day and I don’t remember exactly what day it was, but it was probably a Tuesday and all of a sudden, every customer I started meeting said, no.
Now we do, explain to me why you aren’t virtualizing this, and that was the moment where it just felt like the skies opened up and the light shown down and everyone went, that’s a really good point. and I think we’re coming closer to that day.
[00:01:38] W. Curtis Preston: Yeah, I liken it to the virtualization journey as well, because if you’ve never tried virtualization, then you don’t really understand what’s the big deal. And then someone sits you down. And you show them vMotion and your mind is blown and you’re like, how did you do that? I did it because we virtualized, and because we virtualized, we can do these magical things.
And once you’ve seen that, you don’t want to ever go back to physical servers again, and you find yourself in a situation of saying, why shouldn’t we virtualize this particular application? And for a while, there were some reasons for some, if you really wanted that bare-metal performance, then it didn’t make sense to virtualize it.
But even then, they dealt with that issue. And so then basically people were virtualizing everything. Again, you and I have been in the backup space for a while. We have seen the pain from two different angles, but we’ve seen the pain
[00:02:40] Stephen Manley: yeah.
[00:02:41] W. Curtis Preston: of what it’s like to try to build and maintain that last part being really important an on-premises backup system, which is the alternative to what we do.
So either you can have us do it, or you can, buy a box or buy some software and put it in your own box and do this yourself. So I thought we could talk about what it’s like to build an on-premises backup system. And so let’s talk about the first aspect, which is the first thing we have to do is we have to figure out what it is that we’re backing up.
[00:03:17] Stephen Manley: Right. Because to me that ties in with, you know, everyone has their least favorite part of, of running a traditional backup system. For me, it was always capacity planning because it is guaranteed 100% that you will be wrong. And even if you’re right today, by tomorrow, you’ll be wrong. But we always have to make an attempt at, or you, and so you would go around and you would say, okay, let me look at my environment.
All right. what’s the data I’m backing up. How much does that add up to? Okay. Well now I have to think about. how many, fulls, incrementals, what’s my change rate. What’s the growth rate versus the change rate. and then well we’re gonna throw dedupe and compression, in sure.
Why not? Because we could just have a guess on top of a total hyperbole, and our answer is we need 87 boxes and you’re like, It, there are so many variables in figuring out what you need to back up and how much of it there’s going to be that I, I don’t know.
I mean, just, just from a pure capacity standpoint, I’ve never met a customer who went like our error bar is like four X and you’re like, that’s probably not even a big enough error bar.
[00:04:26] W. Curtis Preston: Yeah, I’ve helped many customers basically redesign or redeploy a backup system and move from system A to system B. And question number one was how big is a single full backup. No one ever knew the answer to that question. And now you a, you add to that, the challenge of SaaS providers.
So if you’re going to back up SaaS providers, if you’re gonna back up G Drive if you’re gonna back up your, SharePoint or your online Exchange environment, how big is that from a gigabyte standpoint? Do you have any idea? and then you also added on to that, the change rate, how much changes every day, because if we’re gonna do this to an on-premises system, we’ve gotta figure out how much bandwidth we need from there to here.
[00:05:17] Stephen Manley: right.
[00:05:17] W. Curtis Preston: Because we have to allocate over here, the incoming bandwidth that we’re bringing the cloud back down, but we have to figure out all of that before we buy anything.
And you’re right. You’re never going to get it right. And so the, have two dangers, you either undersize or you oversize, if you undersize, you had this big meeting with your company and you asked for $1 million and, and then you got it wrong and you’re back with them six months later, and they’re mad at you, right? And, or maybe even sooner than six months later, I remember a customer of a dedupe product that is still on the market, but when it first went on the market, it had this really bad problem that it was really good at backing up, but couldn’t restore.
So, it had a 400 megabyte, a second throughput incoming, but the maximum aggregate output of it was a 10th of that. And he discovered this after he’d bought the system and he couldn’t tell his boss because he’d messed up. So you either undersize. Which is a disaster for everyone.
Perhaps the backups don’t even work. Perhaps you run out of capacity too soon, perhaps you don’t have enough throughput. And if any of those happens, you’re back in the boardroom explaining why you need a whole bunch more money. When you said you only needed X, apparently you needed two or three X.
So that’s one, the alternative is to oversize. And that’s where most people go. I don’t wanna do the first thing you said. And so I’m gonna buy way more than I think I need. And so the dream is that you get that right, that you way oversize, which is a horrible thing to do.
[00:07:09] Stephen Manley: All right.
[00:07:09] W. Curtis Preston: And from a, because basically what it means is the best you can hope for is that you wasted a bunch of money.
[00:07:16] Stephen Manley: Yep. Yeah, no, I, there were many customers of my previous company. Again, love the product, but you would look, and I remember once there was a customer call and we’re a little bit worried that we’re pushing the system too hard. And so we pulled up the, the auto support, the phone home, logs, like the maximum CPU you, your system has hit ever is 10%, but if you wanna buy more, you bet.
[00:07:42] W. Curtis Preston: Absolutely. Absolutely. Yeah. and the thing is, why do we bring this up? Both of the thing is that our customers don’t have either of those concerns. yes, we want you to size your environment. Your, if you’re gonna back up Microsoft 365, what do we need to know? We need to know how many users we’re backing up.
Yes. We want to size your data center, because we do want to bill you properly, but it’s not the same in that. If you get the size wrong. if you drastically undersize yes, you will need to renew early, but you don’t ever end up paying for something that you end up throwing away.
and it’s not, you don’t, undersize it in, it’s not possible to undersize it in the same way. What you could just undersize it from a long term capacity standpoint, you will run out of capacity over time, sooner, and then end up, you just need to buy more capacity, but you don’t have this possibility that you create a system that it’s just physically impossible of doing the job on day one. That’s not possible with our system. And the other thing is it’s not possible to over configure it because there’s really no need to do that. If you did over configure it, basically you have credits that roll into the next year. It’s not a big deal.
[00:08:55] Stephen Manley: That’s one of the things I tell people that I think is the difference between SaaS and even sort of, well, no, I’m gonna run a virtual appliance in the cloud. With a virtual or physical appliance, when you hit the limit, you have to get another one. There, there are finite sizes that they can grow to.
that’s just the, that’s the nature of having a box, either physical or virtual is there is a box has a limit. You can only get so far. and your growth then becomes a step function, which can be very tricky because you can very quickly go from under provision to over provision. And you’ve got the worst of both worlds in, in the same timeframe.
when you look at SaaS and you look at something that’s truly cloud native, again, it’s a smooth line and so there’s no need to over provision. Because, know, there’s not that step function penalty. and if you under provision, yeah. then you’ll size it up. You’ll true up. But again it’s that smooth line where you can draw that clear spot between, yeah, we have more data, so it, we need a little bit more backup space.
so the other, the other thing that you hit on that I think is important is. In a traditional box world, your performance is tied to the type or size of the box, especially when it comes to recovery. the nice thing in a SaaS world is, you could be our smallest customer. You could be our biggest customer when it comes time to restore your data, we unleash the full power of the cloud for you, no matter what size you are. And, and so there’s not some sort of thing of, now you bought the smaller box, so you only get half the performance. It’s you get it all. No matter how much you’ve got, you know, that’s sort of a nice to me, a nice aspect. You never have to worry about that performance part.
[00:10:28] W. Curtis Preston: Yeah, that’s huge. The other thing I want to talk about in terms of the challenges of on premises data protection. So we talked about capacity management, the second, and it’s moving to the front challenge of this is the cyber security aspects that, when you and I were worried about backups, on behalf of a customer, no one was attacking the actual backup system, but now they are. Yes. I’m more worried about customers that are using Windows-based backup servers, because Windows is such a hot target from a ransomware perspective. They come in with their infected Windows-based laptop.
They connect to the VPN or they come into the, into the office. Do you remember when people used to do that? And then it spreads to the data center. And then your backup system is running the same OS. Yes, I’m. I’m more worried about the window systems, but even the Linux systems, there was a, we mentioned it on last week’s episode that there was there’s a new exploit that came out this week that allows a regular user to become a super user, which means they can get around every single one of the security features that your backup vendor sold you on, they might have told you that they have an immutable file system or that they have a, an append only file system.
Both of the, those are completely defeatable with the newfs command as root. You couple that with the fact that the typical backup system is being administered by a junior person who perhaps isn’t the most, sophisticated when it comes to cybersecurity, needs.
So that to me is now becoming the number one concern with managing an on-premises backup system.
[00:12:05] Stephen Manley: Yeah. one, one of the things I try to just simple question I ask customers is how many different things does it take to put together to make your backup system? Because the reality is almost all bugs. When you’re developing software, when you’re deploying a system, in general, you tend to get good components, but the component itself is rarely the problem.
It’s where the components interact. That’s where things tend to break that’s it’s that murky middle of who owns it, or are we integrating properly or our mental models the same? And the average backup person’s environment. You have a server running Linux or Windows, and you have backup software and you have, a network configuration that you’ve gotta manage.
And you’ve got. you’ve got different components that you’re putting together to manage all that. That’s where this is going to go wrong. And like you said, if it’s a junior person who probably hasn’t seen as much of the world, they may not even know exactly what things to be worried about.
so I always look at it as, the more you can turn that into there’s just one thing you’re a lot less likely to have a problem when it’s just one thing than if it’s just five or six things.
and this is, again, our customers don’t have this concern because the, we manage the security infrastructure of your backup infrastructure, because that, is our sole reason for being, and you get to move all of your backups behind many, layers of security behind a completely different authentication system, a completely different infrastructure a completely different storage system. We’re not storing backups on files. We’re storing them in object. the object system is, S3 in this case is, configured so that only our application can read and write to it. You’ve got all of the encryption there that is, is entirely managed by your keys, not our keys.
[00:13:54] W. Curtis Preston: We have zero ability to see your backups and by the way, not all of our competitors. That or I’ll, put that in the affirmative some of our competitors that have SaaS based products still maintain root on the servers that, which means they have the ability to see your backups. And I would just really think about that and ask about that.
Is there any way that someone at your environment can either see or do damage to my backups and our customers just simply don’t have to worry about that.
[00:14:25] Stephen Manley: right. and the way I always look at it is, look, we do a lot of complicated jobs. So that you have that just one thing, which is my data goes to Druva. Cool. I don’t have to worry about that. I’m not stitching a bunch of things together, So we do the job for you because let’s face it, there’s 27 other things you need to be doing right now. Let us take it. It’ll be okay.
[00:14:50] W. Curtis Preston: The way I’ve been saying it lately has been, get out of the backup business, go straight to restores. And that’s basically what we allow our customers to do. it’s hard to cover this in, 20 minutes, but we did our best, so so thanks. Thanks for giving it a shot.
[00:15:04] Stephen Manley: It’s always fun. and again, the thing I wanna tell everybody out there is, joking aside, really think very hard about where the exposures in your environment come from and how you can close them because there will always be risk. There will always be danger of things going wrong. If you can shrink that window, you’re gonna make your business a better place to be.
[00:15:25] W. Curtis Preston: Absolutely. Well, thanks for listening to this episode and don’t forget to subscribe so that you don’t miss an episode. And remember here at Druva, there’s No Hardware Required.