Why Back Up the Cloud – An Example

In this episode, we discuss new information about the infamous OVH Cloud fire in 2021. It’s been about a year and a half since it happened, and we now have reports from the firefighters that were on site when it happened. Sadly, we still have very little from OVH – and what they have said is BAD. We discuss their only public comment regarding lost backup data, and it’s not good. This story is a good one that drives home one of our usual points: you must back up the cloud.

Transcript:

On this episode of No Hardware Required, we’ll be talking about a great example of why you need to back up the cloud With me, as always, is my co-host Steven Manley. Thanks for joining.

Hi, and welcome to Druva’s No Hardware required podcast. I’m your host, W. Curtis Preston, AKA Mr. Backup, and I have with me my fire specialist, Steven Manley.

How’s it going, Steven?

It’s going well. I feel like pit bull, right? Cause it ain’t gonna boogie, boogie, boogie until the roof’s on fire. So, you know,

The house is on fire.

uh, so many, this is about as hip as I get. My daughter’s been listening to a lot of pit bull, Mr. Worldwide. Um,

Interesting. Interesting. You have, you have failed as a parent, but, okay. Um,

well before this, this is, this is just the, the after effects of all the failures.

Exactly. Um, I, I wanted to talk about, there’s a story, it’s actually an old story, but I, I found myself reading an article yesterday about the story. So the, the story in question is a. Calling it a fire, I think conflagration, you know, uh, what do you call it, A apocalypse? Uh, yeah. You know, um, you know, this was the OVH data Center fire.

So this is OVH Cloud is now the name of the company,

Okay.

and they are, were my understanding, the largest cloud provider headquartered in Europe.. Right. Um, that, that, that, that last qualifier is really important, right? Obviously AWS is still, you know, king over there as well, but they’re the largest one headquartered over there.

And the most amazing thing about this story is that they filed to go public after this happened. Um, but there was, so they had a data, they had multiple data centers that were built using the, the shipping container model. And, uh, there was a fire in the UPS room and that would be uninterruptible power supply, not where people drop off packages. And, uh, for those of you that haven’t run

FedEx room, the DHL room, uh,

and. The fire got really, really bad to the point that the firefighters went in and, we’ll, we’ll, we’ll talk about that. We’ll talk about that in a minute. But the, but the result was the entire place was. Destroyed and multiple other buildings were also impacted, right?

Uh, there were multiple data centers on the same location and there were outages that happened, uh, in those other data centers, partly because they used the same power, um, and. The real key, how I ended up getting sort of personally involved with this is that one of the listeners of my other podcast, uh, restore It All, feel free to check it out, you know, your favorite podcaster.

Um, that, uh, he reached out to me because he was an OVH customer and he wanted sort of a sanity check they had used. The, the, the backup service, they had actually paid extra because, you know, you know, last episode we had talked about things that, you know, scare us and you had mentioned about people that don’t back up the cloud or think they don’t even think they need to back up a cloud.

And, and so when I first. Saw this story, I was like, well, this is just another story to prove that you should be backing up the cloud. But it’s a little worse than that because there were people who actually agreed with us and they had paid OVH for the backup service and the the, um, The contract specified that the data would be stored in a, in a, I forgot exactly how they said it, but it didn’t say separate location.

It said data will be stored separately than it was some weird phrasing stored, you know, separately from the, the data and it turned out separate, meant over in the corner. Right, like bad data, you’re gonna go in the corner. And when they lost the data center, they lost everything. They lost both the primary and the backup, to which I just put my hands in the air.

Why? Why do I put my hands in the air?

You waved them around like you just don’t care.

There you go. Exactly. . Sorry. That’s what we call a callback. Anyway. Um, so I don’t know what, what do you think when you hear this story? What comes to mind?

I think, I think a couple things. The first one is always, The terrible thing that’s gonna happen is never gonna happen to you until it happens to you and, and again, we, we’ve talked about this, this is true in life. This is true in, in, in the backup world is deep down inside. Most people don’t really think it’s gonna happen to them.

And so, It’s very easy, and I don’t wanna use the word lazy, but I’m gonna use the word lazy. It’s very easy to get lazy. So it’s easy for OVH to say, eh, we’ll keep the backups on premises cuz it’s a lot of work to move it off premises. And we’d have to probably implement something a lot more complicated and do a bunch more work and nothing bad’s gonna happen to us.

And, and then, and then to those customers who say, well, I could dig in and figure out exactly how my data’s getting protected. Or I could just assume it’s gonna be fine because nothing bad’s gonna happen to them. Nothing bad’s gonna happen to me and I’ve got a hundred other things to worry about. And, and there’s that balance, obviously, right?

There’s always the balance between paranoia and, and, and, and good due diligence. But, but you know, if you have that little nagging thing in the back of your brain saying, I wonder how that actually is working. I, or I wonder what would happen if this bad thing happens and you kind of go, yeah, feel a little uncomfortable.

You should go with that feeling and, and, and really follow it through the end.

Yeah, I, I know when, the thing is that if, if somebody asks us, right? Like, what would happen if, right. Um, what would happen if somebody logged into my account, somebody compromised my username and password, right? In the, in the Druva world, your username and password is everything. And, and yes, we have mfa.

What would happen if somebody compromised my account and managed to somehow work around mfa? And then want to do damage, what would happen? We have an answer to that question, right? Our answer to that question got much better, uh, last year, right? And we have some great features there. But we would’ve answered the question regardless, right?

And we would’ve said, uh, don’t do that. Right? don’t, don’t. This is why MFA is really, really important and why perhaps you should be using OAuth MFA and not SMS MFA or email mfa, right? Um, But, but if you asked the question, if you asked the vendor the question, what would happen if this whole data center, what’s my recovery process?

So the, the reason why I ask is, um, OVH’s response. So I was reading this article, like, it, it’s been a year, it’s been a little over a year at this point. The article

Almost a year and

a half now, right?

Yeah. Um, but, but the article was from, from a little while ago and they said it’s been a year and we still don’t know everything.

Right. The, the official response from OVH hasn’t, they’re like, we’re still, we’re still conducting the investigation, Mary. Um, what, what we know comes from reports from the firefighters, right? So we know, for example, that there was, according to the firefighters, they saw no fire suppression systems in the data center itself, which makes me want to go, I’m sorry, what?

Right. It also shows that there was a single power route, uh, to multiple datacenters. when that power route got, uh, compromised, uh, it, it compromised multiple data centers. And, um, but the response, the one response that we do have from OVH is that the customer should have known that the backup was in the same data center. It reminds me, it reminds me, you know, you remember the old phrase, uh, this comes from, you know, very early internet days. I remember the phrase on the internet. No one knows you’re a dog, right? You remember that? So on the internet, you, you, this is, this is the problem with using a cloud service. You don’t know what you don’t know.

You don’t. I mean, these are several bad, or at least three. Bad design decisions that were made. I’m gonna guess all based on cost. Multiple power feeds, cost, money, fire suppression systems, cost money, uh, you know, sending data offsite to a separate location, cost money. These were several decisions made based on cost, and none of them were evident to the average customer.

right.

Right. Um, I, I guess the question that, that I want to ask is, you know, we’re a cloud provider, right? Or we’re, we’re a SaaS provider that, that uses the cloud. How does a customer who’s using, um, you know, let’s say a, let’s say AWS or Azure or whatever, how does a customer assure themselves that the vendor is doing the things that they should be doing.

So, so I, I, as, as someone who spends a lot of time on calls with customers, especially let’s say security, uh, I get a lot of security questions. And again, this isn’t necessarily ransomware recovery. This is how, how does Druva make sure we’re, we’re trusting you to hold this data, right? Just like the O OVH customers trusted, uh, trusted them.

We’re trusting you to hold this data. Give us that confidence. And, and, and I think a lot of what we try to do is walk through our paranoia and we say, these are the things that we’re worried about. Um, and, and the really good customers will often bring ’em, bring, bring some others in. They say, well, how do you deal with this?

Because, and this is, this is one of the things I think is really important because, you know, so many times I get in these conversations with customers and, and you could tell there’s somebody on, on the line that’s actually really experienced in this stuff, whether it’s backup security, how to operate a data center, and they seem a little bit unsure of themselves and.

That’s usually the person to try to, all right, what? What are you worried about? What are you thinking about? What questions do you have? Because the things that you’ve worried about for the last 20 or 30 years in your data center. , you need answers to those questions in aws, answers to those questions in gcp, and you need answers to those questions from Druva because those problems didn’t go away.

Now, they may have become our problem, but you need to get satisfied with those answers. If you’re not, then we need to satisfy you because the only way this is going to work is if you understand at the right level that shared responsibility model. What do we take responsibility for and what are you, what are you responsible for?

And if you aren’t probing on those questions, you’re putting your data at risk.

Yeah, I, I, I would say, you know, it starts with reading your, your EULA, right? Um, you know, your contract, whatever it is that you have looking for weird stuff. Looking for, looking for stuff. A lot of stuff looks the same. And then suddenly you read a weird phrase, right? Like, like this phrase that they had about the backups will be stored.

I forgot. I forgot what it said. It just, I remember the phrase going, that’s weird phrasing. What does that mean? That is not normally, like, it doesn’t say the backups would be stored in a separate facility over, you know, five kilometers away. It doesn’t say anything like that. Um, it it said something different.

So you, you, you know, this is what, by you, you talked about experience. This is where experience comes into play. Read those things. Look for odd phrases.

Yeah.

I was just, I was reviewing, um, One of the other, one of our competitors ransomware guarantee. And there was a, there was an odd phrase in that one too. I was just reading along.

I was like, yeah, it makes sense, makes sense, makes sense. And then it said that in order to qualify for the ransomware guarantee, Two things. One is the ransomware had to trigger their ransomware detection, right? Uh, number one and number two, um, the backups couldn’t have any viruses in it.

Oh my

And I go, okay, so if your, if your ransomware detection didn’t work, I get paid less.

And if, and if you happen to back up the virus to cause the thing, then I, I don’t get, like, I don’t, I didn’t understand that, that, that, that was like a beepy, like

you fail, it’s my fault. Wait, what?

If you fail your, your protection, your ransomware guarantee doesn’t apply. You look for weird things and you ask about them, and then I think you should, you know, like when it comes to data protection, You know, it’s like, where are, where are the copies being stored?

I know AWS probably the best. If you just use a regular AWS snapshot, there’s no guarantee that that snapshot is stored anywhere other than right next to the thing that it’s backing up. Right. Um, probably in the same availability zone.

But it says that right in the description, right?

If you ask that, and then if you want, if you want it somewhere else, and trust me you want it somewhere else, there’s a feature to do that. But it’s on to, it’s up to you to use that feature. We, of course, help you orchestrate that, right? And we now, uh, we now offer an even, I think, even better option, which is to take that local snapshot and have that available for availability purpose.

But then we de-dupe it and transfer it into the Druva cloud so that you’ll be sure that it’s in, you know, a different location, different account, all of those things. But you, it’s, it’s really still on you to, to understand your cloud vendor and to understand what they’re doing, what they’re not doing. Um, and

I, I will say though, I mean, one, one of the things that, that you just pointed out that I think matters to me is, You know, there, there are going to be the two kind of cloud vendors or vendors that you’re working with in the cloud. Those that are going to be straightforward and transparent. AWS has in, in my opinion, I, I like, I like aws, you know, a lot and they’ve never made a secret of.

If you want that kind of protection, here’s this functionality. This is what covers you. They’re not trying to obfuscate it in, in, in language where you feel like they’re trying to trick you. Right. Um, and I do think there’s a distinction there that, that, again, going back to if it smells funny, it probably is, if you’re working with a vendor that isn’t willing to be straightforward, that isn’t willing to, to, to bring answers to your hard questions and, and is constantly tap dancing around them.

There’s a reason for that, and they’re probably not someone you want to be working with. Not everybody in the cloud is the same, and so measure them not just by what their technology is or what their marketing pitch is or what the price is. Also measure them by the comfort you feel because again, you know you pay 10% less, but you lose everything.

I’m not sure that was worth it.

Yeah, and you know, and something that I’ve often said that I continue to stand by tap dancing is the worst of all the dancing

It is, and you could feel it, right? I was, I was on a call with a vendor trying to sell us something and, and we asked. We asked to. I mean, a reasonably challenging question. It wasn’t a gotcha question, but it was important to us. And I got the whole, well, we’ve got lots of customers who are doing things and we’re very, very good at this.

This really scales. We’re, we’re very, very scalable. And, and you start to hear the adverbs pour out and you’re like, you’re lying to me. Like, I can feel right now that you’re lying to me. And, uh, and so you follow up question. All right, I wanna speak to a customer that’s done this. Well, we’ll have to look into that.

Okay. This isn’t a sale. Yeah.

You do that? Yeah. Yeah. So I, I just, you know, trust but verify. Right. You, you, I mean, there is a certain amount of trust you have with any cloud vendor, but I would, and, and then again, I, I know that this, this may ring hollow for some. The best way to make sure that you have a decent backup of your, of your cloud thing, whatever it is, is to not use that vendor to do the backup, right?

Use a third party that will make sure that what happened with OVH doesn’t happen to you. Right. Um, that just, that just killed me. I know there’s a big class action lawsuit. Uh, I, I’ve actually offered my services as an expert witness. Um, nobody’s taken me up on, I’ve done that a few times. Uh, but, but I haven’t, nobody’s taken me up on that.

But maybe one of our 12 listeners will, uh, know somebody who knows somebody. But, um, yeah, it just kills me when I hear something like this when anytime anybody loses data. This one, I think it was a little worse for me because it was. There were customers who had kind of done what I say to do. Right. Which is to pay for the backup.

At least try it a little bit.

yeah, they tried a little bit. They didn’t make the, you know, I think they should pay for a third party backup. Um, but doesn’t have to be us. But I do think they should pay for a third party backup. But, uh, and then they lost data that just that. A little part of me dies when I read that.

And it kills me that we’re, we’re at a year and a half out and we still don’t know, uh, what really what happened with them. But the only thing we do know is that customers lost data. Customers who paid for the backup service, lost data. Um, that’s just, that’s just tough. And,

just not right, man. It’s just not right.

Just not Right. All right.

Well, uh, thanks. Thanks again, Steven.

always hear to darken your mood.

Absolutely. And uh, thanks for listening folks, and remember to subscribe so that you don’t miss an episode. And remember, here at Druva, there’s no hardware required.