Lessons Learned from the LastPass Hack

If you’re not concerned about the LastPass hack, you probably weren’t paying attention. Even if you’re not a LastPass customer (like our host, who uses Dashlane), there are some important things we can learn about what to do with your password manager. From a backup perspective, there’s a huge lesson about what happens when you roll your own backups. Should you be doing that in this security climate? We don’t think so. This and other lessons from the LastPass hack in this exciting episode!

Transcript

[00:00:00] W. Curtis Preston: This week on no hardware required. We’re talking about what we can learn from the last pass. Hack with me, as always, is my co-host, Steven Manley. Thanks for joining. Hi, and welcome to Druva’s no Hardware required. I’m your host, w Curtis Preston, AKA Mr. Backup, and I have with me my co-host and RSV expert Steven Manley.

How’s it going?

[00:00:25] Stephen Manley: Al, hey, I was able to walk up the stairs and, and, and not collapse. So today’s a good day for.

[00:00:31] W. Curtis Preston: So you actually got rsv.

[00:00:33] Stephen Manley: I did, I did. And, and I will tell you, I mean, literally the, the, the stairs thing is not a joke. The, the first time I started feeling a little better, I tried walking up the stair. I thought my, my heart was gonna explode outta my chest. It, it has been. Yeah. Uh, this is, this is, this has been an adventure. Uh, not one of the good ones, but yeah.

[00:00:53] W. Curtis Preston: Not a good adventure, not it’s more like the, um, flying during Christmas of 2022. Adventure

[00:01:01] Stephen Manley: I, I, in fact, in fact, uh, you know, when, when I was in the office yesterday, someone said, well, you know, the upside of you getting so sick is that you weren’t traveling over Christmas. So I got stuck in, you know, blah, blah, blah, spot X, at least you were home in bed. I’m like, yeah, totally incoherent, sweating, uh, sweating like a madman.

But yeah, yeah, basically the same thing.

[00:01:23] W. Curtis Preston: Yeah. You know, the, the, um, you know, there was this minor, uh, blip a couple days. Uh, based on when we’re recording this and with, with the, some FAA software, and, uh, they were, they were, I was watching it on the news and I, I tacked on my end. They’re like, yeah. So there’s been a minor disruption with, uh, you know, air travel and then I added, uh, unfortunately Southwest, for example, uh, it’s going to be delaying flights for the next three weeks,

[00:01:52] Stephen Manley: Exactly. I I thought the other interesting thing in that though was how quickly they said, no, no, no, no, this, this isn’t a cyber attack. You’re like, there is no way you could have diagnosed it and I get it right. You, you don’t want people to panic, but there is no way they could have

[00:02:07] W. Curtis Preston: right? And, uh, and speaking of which, uh, we’re, you know, we’re talking about a cyber attack on this. This was a, um, and it, and it’s, and it’s backup related, which is very interesting. You know, we talk a lot about backups and we talk about cyber attacks. Uh, this one we’re talking about what happened to last pass, which is, Sort of, um, doubly tripoli concerning because this is the, you know, they hold the keys to many kingdoms, right?

Um, it is, I I, I, I don’t have numbers to back this up, but they, I think they’re easily the most popular. Were the most popular password manager, and I’m a huge fan of password managers. Um, and, and so, Yeah. So let’s talk about this. So this was a two phased hack. There was, back in August, there was the hack that basically used, uh, a variety of techniques.

We don’t really know the, the true details, but it sounds like it was some. Some, uh, you know, combination of stolen credentials and, uh, privilege escalation sort of techniques to get into a variety of environments, one of which was Twilio, which interestingly enough, that’s another that didn’t, that didn’t immediately set off bells and whistles for me.

But you, you know why it should, right?

[00:03:39] Stephen Manley: Well, I mean, among other things, I know Twilio connects communication from tons of companies around the world, but I’m guessing there’s something else that triggered

[00:03:49] W. Curtis Preston: they, they bought Authy.

[00:03:53] Stephen Manley: Okay.

[00:03:54] W. Curtis Preston: I’m, I’m Authy customer. , right?

[00:03:57] Stephen Manley: So that definitely sits off alarm bells

[00:03:59] W. Curtis Preston: Yeah. Yeah. So Authy is another one time password. It’s basically a replacement for Google Authenticator. I liked the features of it over Google Authenticator. Uh, so I, I used Authy so that that set off some, some, uh, you know, alarms. But they also got into LastPass and basically over time, what we found was LastPass, uh, said that they had been in the environment for a couple of days.

They hadn. Done anything in terms of damaged anything, but they had exfiltrated some data and they used the term source code. So fast forward to, um, December or November, I, I don’t remember. I think it was December. Anyway, what ended up happening was, This is, again, this is based on the information that we have available.

If I’m incorrect on a few details, I do apologize. But what it appears is that one of the pieces of source code that they obtained was a backup script. So apparently, um, they rolled their own backup system. And, you know, I immediately flash back to, um, you know, 1993 for the record. This month I have now been in the industry for 30 years, so this time th.

[00:05:27] Stephen Manley: How have you not fixed it by

[00:05:29] W. Curtis Preston: I don’t know. Uh, but, but I, I flash back to, back in the day when I was managing backup scripts, right? That was, it was really the only choice, um, that I thought I had because back then there wasn’t really the commercial backup market that there is today. There were a few companies, only one of which is still around, um, and.

Uh, and I, I still remember the first time that I, uh, was able to go and buy b a commercial backup software. Now, I, I’m gonna say that it, it, it’s very hard for me to fathom a company. that. So in the case of LastPass, it’s a 200 million company that isn’t using commercial backup and recovery tools of some sort.

Um, but we’ll, we’ll come back to that, but let’s finish the story. So they apparently, in this homegrown backup script, had hard coded credentials to their cloud provider and. The, then the hackers basically read it cuz it’s a script, right? It’s plain text. Um, and they used those credentials to access the backup environment and exfiltrate their backups.

Okay. Um, which just means download, it’s just a fancy word for download their backups. And now they have an unencrypted. Right, a copy of the backups. Now, I, I need to very quickly say the thing that they backed up was encrypted, right? So the, the database itself, which is the key , the actual, you know, the crown jewels, the usernames and passwords that was encrypted, um, although parts of it weren’t.

So, the URLs that each username and password go, for some reason, that field was not encrypted. I don’t know why that would be the case. But anyway, neither here, they’re there. The point is they use their backups to hack the company, which is, that’s just, uh, that’s, I think that’s, that’s a new one, right?

[00:07:44] Stephen Manley: Well, it, it, it’s, I don’t know if it’s new. I mean, certainly there, there have been times in the past where I believe this happened. If you want to go back, Oh boy, maybe 20 years. There was a Stanford, uh, healthcare breach that was, now this was the old tapes fell off the back of a truck, et cetera, et cetera, et cetera.

But, uh, but, but there are examples in the past, I, I remember where, where people used backups to hack a company. Having said that, yeah, not, uh, not, not a great day for the backup team. Certainly.

[00:08:15] W. Curtis Preston: Yeah, I, I, I guess the, the, the real thing I, I, I want to talk about here is, and, and, and I’ll, I’ll say this. What I, what I do want to give LastPass credit for is an incredible amount of transparency in terms of, I mean, they. You know, they, they owned up to their part in the story. They, they’ve been very transparent as to exactly what happened.

They’ve been very transparent in terms of, if you’re a customer, here’s the scenario under which. Your, your passwords are potentially vulnerable because even though they’re encrypted, there’s some details in there about a combination of the strength of your master password and when a particular password was encrypted, it gets very complicated.

Um, there are a number of people that are calling for people saying, I, I don’t know why you would stay, but that’s a, that’s a different discussion. Um, what I thought was, Important for us to talk about is this idea of why are you, you know, if, if you’re a, if you’re a company and you’re still using scripts, uh, and, and I’ll say it’s even potentially, it’s even easier to do that today than it was 30 years ago.

30 years ago at least, I had to pay for some tape libraries or some tape drives or something. Nowadays the, the cloud bills. Um, you know, you could back up if, if you know what you’re doing. You could back up directly, for example, to Glacier Deep Archive. Uh, you know, and, uh, especially if you’ve never tested a restore for, for Glacier Deep Archive, right?

You could, you could back up to that and, and you could do it relatively quickly and, and potentially save yourself some money. Um, what, what do you think about that?

[00:10:12] Stephen Manley: Well, you know, I wonder, right. You know, like, like you mentioned, this is, this is a company that, that that’s grown. Uh, and, and, and you know, when I, when I worked at a really, really small startup, there were, uh, boy, at least like a dozen times a day where you’d say, you know, this is not the right way to do it, but this is how we’re gonna do it right now.

And then you make the little xxx, go back and fix this. and, and, and I will tell you, you know, I mean there were certain times we got burned. Like, you know, we hard coded again, some cloud permissions into some testing scripts. And uh, shockingly a few months later we found that those had gotten compromised and we had a $30,000 bill because, uh, you know, basically, you know, someone had gotten it and was using it for Bitcoin mining back when.

Like, yeah. Back when that was a thing and uh, and that market hadn’t collapsed as a big Ponzi scheme. But anyway, the, uh, I digress. The, the, I think the interesting thing is, is it’s very easy, I think to just sort of keep saying, yeah, we should probably do something better, but I got 10 other things to do.

And, and it’s stories like this that you have to just remind people. One, it’s a scary world out there and. You know, you may think, oh yeah, we, we’ve got this covered. You know, you get to a certain size and that size isn’t even very big. You don’t have this covered anymore. It’s complicated. And, and as good as your scripts are, as smart as your team is, , you know, you don’t know everything.

And, and I think to me, that was the biggest part is each time we made those decisions, it was that feeling of, yeah, yeah, well this is, this is more than good enough and it’ll be fine and we’ll fix it when, when, when the time comes. And usually the time coming was when we, something bad off happened.

[00:12:00] W. Curtis Preston: Yeah, that there’s a phrase I’ve used a lot in my career and that’s never time to do it. Right. Always time to do it over. Um, It, it reminds me a little bit like I, I, I get, you know, I, I, I built my own company as well. I get, I get bootstrapping, I get, you know, chicken wire and chewing gum. I get it. Um, I just think that maybe there are parts that you just shouldn’t mess around with.

Right. I, I think of. You know, I’m a DIY person, right? Um, I, I do a lot of my own stuff. I do a lot of my own car maintenance. I do a lot of work in the house. I just finished this big flooring project that I’m sitting on.

Happy, uh, with that, that it’s finished. Um, there are things that I don’t do. One of them. is plumbing. And you know, and, and I often, I often mention a lot of people talk about backups being like the plumbing of it, , because it’s the part that nobody wants to do, right? There are reasons for that. It’s not just that plumbing and backups are difficult and complicated.

And every time I did find myself do pl doing plumbing back when I would do it, um, I was always scrunched over underneath the sink. super uncomfortable using tools that I don’t rarely use. I don’t really know what I’m doing. The, but the real reason that I don’t do my own plumbing is that when plumbing goes wrong, um, It’s a mess.

Right? It it does, it does a lot of damage to your house. And, um, and, and that’s very similar here, I think with, with backups there are a lot of parts I think that you can, you can piece together and you can do, you know, never time to do it right. Always time to do it over.

I’m just thinking that given. Companies like Druva, right? We have made backups so easy and so inexpensive compared to what it used to be. It used to be so hard, right? It used to be so difficult and nobody wanted to do it.

[00:14:18] Stephen Manley: Okay. There.

[00:14:19] W. Curtis Preston: uh, we’ve made it so easy and less expensive. It’s still not, you know, cheap. Uh, nothing in it is cheap.

Um, but it’s less expensive and way easier than it used to be. I, I just, I don’t think that backup is the part of your IT infrastructure that you should be messing around with. I, that’s why I like your, your plumbing, um, phrase.

[00:14:45] Stephen Manley: Yeah. And, and, and, and I think I, I think the biggest thing in, in all of this too is that, Again, 10 years ago. Yeah, you, you were still doing a fair amount of DIY if you bought backup, and so I could again, almost understand, especially at a certain size, especially if you were cloud, where you’d say, well, I’m writing a bunch of scripts to make, you know, competitive backup software, product X, Y, or Z kind of fit my environment anyway, so I’ll just do the whole thing myself.

Well, again, like you said, DRE has taken away most of that DIY piece. You just plug it in, it just works. I think the second one, and this ties back to to, to this, this last pass issue is, and they’re coming after ’em now, right? It used to be a decade ago. The things you worried about were things breaking and people making mistakes. You know, your biggest fear now is bad people coming for you through your backups. So it isn’t something where you can, you know, it’d be as if, uh, again, your neighbors who hate you are deliberately trying to sabotage your plumbing. It’s not just you that can mess it up anymore. they’re coming for you. So, so why, why, again, why you have a million other things to do in your job.

You know, it’s easy enough just to, just to offload this job to, to a bunch of people who obsess over it rather than trying to script it yourself.

[00:16:04] W. Curtis Preston: Weirdos like me, that

[00:16:07] Stephen Manley: that’s it. 30 years, right? Yeah.

[00:16:10] W. Curtis Preston: yeah. I, I, I like that idea. Um, you know, that was a good point that you made that maybe, maybe 10, 20 years ago. In fact, I’m thinking back to, man, I did, when you’ve been in this, it just. When I start thinking about when things were, it’s like, that was a really long time ago, but I remember working at a large sub software company, uh, household, household name software company, and we were rolling out, um, a certain very popular backup product.

Uh, started with an “N”, um, and I remember writing 150 custom shell scripts to get it to do what I wanted it to do. So you’re right. 20 years ago, backups were still a d i y if you, uh, if you used a commercial product. But that really isn’t the case anymore. I mean, we, we work really hard with customers to, to do things in a plug and play way, including d uh, a Dr.

Right. Uh, so to, to make it super, super simple so that you don’t have to roll your own so that you don’t end up creating basically just another attack vector. Um, you know, don’t have your backup system be yet another attack vector, let alone have your backup system be worth something when you actually need a backup system.

there’s also that, that problem as.

[00:17:38] Stephen Manley: Yeah.

[00:17:38] W. Curtis Preston: Uh, well anyway, well, well, thanks for, thanks for chatting about this little problem.

[00:17:43] Stephen Manley: Ah, my pleasure. And again, everybody out there look, you know, make sure one, first and foremost, whatever password tools you’re using, keep an eye on those because whether it’s Last pass or it’s somebody else, or you’ve got it written down on a piece of paper somewhere, God help me, or you’re using the same password everywhere.

Right. The, the security threats are so much greater than they used to be. The stuff you could get away with, you can’t get away with anymore, and it hurts and security’s painful, and it costs you time and energy. But boy, you know, like Curtis said, it’s a whole lot better to get it right the first time than it is to redo it later after you’ve been hit.

[00:18:24] W. Curtis Preston: Absolutely. I, yeah, that, that, that, that idea of definitely revisit your password. You know, if you’re a LastPass customer, make sure you go look at that message and look at the stuff. That they’re saying you need to do. Um, I know it caused me to revisit the strength of my master password. For a competitor of Dash Lane and I, I decided to, to increase the, the size and complexity of my master password.

Nothing wrong with updating your, your, uh, practices, uh, when you, when you know, this is one of those, um, you know, moments when you, you know, just like your battle with rsv, you know, you it causes you to re reevaluate things. Uh, well anyway, thanks to everybody for listen. And remember to subscribe so that you never miss an episode.

And as always, here at Druva, there’s no hardware required.