Greg Schaffer: Hi, I’m Greg Schaefer and welcome to the virtual CISO moment. Today, I’m joined by Zach Lewis. He is a CISO and a CIO, author, keynote speaker, and board level advisor with deep experience leading cybersecurity and technology strategy in higher ed, also a passion of mine, and regulated environments. He currently serves as CIO and CISO at a private health science university where he leads enterprise IT and security operations, supporting thousands of users, manages multimillion dollar budgets, and chairs the Enterprise Risk Management Committee. Maybe even more importantly, he’s also the author of the new book, Locked Up, Cybersecurity Threat Mitigation Lessons from a Real-World Lock-Bit Ransomware Response, a practitioner-focused book based on his firsthand experience of dealing with all that stuff. Really looking forward to talking with you, Zach. Thank you so much for joining us today.
Zach Lewis: Yeah, thanks, Greg. I’m real happy to be here. This should be a good time.
Greg: So I always like to know, this is a pretty tough field to work in. It’s also a very rewarding field. But what made you want to start down the technology IT path?
Zach: Yeah, I mean, I think I got into computers from an early age, and then as I went through schooling, high school, I took some computer repair courses and got to start fiddling with the physical components and troubleshooting computers around the high school, and I don’t know, kind of just really enjoyed it. So as I transitioned into the university and college level courses, IT just made sense, and I wanted to keep really just thinking about fixing computers. I just enjoyed removing viruses, installing software, helping people out. And that transitioned into a career right when I graduated and set some goals along the pathway of becoming a system administrator at first. And that transitioned into different engineering roles and administrator jobs and director roles and managing people and stuff. And finally landed in the CIO seat, which had been a real goal, real passion for a long time. got introduced to security and kind of just got added on the back end, and here we are today.
Greg: Yeah, I love the fact that you mentioned that even early on in your career, you had goals of where you wanted to go, and you adjusted the goals with each step along the way. First, the system admin, then doing some networking stuff, and really, your career mirrors mine an awful lot in a lot of ways. I ended up starting out in the networking side just as a networking administrator and then became more of the director level and then eventually ended up in security. But another thing that we have in common is that we both fell into the higher ed space. So I know from your LinkedIn profile, I think if I’m not, well, I guess, was St. Louis College of Pharmacy your first higher ed?
Zach: That was the first one, yeah.
Greg: So what drew you to higher ed? Or was it kind of like me where it was like, well, that was the job that was offered to me?
Zach: I mean, that’s really it, right? I was working in another place and a recruiting company came out and I had a specialization in a technology called Microsoft’s SCCM at the time. And I guess I was one of a handful that were pretty good at it in the St. Louis area. So there was a real need at one of the universities. A recruiter reached out and was like, hey, we got this job. We think you’d be a great fit for it. I kind of looked at it. It looked like a step up sort of in the career journey. And I jumped on it and I’ve been in higher ed ever since.
Greg: So I’m curious about your experience in higher ed. For me, I guess I have to lead with that in order to kind of explain where I’m going. I always tell people that if you can work in higher ed, That’s one of the best places to learn because the audience, the customers, the employees, the students, all of the people you serve is so eclectic. Did you find that as well, too? And did that serve you beneficially as you went on in your career?
Zach: Sure, I don’t think I’ve ever been in a place where I get to touch more technology and deal with more varying issues just across everything than anywhere else. And even from a security standpoint, when you look at it from that aspect, every type of data you know from finance to regulatory data to instant institutional research and intellectual property um student student records so you have like everything there you have all this technology across the business side the education side the research side you have all the possible types of end users customers students faculty staff alumni just everything you get a little bit of everything here so you get to touch a lot of technology learn a lot of stuff and just try to deal with a lot of different personalities
Greg: Thank you. And that really serves, I think, well as you step up into the executive roles, both CIO and CISO, because really at that point in time, when you get to that level, you have to understand the business. And one of the best ways to understand the business is to understand the technologies that are in place, the information flows. Did you find that you were pretty well-rounded and set up for when you did now start with the C in front of your name?
Zach: Yeah, I think some of the early management roles kind of prepared for that. So I took over management after a couple of promotions and started running a help desk. And there it was more like, managing people, how do they operate, how does the inner team operate, how do I start giving up resources, helping different departments achieve goals and stuff like that. But then when you get to the director level and you have even more teams and there’s more projects and those are touching several other departments, you really have to start understanding how that data flows, how these people work with these people, how these processes work here and the software ties in. really getting a good picture of that and we got to the point where we’re doing a lot of workflows and like setting down with teams and like All right, you have this piece of data. Where does it start from? Where’s it gonna go? Where does it end like mapping that entire journey just so like the team can really understand it you can understand it and then when you’re doing a you know, upgrades or changes, or you’re looking at flipping out software or something like that, you really know how that’s going to affect and how it’s going to affect people.
Greg: And that’s so critical to have that understanding too when things go south, and things sometimes do go south. I know for me, one of my first experiences that relates to what we’re going to talk about with yours is, it was a few years earlier, it was probably like about twenty-three years ago, but we, in my university job, was hit with a really bad virus. I don’t know if it was Melissa or something like that. Campus network struggled to keep up. We did put in play things. I always have to say this whenever I tell this story. We put in some mitigating controls because we understood the network, we understood the business, and we weren’t really knocked down. We were able to keep operations running. But another school in our state that wasn’t the that had it as well, too, got whacked pretty bad. I usually don’t like to say their name, Memphis. So I still like to note that we were able to manage through it. Very hard to manage through these things to now. This turns this turns me now to talking about the book. So why don’t we just start from the beginning? You had an incident and and. What happened?
Zach: Yeah. So in April of twenty twenty three, we had what what became a ransomware incident from the ransomware group LockBit, which was at the time probably the most prolific ransomware group on the planet. They had been through twenty twenty two and twenty twenty three. They had several large, large companies. They take it down from, you know, ministries of defense to law. major airline manufacturers and different things, TSMC, the semiconductor out of Taiwan. So like really big companies that had like tons of revenue, tons of money, you would think really robust security teams still getting cracked and they eventually cracked us. What that started out as what we thought was just a regular IT issue. We had some old equipment. We were looking to sunset and replace. And we were thinking like this stuff’s end of life. It’s starting to fail. We’re trying to lump it along. And what started as like sort of a disaster recovery, try to get everything stood back up and operational again became, oh, hey, we found a ransomware note in the root system. That’s not good. And they’re demanding a space of ransom. They’ve got a bunch of our data, allegedly. And how do we pivot from that? So really moving from a disaster recovery to sort of an incident response. We were fortunate enough to, much like yourself, keep keep open we were we were operational on the fact that like faculty could teach students could learn they can take courses and stuff but like back in we were dealing with a lot of havoc a lot of mayhem trying to patch everything up and get everything rolling again and figure out how these guys got in it and what we were going to do
Greg: But you eventually were able to recover, obviously. And I think that part of what your book touches on, and for those that want to look for it, it is on Amazon. It’s called Locked Up Cybersecurity Threat Mitigation Lessons from a Real World Lock-Bit Ransomware Response. A very long title. I can’t believe you got it all right. I only ever do the main title and never the subtitle. Of course, I was reading here, too. But you eventually moved to the point where you were able to restore operations. But one of the hardest things of dealing with an incident is dealing with the incident when it’s happening, because you get a lot of pressure from all the entities, like, when is this going to be up? What’s going on and all that? I’m very curious how you responded to that and how, like, did you have a lot of pressure, a lot of finger pointing? Or what was the environment like?
Zach: Yeah, so, I mean, some of the big questions we dealt with early on was, I think the first iteration with the board was meeting with them and then wanting to know like, hey, the LockBit group says they have data. How much data do they have? What do they have? What’s in it? And at that point, that early in the conversation, we couldn’t say. We had the claims of how much LockBit had, which the first claim was and then they actually grew to like, a hundred and seventy-five and then to like, three hundred and eighty gigs. Their claims kept growing and we’re just sitting there kind of like, twiddling our thumbs figuring out like, where did this data come from? Like, what could be in it? We had some ideas, we had some guesses, but nothing concrete at the time because we had nothing sort of watching that landscape. And that was pretty new to, you know, I’m not gonna pitch tools here, but like sort of the DSPM space and data tracking classification, like that was all very new technology at that time that we hadn’t adopted yet. So we didn’t have a good, understanding of what was out there and trying to understand that. So that was sort of the main question. As we worked a little bit further down through the incident, the main question came up was, when do we tell people? How do we tell our users? When do we tell our users? Do we go early and let them know that this has happened? just, just to get it out there. So there now, or do we wait a little bit longer and gather some information? Because once you tell your users, then, then it gets out into the, into the, the whole ecosystem and you’re going to have press and all that as well too. Yeah. You’re going to have tons of questions. You’re not going to be able to answer some of those questions yet. Um, So there was a real balancing act and a back and forth around some of the leadership team about when we do that. And we actually decided to wait a little bit longer than coming right out as soon as we knew there was an attack to gather some information. So when there were questions, we could answer those. We wanted to move before Lockpit posted any information, before they did anything on the dark web, because that stuff gets scraped. And as soon as it’s scraped, it hits Twitter. you know, Facebook, it hits all those different sites, you want to be before that, but you want to have a little bit of information. So a real delicate balancing act. And I kind of run through that in the book, because again, there’s, there’s not a lot of media out there, there’s not a lot of information out there about how to really handle that every incident is a little bit different. But, you know, letting people read through that, like, oh, this is something I’m going to have to think about, whether we have regulation that dictates when we need to publish this information or give it out or how we handle it like there’s a lot that goes into that a lot of nuance that’s going to be different for everyone so just here’s an example i hope it helps
Greg: How did your incident response, business continuity, disaster recovery plans, how did they work out for you? Because at least what I found in my career is that you can write the most detailed, best IRPDR, BCPs, and yet every incident… is different and there’s always going to be variables that aren’t necessarily covered. How did your planning help and how did the incident deviate?
Zach: I mean, without getting into too many details, because we want people to read the book. Absolutely. You know, starting with when we thought it was a disaster recovery, we were using a disaster recovery plan. And that goes into how we restore servers and in what instance we would restore them and where things are kept and that worked really well with this is something we’ve been testing for a long time we were pretty pretty old hat at that um and that worked okay when we actually had to pivot into incident response a lot of the information we needed at least to start was there so that we’d run through tabletops with leadership using this incident response plan and inside of it we see things like who do we call first? Here’s the number for the cybersecurity insurance provider. Here’s the information for CISA or the FBI or whoever it is that we need to contact. Here’s the forms we need to fill out. Here’s sort of where critical stuff’s kept and what we need to do. There was a lot of stuff in them that really helped at the very beginning on what we needed to do. It was after you get into like, you know, the first week and after that things start getting off the road, you know, things you don’t plan for. And that’s where we found stuff that, you know, you know we couldn’t access for instance one of our recovery passwords for a backup and it’s like wow that’s that sucks that’s something we didn’t plan for in this incident response and the way that happened was very unique so after the event was over we did something a little different where we actually took a couple of those real high priority passwords that we would need they’re on a physical copy of an incident response plan now they’re locked in a safe in a different part of the building like only a couple people have access to it but we didn’t want to be in a situation again in the future where know if a password manager was down or internet was down or something happened we couldn’t get to those you know critical passwords we needed to recover with we we went down on a physical copy somewhere so that that now lives in an incident response plan uh and there were other things we learned along the way around uh you know out of band communication and setting up a way for the leadership team and our external partners to communicate because once you’re you’re compromised um the threat actors lock bit in this instance might be reading your emails they might be reading your chats you know if they’re watching that they can kind of pivot based on what you’re doing to recover so we need to move our planning outside of that so now in our incident response plan we have things about setting up gmails for people to talk outside of our normal communication strings so we can can kind of plan and organize uh just different little things like that so we we definitely made the incident response plan a little bit more robust after the incident but at least in the early days it did help and having that help you know you’re stressed out right you’re you’re of freaking out this is going on you got to move you feel the pressure and just having something that says like step one do this step two do this um really helps just kind of streamlining and get the ball rolling and that’s what you need to do in the early days
Greg: I need to come back to one thing you said that I think is critically important to point out because I don’t think a week goes by when I’m on LinkedIn and I see some posts where somebody posts a picture of a password book, a regular notebook that you write things in and says, if you’re doing this, you don’t understand security, this and that, you should use a password manager and so on and so forth. But there are valid cases where for writing things down and storing them securely. Like you said, another part of the building in a safe, only a couple of people have access. I don’t know if I have a question associated with this. I think I just, I feel the need to emphasize that there are, that’s a control in itself too. And if you protect it in a certain way, having physical copies of passwords is not only a good idea, it can be essential.
Zach: Yeah. Oh, absolutely. I mean, I’m not worried about a hacker showing up and breaking into a physical safe and stealing an incident response plan. You know, if you have that NSA for limited access and, you know there’s a door in front of that that’s you know badge access only so i’m gonna have a log who’s going in there we got cameras like it’s pretty secure i’m not i’m not overly concerned about that i want to be able to recover and you know save the institution recover the institution if the need arises more so than i care about you know logistics of hey i put a password on a piece of paper and that’s frowned upon And for those who don’t realize that password managers have not been around forever, and we did manage passwords before password managers. When I was at the university, I was talking about the top of the podcast. We had a, in our locked office suite, we had a file cabinet that was also locked that contained a folder that contained the passwords. And so you had to step through. You had to get into the IT office area. Then you had to get into the networking office area. Then you had to get into the file cabinet. And you had to know that the passwords were there, which was not obvious. And you had to know what folder it was. It’s like, I mean, if you got someone playing like Tom Cruise and kind of like coming in and like, you know, getting the passwords on a wire and all that, that’s one thing. But you have to also think about, pragmatic risk here. It’s like, and I think that we fail a lot with controls with that, that we just think that we have to do one thing and that’s it. And we never look at compensating control.
Greg: Yeah. They don’t have to be sexy. It should be kind of boring.
Zach: Yeah. They don’t have to be sexy, right?
Greg: So what made you decide, I’ve written a couple of books before in my past, and it’s not an easy thing to do. What made you sit down and say, you know what? It’s Sunday, and there aren’t any good games on. I think I’m going to start writing a book.
Zach: uh so after the event wrapped up and and you know everything is is good again i was uh i was asked to speak at a conference and i i do panels and stuff every now and then and someone asked me to speak at a conference and uh i needed to come up with you know a keynote a presentation i was like oh yeah i’ll talk about the the incident we just went through that’ll be that’ll be kind of fun um and so i went i presented on that and and when i wrapped up i had so many questions from people in the audience that they eventually had to cut us off get me off the stage so other people can come up and then people were still coming up afterwards just like hey what happened here how about this well this is what we’re doing and i was like okay this is interesting um maybe it’s just this group of people so i did it a couple more times um different places across the country not just here in st louis where i’m at but uh the same thing was happening like the amount of questions i was getting was just crazy and i love taking questions because you get to the heart of what people really care about um and a lot of the same stuff was coming up like how did you pivot here who did you communicate to how do they get in and like just wanted to know like a real desire to hear these stories and uh and then i i partnered that in in contrast i run this conference with infra guard every year in st louis called state of cyber and we bring in you know different agencies fbi or different people and they they talk about their cases and when they come up and they present on cases that they’ve ran and ransomware groups they’ve taken down or whatever. Again, the same sort of thing. Lots and lots of questions. People are just really interested in hearing those stories. I was like, hey, there might be something here. I did a little research. Not a lot of books out there talking about this. You get a couple on some of the big attacks, whether that was like Equifax or, or target from back in the day, like real high level stuff. Um, but, but nothing that kind of goes into the decision making the behind the scenes, everything that’s kind of going on. So I don’t know. I just decided one day I was like, I think I’m going to try to turn this into a book. And, uh, I didn’t know how to start. I didn’t know what to do about it. I reached out to, uh, I did some research and found a guy on LinkedIn who was a writing coach, never met him before. And I reached out and I said, hey, I’m thinking about writing a book. I need help. I don’t even know how to start. So I started talking to him and we formed this partnership and he kind of guided me through how, you know, structures where he’d read some of my work and say, well, you know, you’re very analytical, like this happened, this happened, this happened. You need to maybe put a little bit more language in there about what was going on and They’ll make it a little bit more of a narrative. And so he really helped guide me through that process. But yeah, I just put down everything I could remember or that we did during the event and got it out there.
Greg: storytelling is such a powerful way to get across information i know that i get lost when i read books that are like theoretical you know and and or they or they um they they will tell you things that you should do but they don’t they don’t they don’t accompany a story with it or maybe they’ll make up something that’s kind of a little bit like too cutesy or whatever i think what really resonates though is that when when you’ve had an event where you’ve been in the trenches and you don’t know how this is going to end up and you have all that pressure on top of you, that’s something that everybody who has been in our field long enough can relate with. And I think that’s the hook right there. Because whether it’s ransomware or some other business continuity, disruptive event, you know, a cable cut event, We had one at my university that I’ve talked about before on this podcast where we learned the hard way. You learned the hard way about having redundant plates to where your passwords were because of the password manager. We learned the hard way that even if you have like one small area where like both your primary and redundant fiber links run in the exact same conduit, that’s probably where the conduit is going to get cut. And that’s exactly what happened every time. But that’s, I think, what resonates. So did you feel that you had a natural way for storytelling? And has this helped you in your career as well?
Zach: yeah um i mean with the book uh so really try to write it in like almost a narrative there are large sections of the book that are narrative talking about people here at the university and myself and others and just how we converse how we plan what we did um and telling that story i wanted to write it in a way that wasn’t like overly technical so anyone can pick it up a higher ed later can pick it up and read it and understand it so we’re not going like in the weeds we’re not doing you know analyzing the the malware that was used or anything like that like that that’s not what the book is it’s it’s a story and we’re walking you through the story and that’s really good and i think that’s super useful we hear security leaders talk about that at least the really good ones talk about that i think from a from a board perspective from a leadership perspective and being able to tell the story why are we trying to do a certain, you know, security initiative or a certain project, like what’s the value and not just give them the hard facts, but like, here’s how we’re going to lay this out. And here’s the benefits that are going to come downstream. And these are the business use cases and just bring people along. So they understand the ins and outs, the before and after and how everything’s going to really tie together and benefit them too.
Greg: It runs into the basic truth that we always tell people. If you’re going to talk to business leaders, you’ve got to talk in a business language. Talking about an incident where you’re talking about packet analysis or malware analysis, nobody cares about that outside of us. I don’t even care about it either. To be able to tell it in such a manner that they can relate it to the business and do what they need to do. I understand that I always talk about stress in our field, and we always have a lot. But when you’re under the gun with one of those things, it’s even like ultra, ultra stress. And we need, as practitioners, as leaders in technology and security, we need to be able to step away and decompress in a methodology way that I always say, try to be as positive and healthy about it as possible. And this is coming from someone where I probably would have too many twelve ounce curls to decompress back in the day. And that’s really not the best way to do it, at least not for me. So my question to you is like, what do you what do you do to decompress from all this?
Zach: What do I do? So I live, I’ve talked a little bit about this before, but I live out on a homestead about thirty, forty minutes outside of St. Louis and I raise a bunch of chickens. I do a lot of homesteading, so gardening, growing. I like to build different things, woodworking type stuff, a lot of self-sustainability. We do a lot of catch our own water and solar panels to power different stuff and get pressure and do things and just whatever I, I’m not going to call it like prepper or something like that, but like different things to just prepare for worst case scenarios. And I just, maybe that’s part of the security stuff bleeding through, but I don’t know. I just like doing that. It gets me away. I like to be in nature. I take my, I have three kids. I take a couple of them camping quite often. I do some wilderness survival stuff here and there. And I go out and sometimes I bring tools. Sometimes I don’t and, Build shelters, find water, food, different stuff, and just get away from it all and unplug because you need to do that. It’s just nice being out there without the noise or a ping of a phone and hearing nature, seeing the night sky, crackle of a fire. It’s good.
Greg: Yeah, and that’s one thing I mentioned before we started the recording here is that after I got done with this, this is why I’m wearing the, as we said, the John Fetterman dress up today. Yeah. because i’m right after this i’m going to go out mountain biking because i love to be out in nature but uh um and alone in the woods but i’ll tell you i quite honestly i think i’d be even more stressed though if i had to go like build a shelter and do it without tools and try to make fire by you know just using piece of wood and blowing on sticks and all that i don’t know if i could do that you know you learn how to do it a couple times and then you get out and you just use practice and If I had to do it, like Tom Hanks in Castaway, if I had to do it, I’d figure out a way. Yeah. But I haven’t had to do it yet so far. So tell me about future plans. Is this your first book? Or is this going to be another book coming down the road? Or what you got coming?
Zach: Yeah. that’s a great question this was my first book and i i tell you i wasn’t uh i’m not big on writing i did not care too much for writing but um i don’t know i had fun with it i like i like to talk i like to go out i like to present about this stuff and getting able to pitch the book and tell the story has been a whole lot of fun i’ve met some really amazing and cool people so will there be another book um hopefully you know i have a couple ideas kind of spinning around out there um A nonfiction, maybe a fiction. Play around a little bit with that. Hopefully it won’t be on another ransomware incident.
Greg: Exactly. I was going to say, don’t hope for another incident to be able to talk about. I need to pivot to something else. No foreshadowing, please. But yeah, maybe, maybe so. Well, all righty. Well, I don’t usually record during this time of day, and I know why now, because the sun is always in my face. But that’s also telling me that I need to get out there and do something. Right. So I appreciate your time, Zach. Congratulations again on the book. Best of success with it and appreciate your time here today.
Zach: Yeah. Thanks, Greg. I appreciate it, man. Thank you.
Greg: And everybody, stay secure.