Compliance Unfiltered is TCT’s tell-it-like-it is podcast, dedicated to making compliance suck less. It’s a fresh, raw, uncut alternative for anyone who needs honest, reliable, compliance expertise with a sprinkling of personality.
Show Notes: The Through-lines of Business Continuity
On this week’s episode, the Compliance Unfiltered Duo take an in-depth look at Central Logging. It may seem pretty straightforward, but many organizations struggle with a complete understanding of the topic.
Every company should care about Central Logging, but why? What type of things should you be logging? Whether you have questions about log locations, or devices and software themselves, Adam has the inside knowledge to how you ensure your logging practices are keeping your organizations safe.
Highlights include:
- Introduction to central logging and what you need
- What you should be logging
- A true crime story of central logging
- Pro tips you can bank on
These topics and more on this week’s episode of Compliance Unfiltered!
Follow Compliance Unfiltered on Twitter and Instagram at @compliancesucks
Read The Transcript
So let’s face it, managing compliance sucks. It’s complicated. It’s so hard to keep organized and it requires a ton of expertise in order to survive the entire process. Welcome to Compliance Unfiltered, a podcast dedicated to making compliance suck less. Now here’s your host, Todd Coshow with Adam Goslin.
Well, welcome in to another edition of compliance unfiltered. I’m Todd Coshow alongside the one and only legendary compliance master himself. And he hates it when I do this Adam Goslin, Adam, how the heck are you?
Uhhh, I’m fine, Todd, how are you?
I’m great, man. I’m great. It’s been a while since I’ve big up to you. So I figured we might as well do it today. Well, Adam, today, we’re talking about something that seems well, seems pretty obvious, man. But can you talk me through a little bit more about central logging?
Surely. So it’s basically a central location where you would store the logs of your environment. You think about a atypical organization. It really depends on what it is that you’re trying to protect. But it’s different for every single organization. Some organizations are running purely in the cloud. Some of them have corporate offices with local servers and hosting. Some of them have a corporate office, a series of sales offices. They have a co-located facility where their equipment is and stuff in the cloud. So it’s kind of all over the board. But basically, central logging gives you one spot to go pull out all the logs for your environment into. It’s a location which really is separate from the devices and the software that you’re leveraging that are generating the logs. The central logging is kind of a separate location that you would store those logs into. And there’s a bunch of things about central logging and its setup. But the most important part is that once the logs go from your environment and head over to central logging, that they’re unalterable when they get into there so that you’ve got a rock solid repository of the logs that you’ve got within that environment.
Okay, well, I know it’s required for many compliance standards, but like why should every company care about central logging?
Well, I’ll walk you through kind of a scenario or an example as a kind of a use case, right? Let’s pretend you don’t have, you don’t have your central logging and you’re going in, you’re looking at some application, you see on the application server, somebody logged into there and then at the same time, we’re seeing some strange behavior off of that box that’s targeted at the database server. Well, now I gotta go get off the app server, go over the database server. Now I start looking at it’s logs, I look at the operating system, I’m looking at the database logs, those are kind of two more lookups that I’ve gotta go in and do. Then I see some traffic coming in from some other box on the network toward the database server and okay, now I gotta go over and look at that box and somebody on that box had authenticated to the domain. So now I’ve gotta go in and I’ve gotta look at the domain server, I gotta scour my active directory and blah. And at that point in the game, now that I’m trying to, I’m kind of grabbing these tidbits, I’m going to all these separate devices, I’m going in to go take a look at those logs, et cetera, but now if I wanna, okay, oh crap, what time was it that such and such happened over on the blah, blah, blah box? Now I gotta go in, I gotta go back to that server, you know, double check, it’s an entirely manual process. And so, you know, from that perspective, it’s, you know, it’s really kind of a nightmare to be able to piece all of this together. So, you know, your central log, your central logging, you know, location with it effectively inheriting the amalgamation of all of these logs, you know, yes, I can go in there, I can look and I can see, you know, which box did what and filter the results and all sorts of stuff. But all of the logs from all of those systems now have been, you know, kind of merged into this one location. So now I can really kind of see the full picture, you know, what’s really going on. You know, it’s a lot easier when you’re looking at the kind of mass of logs across all of those various devices. It’s a lot easier to start seeing patterns and, you know, and whatnot doing searches, not having to jump from here to there, you know, there, there, there, you know, et cetera, you know, another good example where, you know, kind of your central logging can kind of come into play to give you some additional information and insight. You imagine, you know, you’ve got one of the machines in your environment, all of a sudden, whatever, the power, you know, it’s inaccessible for some reason, right? The, yeah, motion alerts. Yeah, got to love it. So one of the box goes offline. And, you know, if under normal circumstances you didn’t have the central logging, you know, you’d just be dead in the water trying to get it back together. And I mean, I hope this thing comes back to life, you know, was it a disc on there?
Was it the motherboard, power drop, it went bad? Who knows? But, you know, for some reason this thing just isn’t, you know, isn’t functioning. And, you know, at least if you’ve got, you know, the logs and here’s kind of the important part too that people need to kind of keep in mind. You know, there’s a couple of different, you know, kind of theories, you know, theories, methods, et cetera for porting the logs from your existing machines to central logging. And it really depends on the environment. It depends on the devices that are in play. But, you know, generally speaking, your best bet is effectively streaming those logs from the devices to central logging. In some cases, there’ll be circumstances where, there’ll be circumstances where you effectively are going through and, you know, setting it up so that, oh, I’m going to push logs every, you know, X minutes or X hours type of thing. Well, I mean, if you’ve got that scenario, that’s not an optimal scenario because, you know, if something happened and your pushes for central logging are only happening every 15 minutes, let’s say, well, if something happened in the last five minutes, then you could, you know, you very well could just miss the window and those logs are only sitting on the box that you’re, you know, that you’re talking about. So, you know, setting up those logs to kind of stream is your best case scenario. Because that way, when that box goes offline, now, even though I can’t get to it, well, now I can go over to my central logging and I can say, well, what are the last things that this thing, you know, that this thing did? I mean, you can almost look at it as the same, kind of the same case for the black box on an airplane, right? If something goes sideways with, you know, with a flight and they can go in and find the, you know, the black box recorder, well, now they can start to glean glints of, you know, what exactly occurred on this, you know, on this airplane, you know, and whatnot. Otherwise, if they didn’t have that black box capability, then you’re guessing, you know, throwing darts at it but what could possibly happen. So yeah, it’s super, super helpful. Having central logging is kind of one of the tools, tools at your fingertips. Absolutely.
Well, what type of things should be logging?
Well, certainly for any of the new kind of machines within your environment, servers and, you know, servers and station, things like that. You want to be gathering up the logs from the operating systems of any of those devices. So whether it’s Linux or Windows, whatever, you know, grab the OS logs, get them fired off to central logging, critical applications. So we talked earlier, we were talking about the an application server and a database server and Active Directory as an example. You want to make sure that you’ve got all of your critical logs from any critical applications on those devices as well. You know, don’t forget about, you know, your firewall, your switches, all the infrastructure devices that, you know, that you’ve got. I mean, literally, you know, backup systems that are, you know, creating and generating backups if they’re if they’re hardware based, see if you get the logs out of those. There’s devices depending on, you know, the circumstances of the target organization. So things like, you know, IP video cameras and, you know, door badging systems, visitor systems for, you know, so those things that kind of center around physical security and certainly being able to capture, you know, kind of capture the logs off of them, you know, also helps. You know, if you look at it from this perspective, if you have anything within the environment that draws an IP, then now you’ve got the ability to, you know, now you’ve got that decision point. Is this something that I need to go push to my central logging? Is it going to be helpful, et cetera? You know, some of the things that people really don’t think about is, you know, so imagine you’re in a kind of you’re in a kind of a colo environment and, you know, all of a sudden one of your machines. Oh, boy, you go. We’re going to need to cure that. So, you know, one of the, you know, one of the various machines within the environment, all of a sudden goes offline. And now I’ve got, you know, now I’ve got if I’ve got the badge readers as an example flowing into my central logging. Well, now I can tell who gained physical access to that to that device. They may not have come in, you know, logically, you know, into it, come in through the Internet or through some other system, et cetera. They may have literally opened the door to the, you know, open the door to the cage, walked over and plugged in to the, you know, to the device. So that’ll certainly give you a lot of good insight that’s helpful information.
Sure. Now, any cool stories about why having central logging is so important?
I’ll tell you a story. This was in the very, very early days. I think I may have brought this up on one of the other podcasts. But this is specifically related to the logging aspects. So in my very, actually, this was before I switched and headed away from managing IT, just managing IT over into the, sometimes I say over to the dark side, but over and into the world of security and compliance exclusively, is that we had a bunch of devices. It was locally stored at headquarters. And all of a sudden, one of my guys says, hey, you need to basically get into the server room right now. And so I go walking in there, and we’ve got a monitor sitting up in the server room that we would connect to the servers on. And myself and the other individual are standing there. We’re staring at this monitor. Nobody’s operating the keyboard, but there’s stuff moving. There’s stuff happening. There’s things that are going on. We’re going to have to find a dog treat or something.
Pile motion alarm is going off over here. It’s business. Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha
Anyway, so we’re staring at this thing, and the monitor is busy going nuts, right? There’s stuff going on on this machine. The kid, there’s nobody typing anything into the keyboard, but there’s things happening on this device. And so we really quickly went around, confirmed the only other couple of people that may have been remoted into that particular machine. Are you remoted into the machine? No, and effectively we were literally watching an attack on the organization unfolding before our very eyes. And so we ended up literally gawking the network cable out of the back of this device to drop connectivity, and then started embarking on some investigation. And where this plays into the central logging notion is at the time, we didn’t have central logging, you know? And so what we ended up finding out by looking at what logs we could, you know, we ended up discovering that somebody had attacked the organization. I think they’d gotten in through, you know, through kind of a hack on one of the customer service people, you know, type of thing. Ended up finding security vulnerabilities within the systems, got all the way over to where they basically had command control over, you know, over the majority of the network at that stage of the game. And they were actively running scripts that were attempting to gather, gather, parse and exfiltrate sensitive data from the organization. And the interesting part is that as we were doing the looking at this script that was busy running, what we found out is that the script actually aired out. It was the only thing that saved, you know, that saved that particular company is that the bad guy’s script puked right in the middle. It came across something it wasn’t expecting and died. And so it basically, the machine was left in a state where we were able to see some of the stuff that was going on. And, you know, as we started to look through this script that these guys had, they were, you know, they were basically, they had commands that were gonna go through, look for certain files on the device. And at the, one of the last steps of their script was to go back and effectively clear the local logs off of that system. So because of the fact that the script died, it didn’t get the opportunity to automatically clear the logs off of that system. Had they, had this script been successful and nobody had been standing there, then this organization quite literally would have had zero idea. They’d even had a problem. And certainly, you know, nobody, nobody would have tripped across this at all. The only sign that there would have been something happening that would have been left would have been if we’d gone in to look at the local logs on any of the systems that were targeted. And if we saw some, you know, 15, 20 minute gap in the logs on that device is the only way that we would have been able to piece this together.
So, you know, it’s wild. The, you know, in that particular attack, even, and this was, this was probably 15 plus, you know, 15 plus years ago at this point in the game. But this particular attack was, it was actually very eye-opening for me because, you know, at the time there wasn’t a lot in the way of, you know, kind of real world stories about the hackers and what they were up to and how they were doing what they were doing. But when we looked through those logs, it was unbelievable. They, you know, kind of were doing poking and prodding, you know, at the, at the organization just generally. And they had, you know, one IP would come in from whatever. And I don’t remember what countries it was, but one IP would come in from France and it would basically say, is the host alive? Okay, then it would go away 30 seconds later when it, because it had found a live host, now four more different, different IPs, not the same one that was coming in a minute ago, four brand new IPs come in and they’re from Germany, South America, somewhere in Asia and Canada, you know, would come in and they were running a little bit more directed attacks on the organization, trying to determine what ports were open on those boxes and AKA what was running on it, et cetera. And as those scans would find things, discover things, then it would hand off to yet another set of IPs. It was wild, you know, sitting down and seeing, you know, seeing all of this, you know, kind of having unfolded, but yeah, the clearing of the log that was really a, you know, kind of an eye-opening experience because we literally would have had absolutely no idea that this had even had, we probably never would have discovered it even happened, which is kind of the scariest part about it.
That’s wild stuff. Any remaining pro tips regarding the logging for the folks out there?
Well, a couple. So first off is that if you think about it, I’m about to talk about network time protocol or NTP being set up across your environment. NTP, your network time protocol, it’s a service on the various devices and systems that you have that effectively ensures that you have a common time set on all of your equipment. So it makes sure that all of those kind of have the same time because if you didn’t have that, and actually on some earlier days, people trying to go in and set up their central logging, that was typically where we would go in and discover this that whatever, they’ve got 20 different boxes that they’re pushing logs into central logging. Well, meanwhile, one has a time zone, one’s running in absolute Greenwich Mean Time, another one’s in Eastern, another one’s in Pacific, another one’s in Mountain, another one has time over in Asia or Australia type of thing. And it was, if you think about it, you go put all of those logs with whatever, let’s say it’s eight different time zones. Well now, even though, yes, you’ve got kind of the timestamps for when they came into the central logging system, you’re still, you’re trying to look at the line items and see what happened and when, and when did it occur and blah. If the time zones are basically all over the map, then trying to kind of piece everything together, it’s like trying to get all your spaghetti to kind of line up in a straight fashion type of thing, it’s damn near impossible. So running your NTP, making sure you’ve got the same time so that as you’re going through and doing your stuff, that’s seriously important. Another thing for folks to consider is, I’m just gonna call it reporting, whether it’s reporting or running queries or trying to get access to information. One of the things that folks wanna kind of think through as they’re walking into this arena is, what types of reporting is it that they’re gonna be looking for? What types of information and data are they gonna wanna know? Have that list kind of thought out in advance, run scenarios in your head with your team of different things that they may wanna find out and discover, be able to parse or filter on certain parameters, et cetera. And then as you go into your central logging, can this target system do all of those things that you needed to do? The other side of that is whether or not the things that you’re about to go attempt to run, you can even do in a reasonable timeframe. I had one client that their logging system, I’m not joking, took hours, hours to go ahead and generate like a report. I wanna know, I want an export of all the logs that happen between this time and that time. You could punch the button, go walk away, go grab lunch, you can come back, you’re still staring at it, go to a couple meetings, whatever. And you think about it, if you are, we talked about that kind of earlier scenario of you’ve got an emergency, right? Somebody’s in and on the systems and we need answers right fricking now, type of thing. Well, I don’t wanna be sitting around for multiple hours waiting for the export to come out. I need to be able to see this stuff.
So that’s not really gonna go ahead and work. Another recommendation is, you know, things happen within environments, right? As you’re going through day by day, you know, this device needed to have an operating system overhaul. This device was getting all clogged up. So we decided to, you know, basically bring it back down to bare metal and then go ahead and rebuild the whole damn thing. We decided to switch from one platform to another, you know, whatever. There’s all sorts of reasons why, you know, that an organization is going to be going in and basically tinkering with their stuff. Well, you know, I mean, human beings make mistakes and all that fun stuff, depending on how you’re handling your configurations of systems and things on those lines. You know, it may be that yesterday, this thing was logging to central logging, but now because of whatever was happening internally, now it does not. So, you know, one of the things that I recommend to folks is go out and do a periodic evaluation, you know, grab your full inventory, you know, that should be central logging, you know, having that list accessible.Because a lot of folks, what they’ll do is they’ll go in, they’ll look at their central log and they’ll go, hey, we have logs, so we’re good, you know, type of deal bid. And I remember when I set it up five years ago, we had all the boxes in there. Well, meanwhile, you know, of the, you know, 87 devices that are supposed to be logging, meanwhile, you know, 17 of them have dropped off the radar and, you know, nobody knows. So take your full inventory, make sure, you know, for most of the compliance requirements, there’s going to be at least an annual validation. I would recommend just as a sanity check, double check, et cetera, going once a quarter, just to, you know, take the, what should be logging? Is it logging? Do that analysis that way, you know, you’re not running into, you know, into issues. Because honestly, the time that you, if you don’t do that, and you do your annual thing for compliance, whatever, of course, during the middle of the year, when you’ve got an event and I need access to the logs is when you find out that, oh crud, you remember how we upgraded that one? Yeah, well, you know, whoever didn’t go ahead and turn on, you know, turn on the central logging, you know, and so you end up finding out about it and I’ll call it inappropriate or poor ways, shall we say. That said, you know, considering an easy way, you know, or possibly looking at outsourcing for doing your log reviews, you know, you know, reviewing those logs for for most compliance requirements is a is a is a necessity. And really, for pretty much any organization. Right. I mean, yeah, it’s one thing to go ahead and have all your logs pumping to the central location, but You know, those are your those are one of your biggest tools for telling if things are going sideways or if the things you need to be aware of, etc.
So, you know, doing log reviews is an important element of central logging. You know, people often look at the notion of what do I want to, you know, kind of build my own, you know, logging system or do I want to go by the logging system. You know, there’s a especially for organizations that are first stepping into it. I mean, the outsourced the the the the people out there that do You know, kind of logging as a service and manage logging, etc. It’s not cheap. It’s not cheap because it’s not easy. And, you know, they made a good amount of investment for anybody that’s physically done, you know, manual reviews of logs. I’m putting my hand up in the air right now. I got the joy of having to do that back in the day. It was 18 months of needing to go through and kind of parse logs, you know, on a system. Yeah, I’ll tell you what, it is not it is not in the grand scheme things cheap and it’s definitely not fun. You know, so if some organization has people sitting around with nothing to do, then yeah, great. Go ahead and, you know, Whip somebody at it to go spend their day perusing logs and all that fun stuff. But otherwise, for most organizations, it’s not a they end up discovering it’s not a it’s not a valid. You know, valid use of their time and especially as the organization grows, you know, it’s just you would be just floored at how many millions of lines of logs. Some of these things will generate And so it really becomes quite a quite a burden or an ordeal. And the problem with a lot of organizations that out of the gate, they look at the price tag right on the on the by side of that equation that I’m not spending that money out of your mind, you know, and they build it ourselves. And, you know, meanwhile, you know, you’ve got whatever, six to nine months of, you know, putting everything together and getting it in place and, you know, working out kinks and bugs and blah, blah, blah. Meanwhile, you’ve you’ve already in the in the internal cost, you’ve already expended, you know, whatever, three to five years worth of your logging coverage, just, just, just in what you need to do to get it to, you know, get it to semi functional, let alone what it takes to do. You know, kind of do it on a day by day basis. So a lot of folks will have the, you know, we’ll have the by route, especially with the central logging, you know, but the downside of that is it’s it’s money, money out of pocket, you know, but, you know, you’ve also got, you know, kind of a third party involved, you’ve got somebody that is not part of the organization that is, you know, that is doing the log reviews. And so, in some ways, it mitigates the, you know, the notion of an insider. You know, performing nefarious activities and whatnot, not the least of which is, you know, the people that have to go in and look at those logs, generally, or some of the more expensive resources at any Yeah, particular organization because you got to have somebody that knows what the heck these logs mean, you know, man.
Yeah, I do.
I was taking a sip of coffee, I’m dying. All right, that said, kind of the last nugget, if you will, is for folks that are heading down that kind of outsourcing of their central logging slash log reviews, I’d recommend that go ahead and leverage some type of proof of concept to get everything kind of tested out before you end up in some, all the vendors, they’re going to love to hook you onto some three-year contract or whatever, right? Well, don’t go down that path until you get through some type of a proof of concept, because the reality is that, while there are capable folks, individuals, and vendors out there, there’s also ones that aren’t. So there’s a wide degree of capability in this space. And so, number one, the POC to validate and test, all the things I’ve been talking about already, right? Is it doing what you want? Can you get access to what you want? Really for, and this sounds like an odd one, but there are some of the logging provider, log review, outsource logging providers where you don’t have access to the logs, you know? What happens when you’ve got to go in and look at your own logs, you know? It’s like, but they’re not accessible to me? You know, it’s things like that. So, you know, just kind of think it through, take your time, all that fun stuff. And then even if you get, you know, you end up kind of finding your forever partner for your central logging, and everything’s going along swimmingly, you know? I mean, I’m a big fan of trust, but verify. So, you know, go through, do a periodic review of the, you know, kind of what were all the logs that, you know, once you’ve got it up and running, right? There’s certain logs they’re going to go ahead and blow alerts on. Hey, we’re going to let you know about this, we’re going to let you know about that, et cetera, but these logs over here, they don’t need to worry about those. Well, that decision is cool, you know, whatever, in, you know, for this year, but now you fast forward, you know, two, three years, right? You know, sanity check, whether or not the logs that are going in are really logs that you care about or not, because, you know, at the end of the day, yes, it’s a partnership between you and your logging provider, but at the end of the day, you know, it’s your organization that, you know, that really is going to get impacted if something goes wrong in the, you know, kind of in the logging space.
That tracks, appreciate it. And that right there, that’s the good stuff. Well, that’s all the time we have for this episode of Compliance Unfiltered. I’m Todd Coshow and I’m Adam Goslin, hope we helped to get you fired up to make your compliance suck less.
