Compliance Unfiltered is TCT’s tell-it-like-it is podcast, dedicated to making compliance suck less. It’s a fresh, raw, uncut alternative for anyone who needs honest, reliable, compliance expertise with a sprinkling of personality.
Show Notes: Penetration Testing Deep Dive
Quick Take
On this episode of Compliance Unfiltered, Todd and Adam take an in depth look at the interesting arena of Penetration Testing.
Curious about the difference between vulnerability scans and penetration testing? Wondering about the differences in approach to penetration testing? Fretting about how long it actually takes?
Well, you’re in luck! All these answers and more on this week’s Compliance Unfiltered!
Read The Transcript
So let’s face it, managing compliance sucks. It’s complicated. It’s so hard to keep organized and it requires a ton of expertise in order to survive the entire process.Welcome to Compliance Unfiltered, a podcast dedicated to making compliance suck less. Now here’s your host, Todd Coshow with Adam Goslin.
Well, welcome in to another edition of Compliance Unfiltered. I’m Todd Coshow alongside the Friday night to your compliance weekend, Mr. Adam Goslin.
How the heck are you, sir?
Um, feeling fancy today, Todd. How about you?
I can’t complain, sir, I truly can’t.So today, we’re actually going to do a deep dive on penetration testing, a penetration testing 101 for the uninitiated. So let’s set the stage for the folks out there that may be less familiar with this than others. What is penetration testing?
It’s a form of security testing that’s performed by a group of experienced security engineers. It’s an engagement where you can effectively take these experienced engineers, target them toward your environment, and almost look at it as if somebody were kind of, you know, attempting to attack your organization, but they’re on your side.In other words, yeah, they’ve got all the same med skills as the bad guys out there, but they’re on your side. They’re engaged to do the testing on your behalf, and the real intent of a penetration testing engagement is really to assess where do we stand in the grand scheme of things. Do we have vulnerabilities or logical faults, you know, within our systems or applications that someone could otherwise take advantage of? Those would be the, you know, kind of the primary, primarily at a high level, that’s kind of a good overview of the, you know, kind of the penetration testing.
Now, often you’ll hear folks confused between vulnerability scans and penetration testing. How are they different, and what are the advantages of each?
Yeah, that’s actually very common. A lot of folks will look at them interchangeably, you know, refer to their, you know, they’re talking about their penetration testing, but they’re referring to their vulnerability scans or vice versa and kind of blending the two, if you will.So vulnerability scan, a vulnerability scan is the best way I can describe that. But it’s almost like an antivirus because it’s something a lot more people will be familiar with. It’s almost like antivirus, right, where you’ve got this system that’s running on your machine. It’s looking for, you know, no, no bad file patterns or activity on the target device, comparing that to a list of patterns that it has in its, you know, kind of in its log, if you will. And if it sees a file or a pattern that matches something in the no, no bad stuff log, then it raises the flag and says, hey, Houston, we might have a problem here. You know, that’s basically the basics of vulnerability scans is they’re typically targeted at the network layer, you know, and I’ll say they do a decent job with kind of picking up low-level stuff, network layer stuff. But that’s about the boundary line of the, you know, of the vulnerability scanning arena. Now penetration testing, on the other hand, that’s an engagement where vulnerability scanning tools are one of most likely a dozen or more different automated tools that the testing team would leverage. So, you know, when they go in and do a penetration test, yes, that would typically include one or more vulnerability scans of that network layer. But the big difference in the penetration testing arena is it doesn’t just stop at the network layer. So penetration testing could be, it could be on applications in the application layer. It could also be, you know, targeted at APIs and web services. It also could be directed to, you know, directed to, what’s the word I’m looking for here? Wireless with the, you know, wireless configurations and whatnot of the organization. So long story really, really short, vulnerability scan is an easy to use tool that’ll get you low level stuff at the network layer where pen testing is really a much broader scope, you know, much broader capability and brings to bear the, you know, the power of experienced security engineers so that it’s not just, you know, kind of a machine going and doing what it does, but you’re gaining the benefit of real live human beings that can, you know, that can do what they need to do as they’re going through and, you know, kind of doing that test, if you will.
Mm hmm. And speaking of the differences, do different companies do penetration testing differently?
I want to tell you what, the variance in the kind of pen testing companies, it has been for some time, almost like the wild west. I mean, the way that I would typically describe it to folks is at the, I’m going to call it at the entry level perspective when you start dipping your toe into the pen testing pond. At the entry level are, you’ll see companies that are hawking penetration scans and automated penetration testing. And long story short, it’s some organization that’s effectively taken, sometimes they just take a vulnerability scan and add a couple of nifty, nifty doodads and call it a pen test. Sometimes it’s got a greater level of depth, but it’s still just an automation that’s going, running through environments and spitting out results. Effectively at that level, you’re dependent on the skills and capabilities of the team that put together the automated testing to equate to, is this going to be a quality test or not? And unfortunately, the devil is truly in the details. So that’s kind of like your entry level pen testing arena.You know, those ones will also typically be cheaper because all you got to do is go in and punch in your, you know, punch in your scope and hit the go button and X amount of minutes later or hours later, poof, you got a pen test, you know, result in your hands. You know, at the other end of the spectrum are the companies that will bring in what I’m going to call penetration testing ninjas that drop from ceiling tiles. So this is the other extreme, right? You know, you go and you go, you pick up the, you know, you go and you pick up the bat phone and you call in the, you know, you call in the, you know, the crew and, you know, they’re literally, they’re, they’re extracting military professionals from like, you know, sealed bases and blah, blah, blah, flying them in on Black Hawks, you know, dumping them off at a, you know, at your local, you know, your local hotel, you know, and whatnot with the team basically camping, you know, there remotely for weeks at a time. I call them the penetration testing ninjas that drop from ceiling tiles because, you know, quite literally they’re, they’re in, they’re trying to do, you know, like, you know, physical breaches of the, you know, of the, of the organization. And quite literally, you know, you got these dudes, you know, kind of a mission impossible style, you know, coming down from the ceiling tiles and trying not to trip the laser beams and blah, blah, blah. You know, in the same sense, those style organizations where it’s like, you know, it’s personnel on site with the, you know, with the ninjas in tow, you know, that’s where you’ll get, you know, things, you know, oftentimes they’ll integrate in like social engineering, you know, onsite social engineering extravaganzas and, you know, physical, physical breaches and things along those lines.
So, you know, at that end, you know, and they’ll literally, they’ll set up a whole team of people that will come in and sit right on site and, you know, go ahead and execute the pen testing. And then, you know, so you got these two extremes that I’ve talked about and somewhere in the middle is some balance of, I don’t want the, I don’t want the cheapy, cheaply press a button pen test. I also don’t need the ninjas that are dropping from ceiling tiles. I want some sensible middle ground that does a balanced job of, you know, taking advantage of, you know, taking advantage of automation, you know, things along those lines.You know, the, at the end of the day, the good news that that’s always been for a long time is that the pen testing arena has really been like the wild west. The difficult part is that for the companies that need penetration testing, unfortunately, they don’t, they don’t have enough context, knowledge, exposure, experience, capability even to assess, you know, what am I getting here? Is it going to be, is it going to be any good? So, you know, it’s, it’s kind of tough to go through and make those selections. But it’s, it’s definitely been, you know, kind of a challenging arena, you know, to go in and kind of get that, get that piece figured out.
For sure, for sure. Now, for some of the blocking and tackling, how long do penetration testing engagements typically take?
Well, I mean, we just, we were talking about the, you know, the, the, the scan it and forget it’s of the world. So, you know, that could be, you know, minutes to hours, uh, you know, type of a thing. But for anything that’s in that kind of sensible middle ground, you know, arena, etcetera, you’re pro you’re typically talking about, I think it would be the short, shorter, shortest of them, maybe a week to 10 days type of a timeframe for the testing window, uh, for most average engagements, I’d say two to three weeks, um, you know, of a testing window. Uh, but for some large scale engagements, it could quite literally be four to eight weeks, uh, you know, type of a timeframe.It just depends on, you know, how much stuff is it that we’ve got? So I mean, it’s one thing to have an engagement with, you know, whatever less than, you know, less than 10, you know, less than 10 IPS, um, you know, tight type of a deal and no web apps versus, you know, thousands of IP addresses and, you know, a multitude of web apps and APIs, etcetera. Um, so there’s really a, uh, quite a great variance if you will, in terms of the amount of time that those engagements take. But I would say on average, like two to, uh, kind of two to three weeks, you know, the one thing we hadn’t, hadn’t talked about so far as that also, you know, we talked about its contingent on the scope. So from a scoping perspective, keeping in mind the penetration testing can be performed either externally, so AKA from outside of the target environment, emulating or simulating the, the, the kind of, uh, access or capabilities that someone from another country would have, uh, in terms of trying to attack the organization. Uh, and then the other, uh, you know, the other side of it is you’ve got the internal, um, you know, testing as well. Basically the internal testing goes under the guiding assumption that, um, in some way, shape or form, the bad guys have gotten to the inside of the network. Now that they’re there, what damage can they cause? So, you know, what I typically will say to, say to organizations like, well, we, we don’t, if we have a rock solid wall on the outside, you know, for external, then, you know, who cares if we do what we do with the inside? Well, the bottom line is, there’s zero days that are popping up all the time. Uh, somebody screws up a configuration and all of a sudden they’ve opened a hole to allow someone to get inside. You know, basically that the intent of that internal test is get the testing team on the inside, see what all they can find, gain, do, you know, and whatnot. And, you know, that is, uh, you know, that is definitely a, uh, uh, a good approach is to, to go in and get that, um, you know, go in and get that, that perspective of that internal testing. What I’ll see a lot of times is even if an organization is kind of a baby stepping their way into it, I’ll see them, uh, go, go about doing an external to start.
And then once they’ve kind of gotten their arms around the external, then, you know, then folding in the internal, unless they’re required for compliance purposes to be able to, to go through and, uh, to go through and do, uh, both external and internal, uh, and internal testing.
That makes sense. Now, what are the outputs that can be expected from penetration testing?
Well, when you go through and you do penetration testing, typically the net results at the end of the engagement are reports. Different testing organizations are going to structure their reports differently. They may have one report for the external, one for the internal. It might be one report to rule them all. They may even sub-segregate those reports. But at the end of the day, the outputs from the penetration testing typically will include some form of an executive summary of the engagement, confirmation or affirmation of the testing approach, confirmation or affirmation of the scope that was involved, typically some summary information around vulnerabilities that were found, how many in counts and things along those lines.And then it will typically go into the, we’ll call it the meat of the report, where you’ve got detailed individual findings. So basically what you’ll have there is you’ll have each finding. You know, it’ll identify what is the finding, what is the finding in terms of its title or name, what’s the severity of this finding, where all did we find this finding, how big of a deal is it, things along those lines will kind of be up at the header level for each individual item. You know, from there, there’ll be a description of what this finding means, like what is the problem here, you know, type of a thing. They’ll also then have, you know, where all did we discover it, you know, some more detail on how we discover it from within the environment with some examples. And then it’ll give you some directional guidance on how to fix it and maybe some referenced links to different sources that can be of help or assistance.But, you know, the intent of the delivery of that report is so that the receiving organization can go through, review what all’s been found and go about proactively, not proactively, but go about addressing those items so that they can, you know, go back and get them validated. Generally speaking, the reporting is at a fairly detailed level.Different organizations have different, you know, kinds of different approaches and levels of depth and things along those lines. But my expectation of a good penetration testing company is that if you’ve got good solid resources on your team, they should be able to take that report and that report and be able to kind of be effective as they go. So most of the time you’ll see the reports containing that. Sometimes the reports will contain some summary information in the appendices, things like what all open ports did they find on which IP addresses, additional summaries of, you know, kind of vulnerable ports, you know, across the engagement. Apparently, it’s easier to say than say, you know, that type of thing. So, you know, different companies do different things with kind of the appendices. But that report, just so that we’re amazingly clear, I’ve seen too many organizations that when someone says, hey, we need to see your penetration testing report, they’ll hand them that report and to my advice would be, oh, hell no.
No, no, no. I don’t think that there’s an organization on the planet that should be justified in seeing someone’s detailed internal penetration testing report.That should be an internal only asset that you’re definitely not sharing with any third parties because it’s going to have an astronomical level of depth, detail, scary data if it were to get loose, you know, things along those lines. It’s just it’s too sensitive of a report to be sharing help with others.
That makes sense. Now, for one of the most important parts, tell us more about the remediation testing process.
Sure. So when the organization goes in and delivers the reports, oftentimes there’s a stage after that called remediation. What is it? Well, it basically is the target organization that’s now received their report. Let’s pretend for the sake of this discussion, we’re going to say they had 10 findings on their report. So when they go in and they go in, get their 10 findings, you know, then maybe they go off and they fix the first, they believe they fixed the first five of these, you know, issues. They can let the, you know, let the testing team know, hey, I think I fixed the first five. And remediation testing is basically the testing team going back in, validating what it is that, you know, what it is that they located on the report and then validating that it’s been fixed or addressed appropriately. So, you know, it’s a, it’s a, typically a cyclical process. So when the client says, Hey, I think I’m done with five, you know, and whatnot, the team will go in, they’ll go, well, of the five you sent in four are cool, but one of them, you still got these things I’m seeing here, you know, etcetera. So there’s often some, you know, kind of some back and forth as you go through that remediation testing process.But the, the intent of the, of the remediation testing process is that the target organization gains clarity that yes, the items that have been identified are now confirmed as being closed and cleared by the security engineers. It’s just a really important sanity check as you’re going through and doing a penetration testing engagement. Cause I’ve seen even very capable organizations that believed things were fixed, have them not be so. And so it’s, it’s, it’s always good to have that kind of trust, but verify third party view of is this really addressed or is it not. On the second, on the next side of this, we’ve got, you know, the, the notion of the organization going through it, and this is a recommendation that I always give to folks that, you know, that, that I’m working with on, on penetration testing is that, is that I tell them when you get your report, go read through it, make sure that you understand it, make sure you don’t have any questions that the team does. Great. There’s things that pop up as you’re going through and doing the review of that report, you know, maybe the testing team said, Hey, the way you have to go fix this is fill in the blank. And for whatever reason, may be because of your circumstances, because you don’t have access to that technology, you have some limitation on your side that prohibits you from being able to do what they said, you got to paint a little bit outside the lines. So that being the case in terms of how to fix it, if that being the case, don’t feel, you know, don’t be shy. Go, go back to the, you’ll go back to the testing team to talk about any of those, you know, any of those items you don’t understand. But when all said and done, for the target organization, I recommend that they use an internal peer review process to validate that what it was that needed to get fixed actually got fixed and do so in a peer review style before handing it back over to the testing team.
It does several things. One is it will make sure that you are, have a greater chance of success as you’re pushing things through for the remediation, for a remediation testing.There’s nothing more disheartening than to whatever. Let’s say that your report has, you know, has 250 line items on it, different things that need to get addressed, and you go hand in a hundred and all of a sudden what you get back is you get, you get, well, that’s great that you handed me a hundred. These two are good. And the other 98 are not, you’re not doing you any good. You’re not getting any closer to your goal. You’re wasting the time of the testers and testing team, you know, type of a deal and it just stretches it out, you know, type of a thing. So if you do that peer to peer review of the, you know, of what was supposed to be fixed, is it really fixed? I have seen that yield extremely positive results when it comes to the success rates of passing things through, you know, to the testing team and having them come back with being kind of, with them being cleared instead of coming back on another round.
Well, what are some of the downsides of penetration testing teams like folding in remediation testing into their pricing?
Well, okay. So we were talking a minute ago about, uh, you’ve got an organization, one organization only has five, five things that were found on their report. And on the other side of the coin, you’ve got some organization that has 5,000 items on their report. Well, the reality is, the testing team before they go into doing testing, they don’t have any clue what they’re walking into. They don’t know if they’re walking into a five, five things on the report or 5,000. And so as a result, this is just a natural, logical way to go about doing it.You know, the testing organization has to protect itself. It has to go under a guiding assumption that they’re going to use the average. Well, if you, if the average, whatever, I’m just going to make a number up. If the average for this testing company is 250 results, then what are they going to do? They’re going to include enough time to cover the retesting, multiple rounds of retesting, uh, you know, for the average company. Well, guess what? If you’re an exceptional company, you’ve got a great team. Uh, you know, you’ve done a good job, etcetera, etcetera. You know, when the remediation testing is finger air quotes folded in, they’re not, this isn’t, this isn’t a charitable organization. They’re, they’re, they’re there to make, you know, make profit and hours equals cost. So yeah, you can bet they’re going to fold it in on an engagement average and effectively the organizations that are folding in the remediation testing on aggregate for the companies that don’t suck, they’re going to end up paying, you know, paying more, uh, because of the fact that their remediation testing is going to be, uh, you know, it’s going to be, uh, an average that has to be pessimistic. By the, you know, by the companies that are, you know, out there and doing the testing.
And I guess that begs the question, like, what are some things about selecting penetration testing organizations that listeners need to be cautious of?
Well, I mean, one thing for sure is clearly understanding the boundary lines and whatnot of how we’re going to go in and approach the penetration testing. So as an example, some penetration testing companies will fold in to DDoS attacks as an example, basically an effect to overload the infrastructure in the hopes that it opens up some type of a hold or a vulnerability in the platform. And the only downside is, if it’s successful, then they’ll basically bring your organization to its knees.Things like, where does the organization, where’s their head at in terms of identifying an issue? Are you dealing with a testing team that basically, I don’t know, I call it testing team shiny objecting. So they found this shiny object and now they’re on this like all out hell bent mission to prove unequivocally beyond any shadow of a doubt that this is a real, true issue. And so what do they do? They go and they spend, whatever, I’m just going to make it up. They go and they spend, you know, 12 hours trying to beat this one particular vulnerability into the ground to prove it beyond any shadow of a doubt. Well guess what? You know, if some other organization is not, if they’re going to the point where they know there’s a problem and then they move on, what that means is that that organization doing the testing, they’re going to still identify, hey, you’ve got a hole here. You need to go close. I’m just not going to go and bring it to the nth degree. Instead, they’re going to keep it moving. And now they’re going to be able to basically cover a greater amount of breadth on the organization, on the engagement for the organization.So you know, it’s just, it’s asking them some questions, make sure you understand how they do what they do. You know, for me, pen testing engagements is an expensive adventure. I absolutely want them to give me as much breadth as humanly possible so that I can take best advantage of this. You know, another thing to be cautious about, where’s your, not just where is the testing team company, but the individuals that are going to be in and doing the testing, where are they located? Are they in the same country as you’re located in or are they located in a different country? Part of where that comes into play is that if something, you know, it, now granted it doesn’t happen often on these engagements, but let’s say that for some reason you needed to take the testing organization to task, how much easier is it going to be when the testing team is located in some other country? It’s going to be unbelievably challenging to go do enforcement, things along those lines.So my general recommendation is just from an ease perspective, it makes sense, you know, use folks that are within the same country as you, that makes it a hell of a lot easier to go in and, you know, go in and do any enforcement that you may need to. You know, not all testing teams are created equal. You need to go in, scrutinize their capabilities, just like, you know, just like with, I don’t know, for somebody out there that’s got a developer, right?
You can have a developer that barely knows, you know, how to spell the language. You’ve got developers which have been engrossed in it for multiple decades. You know, what’s their breadth and depth of experience, etcetera. That’s another piece that you want to, you know, kind of key in on as you’re having the conversations. And then finally, making sure that these guys are actually compliant themselves, you know, with any critical vendor that has access to sensitive data, you know, within your environment, you’re going to make sure that they’re taking their own security seriously, like, oh, I don’t know where your reports are currently, where your reports are currently stored as an example, is that on lockdown, or is your organization and then the only ones that can get on to, you know, get ahold of those? Because they’ve got some really freaking critical information. You really want to make sure that you understand what’s going on with the testing team and make sure that they’ve got their I’s dotted and their T’s crossed from a security and compliance perspective, for sure.
No doubt. Parting shots and thoughts for the folks this week, Adam.
I’ll keep it relatively short and sweet for the sake of this one, but long story short, if you’re an organization that’s never done penetration testing, if you’re new to the penetration testing arena, by all means, reach out to us. We can give you a hand, answer questions, give you direction, all that fun stuff. I mean, penetration testing in my mind’s eye is one of the most important validation tools that you can put into your arsenal for assisting in the protection of your organization if it’s done appropriately, if it’s done by a high-quality vendor. It can be a very, very powerful tool in the toolbox, and it’s a great sanity check for your day-by-day controls that should be in place. It’s a great way to validate, are these controls in place? One of the things I tell people is I say, look, go through the items that showed up on your report. What you want to do is you want to ask yourself, why is this even on my report? What is it that we were or weren’t doing that caused this to even be possible that this thing is here? Look back at your controls, your internal process and procedure. Go ahead and look at it from that perspective, and that way you can really identify some major ways to make adjustments within the organization.
And that right there, that’s the good stuff. Well, that’s all the time we have for this episode of Compliance Unfiltered. I’m Todd Coshow and I’m Adam Goslin, hope we helped to get you fired up to make your compliance suck less.
