Small-town street scene in Illinois American flag flapping in breeze by huge painted American flag fading from brick wall

Episode #23 - Penetration Testing: What You Need to Know (Part 4)

This week in the final part of our Penetration Test Mini-Series, the guys discuss the realities of automated vs. manual penetration tests and what those terms actually mean. They also talk about timeframes, approaches, and situations that seem to cause some confusion for companies undergoing their first penetration test. The team also gets into Virtual CISO Penetration Testing and the VCISO Pen Testing Requirements.

Pick up your copy of Cyber Rants on Amazon.
Looking to take your Cyber Security to the next level? Visit us at
Be sure to rate the podcast, leave us a review, and subscribe! 

welcome to the cyber rants podcast where we're all  about sharing the forbidden secrets and slightly  
embellished truths about corporate cyber security  programs we're ranting we're raving and we're  
telling you the stuff that nobody talks about  on their fancy website and trade show giveaways  
all to protect you from cyber criminals  and now here's your hosts mike rotondo  
zach fuller and lauro chavez hello and welcome  to the cyber ants podcast this is your co-host  
zach fuller joined by mike rotondo and lauro  chavez and we are finishing up our series today  
on penetration testing hope you've learned some  from the previous episodes i've talked a lot about  
different types of pen test qualities  factors and considerations so we will  
continue to dive a little bit deeper today  and as such we are going to do a very  
brief overview of the news and exploits so we can  dive right into the topic of penetration testing  
mike zach so just got four quick headlines for you  today first headline is that billions of records  
have been hacked already make cyber security  a priority or risk disaster warns an analyst  
basically there's been 55 billion data records  that have been compromised since tok since 2005  
however in the last year alone for 2021 31 billion  data records have been compromised that's up  
171 percent from the previous year so it's getting  worse new bugs could let hackers bypass spectre  
attack mitigations on linux systems they found  two new vulnerabilities on linux-based operating  
systems that if excess if successful exploited  could let attackers circumvent mitigations for  
speculative speculative attacks such as spectre  uh their cve 2020 27 170 and cve 2020 27 171  
this i put in here because i couldn't resist  ziggy ransomware admin announced it will refund  
the victims who paid the ransom victims that  paid the ransom should contact you by email yeah  
at ziggy ransomware at sec  to be refunded in about two weeks  
victims have to provide the payment  receipt and the computer's unique id  
so yeah i never thought i'd see that  anyway microsoft exchange attacks increase  
while wannacry gets a restart yup wannacry's back  company saw 57 rise in ransomware attacks over  
the past six months at a global level there's a  constant monthly increase of nine percent since  
the beginning of the year we thought 2020 was  bad from a uh your perspective looks like 2021  
is going to be bad from a tech perspective as far  as attacks go don't make you want to cry exactly  
i cannot believe they're giving money back  i don't know if this change of heart should  
warrant some kindness from us but maybe not um  for uh for exploits this week we've just got one  
the vs ftpd uh product that offers secure and  fast ftp server for unix-like systems has a  
remote denial of service which is really  interesting so there is a really really  
cool python 3 script that was developed and it  will video us your ftp service so make sure if  
you've got vsfpd running that you're upgrading to  version 304 or greater and that is it for exploits  
this week great thank you both well i'm excited  about my refund uh that's i we've heard about  
we've heard about ransomware customer service  improving over time but this is this is next level  
yeah do you want your money back in  monopoly money or would you prefer  
cash you want boomer paper i'm just  kidding that they they want the receipt  
you know so but you know the other part of the  article when i wanted to say that you know they're  
not being all that altruistic basically they got  paid in bitcoin and bitcoins value has gone up  
so much so like yeah we can refund the original  amount and take the skim off the top so basically  
sure they still make money yeah i'm i'm still i  still don't don't believe it but we'll see well  
well let's talk about pen testing and answer  everybody's burning questions how does that sound  
sure let's put some fire out yeah yeah we've uh uh  covered it a lot over the last few episodes as far  
as just considerations and such go and i think  behind all this people are probably starting to  
see that you could look at it as kind of a good  better best approach right so there's there's  
no one-size-fits-all there's not necessarily a  right answer um and people are going to try to  
sell you all kinds of stuff so um that being said  i can i can say with almost certainty that that  
that kind of commoditized rinse repeat canned  approach is most the time not very beneficial  
but then beyond that there are varying levels so  one of the one of the questions that we get a lot  
is around manual penetration testing versus  automated and i think a lot of people from  
outside the industry at no no fault of their own  but they've heard this this thing about oh your  
pen test needs to be manual and so they ask oh  is it manual or what percentage of manual work is  
done without fully understanding what that means  um of course there's a bunch of great information  
about this in the book um but lauro do you want  to dive in a little bit and kind of explain your  
your theory on this and what's what and is there  a better option uh on one side or the other  
sure zach i i think there's a lot of convolution  around these these terms they're getting i think  
they get thrown around you know indiscretionally  and and it doesn't it doesn't help anybody  
so so let's let's talk about when when someone  asks you if you're if you're a cybersecurity  
professional or you're owning a business you're  getting a questionnaire or something and they said  
did you do a manual test or a automated test  what they probably are asking you is did you  
get a a tool to run a series of tests or  you know a scan basically did you get a  
an automated scan from one of your tools or  technologies or did you actually assign a third  
party with a human resource that had tools  and techniques at their disposal in order to  
conduct the test i think that's really where  the muster meets the brand right i think when  
when you get when someone gets questioned that  when i say hey did you do a manual on automated  
i think that's probably what they're trying  to ascertain is did you get a printout from  
a tool or did you actually have a knowledgeable  accredited certified practitioner conduct the test  
now let's let's look at the terms in general  how they apply to penetration testing as  
as the the the job would entail right from from  my perspective and and we talk we do talk about  
this in the book um i think it's it's it's quite  comical to me so i'll try to delete my humor  
aside for the moment but let's i always and in the  book i talk about this and and i i also use the um  
the analogy of of changing your tires okay and so  if i were to change my tires manually i would need  
to take the lug nuts with my fingers and undo them  right if i'm going to use an automated approach  
well then i'm going to then technically take the  impact wrench or even a manual crescent wrench  
and i'm going to to loosen those lug nuts  right now some people would still say that  
that's a manual approach right because i'm  manually exerting work it's just a kind of a  
backed off stage of manual right how manual do  you want to get reference and automated would  
be if i had you know a tire company or maybe  a mobile tire company just come to my house  
and do my my tires for me right and all i had to  do was swipe my card and that's it that was what  
i would consider to be an automated tire change so  when we're in the world of pin testing okay we're  
going to use we're going to use tool star disposal  it's like i have the impact wrench sitting there  
why am i going to try to take off the lug nuts  with my fingers just so i can prove a point that  
i'm really strong and and my bones don't break  when i try to do this it's going to be impossible  
right if there's 100 foot pounds of torque on  the bolt and the same in pin testing work it's  
going to take fundamentally longer for me to  do a bunch of nmap scans and a bunch of manual
msf open you know openmsf exploit validation  especially if we're talking about like a class  
c right a classy subnet 254 hosts that we need to  check okay why would i cost my client more money  
in saying that this testing period because i have  to do everything manually right quote-unquote um  
is going to take me you know 700 hours to do  versus if i use the tools at my disposal that  
automation and intelligent i.t has given us over  the years just to cut that time down and still be  
able to get the same type of data so we're going  to do a blend of both right as as the term goes  
so i'm going to use automated techniques because  they're fast and they deliver information quickly  
i don't have to wait around for them but i'm  also going to do manual tasks like validating  
that the vulnerability is actually a vulnerability  and it in and provides some form of an attack  
surface and is not just a misconfiguration that  you need to patch there's a complete difference  
between an exploitable vulnerability and a  vulnerability that's simply a misconfiguration  
so we're going to do that manual validation we're  also going to to do manual tasks around like web  
app calls and things like that we're going to  you know get in the command line and do curl  
things right we're going to maybe write python  scripts that's going to be some manual tasks to  
support other automated tasks right i'm gonna i'm  gonna spend some time manually creating the script  
but when i push the button it's gonna run right so  there are a blend of both that's gonna happen and  
so i kind of think that there's a there's a little  bit of a misunderstanding in in the industry about  
what this term means and also how it's applied  in in actual penetration testing work versus  
what maybe asked in a questionnaire or by  a human in conversation about a business  
connecting right and and proof of uh a technical  assessment of penetration tests being conducted  
and and if it followed nisk protocol right and i  think that's really what they're trying to get at  
mike you've got anything yeah i think i  think there's just so many nebulous terms  
out there that the standard just really isn't  defined and nist is as close as we get and
so and the problem is that a lot of times the  people requesting it don't have a full grasp  
of what they really need not anybody that's  listening to this podcast of course because  
you're all intelligent human beings but um you  know they a lot of people out there just don't  
know what they really want or need so you've  got somebody in sales side who throws a bunch of  
technical jargons out there that sound really cool  and they say okay great that's commoditized it's  
you know within my budget and that works and they  wind up with a scam as opposed to a pen test um  
so i i think the problem is that you know we don't  have a terminology a glossary that that mixes you  
know we've all moved to company to company where  you one of the first things you get is access to  
the lexicon so you understand what everybody's  talking about because each each company has  
their own language um in certain things so um but  yeah i mean the standards would be nice nist is as  
close as we get to a standard but you know again  we have different terms that have evolved over  
the years and if you've been around as long as  i have you've seen terms come and go and change  
the building of the pyramids the creation of the  wheel exactly you know when we made those first  
computers out of stone um you know that's when it  was all manual pen testing it was all manual yeah  
but uh but yeah nowadays if somebody tells you  they do a hundred percent manual run because  
they still haven't completed their first pen test  that they started working on 15 years ago because  
that's what it would take right i mean you have to  use tools but that doesn't mean it's an automated  
pen test i mean imagine if you try to what does  metasploit have in it like something like 4 000  
exploits or something at any are you any given  time i mean if you tried to build every single  
one of those yourself without a tool it would be  that would be your career right there it would be  
it take it would take too long and so i i think  that to me this spawned out of sales right where  
they they tried to get they were trying to they  were trying to ramp prices based on terminology  
it's like well we can sell you an automated test  for this or a manual test for this right and and  
it i see that that's typically where a  lot of the new terminology is coming from  
yeah is is really just around that i hate  to say bernasey and marketing paradigm  
right where where they're trying to use words  that create a different response in the human  
but you know it doesn't really hold ground to  any practitioners who do this work because we  
i mean that's the beauty of automation that's  the beauty of technology is that we we create  
these tools so we don't have to do it again we've  invented the wheel once right so we don't have to  
try to figure out how we roll around you know  i would have you know figured out the propeller  
blade first you know so but in any case that's  what's important is that we create these automated  
tools that we can use that provide value um and  and cost savings to our clients like you know  
again that that 10-year pin test is going to be  expensive versus one it's going to take me a week  
right well let's talk about the term threat hut  real quick that sounds like a video game to me so
this is like a whole can of worms  it is devsecops and zero trust and  
sassy and all the terms that are really  just describing something yeah it does  
it's like you should have a nintendo uh with  with some some plastic guns playing threat huh
running through a computer and you're shooting at  it yeah yeah i mean it technically is shooting at  
moving targets though right yeah well some of  the malware some of the stories that we've been  
reading lately or the news stories some of the  stuff is just you know exists in memory only you  
know there's no there's nothing to find but traces  of the fact that it's been there but some scary  
stuff out there that they've been developing from  a malware perspective yeah oh sure absolutely yeah  
i think i think we'll always see this this  asymmetric type activity right always increase  
in its upward evolution yeah be i mean you know  right because it has to um you know they they  
they see defenses and then they you know they work  around them based on the tools and techniques that  
we're using to defend right so i think it's just  an ever-evolving it's an ever-evolving battle  
right i mean it's just you get you throw more  cats and dogs in and that's all you're doing right  
you know it's just it just gets bigger just  becomes the brawl um so you know how do we how do  
we you know how do we really defend against that  and and i you know i think that's a good that's a  
good segue to to to ongoing penetration testing  versus point in time right which is i think  
i don't even want to dive into the threat hunt  can of worms really because it's i i could pull  
out the soapbox in a lectern and chat about this  for a really long time a whole podcast on that  
i got a week's i could i can i could be like a  threat a threat hunt is nothing more than a than  
a pen test on steroids change my mind right and  i would love to see what people have to say about  
this but again i believe that was a marketing term  that was thrown out um in order to provide a a  
seemingly more seemingly more um excited response  in in in their sales targets right but let's talk  
about what what really sticks and that's ongoing  penetration testing versus versus point in time  
right and and that's what we typically see is  everybody's usually point time right because they  
have their own box right absolutely right they got  they got cis or they have pci or whatever and it  
says you gotta get that test done right so they're  gonna get one done once a year and you know  
usually we find it's the end of the  year right the month like in november  
we've waited all this time let's hurry and  get it done right and so they're they're  
really not looking at the intent of what a  penetration test is supposed to serve you from  
a risk perspective right i mean they're they're  not looking at it from that perspective at all  
they're just like framework says we gotta  get a pen test what's that we gotta get one  
right do we need an automated or manual right  so and that's okay right you're doing something  
and we're we're never going to discredit that  right you're doing something that's important do  
something right it's better than doing nothing but  a point in time pen test is never going to give  
you an accurate risk assessment or risk ranking  on your environment as a whole right it's going to  
tell you whatever you have in scope you're one web  app maybe you got 40 and you're just like well we  
can't afford to do all 40 the shoes we want right  so you're going to do one web app in one time  
of the year and you're going to say yeah we had  to pin test it you know this this this one app  
was fine okay well how often is your agile spreads  you know how are how some of you making changes  
to that web app there's a big difference um with  with having ongoing risk assessment services right  
penetration testing activities versus a point in  time and if you don't have those types of things  
and you know you made a good point earlier  mike about the output of pen test is important  
because it's gonna it's gonna find some trivial  misconfigurations on your web app as an example  
right well why did they why did you deploy with  these trivial misconfigurations in the first place  
are you considering this for your business is  that okay to do are you saying that having okay  
so we've talked about this before in other pin  test episodes right the x frame options headers  
not set is not an exploitable vulnerability  and 99 of the cases you're going to find  
but but basic entry little noobs in this game  we're going to call it out because i'm going  
to try to get a bug down here when i try  to show them impressive on a report right  
but let's get past all that and say why did you  deploy a web application with the x-frame options  
that are missing in the first place is it because  you had a compensating control in the javascript  
or is it just because it was too hard it required  too many dev hours there's a reason there or was  
it negligence right but like you said mike that  that information can come out of that pin test  
and say there was no exploitable findings right  nothing serious but you had some trivial things  
why did they get deployed like that can we  go back to the document set and say in our  
software delivery you know standard we need to  add a place for doing this type of ongoing pen  
test right now it can just be if you're just  a vet if you're doing the right devsecops  
right which is what this is and you're like oh  we probably should deploy code with all this  
crap in it yeah you should probably have a you  know in the military we call it certification  
and accreditation so you certify that code before  it goes live that certification process includes  
a pen test right we want to run it through  some form of probably an automated scanner  
right to just determine or you know something  like a a a three letter i won't use their name but  
one of the three letters big tools that they have  that they sell that are hundreds of thousands of  
dollars you could run your code through that  and validate if you've made any mistakes in  
this change before you go live or you can go live  to a staging environment and have a third party  
pin test it it depends there's a million ways to  do this process but that's the benefit of ongoing  
activities you don't have to call it an ongoing  pen test right no one's going to come in and just  
threat hunt for the rest of your life right to  throw the tiki turns around but pen test like  
processes can be built in and baked into your  everyday operations so that you are conducting  
seemingly ongoing campaigns the other thing too  is the cleanup from the fine why do you still have  
ssl 2.0 out acceptable i mean you know why is that  protocol i'm still running you know where you can  
accept those why is it still why is it still being  found in this game why is this ssl 3.0 being still  
being found why what you know why is that stuff  and not cleaned up right right can't be looking  
for your age so you find this stuff and then you  don't don't fix it well it only takes you know  
15 minutes to make that change well it doesn't  make the change right so what's the document say  
and i want to ask a lot of it you know it's not  it's not our place right at the time to you know  
obviously right but i in my head i'm thinking  like i want to ask a client like what is your  
what does your um what does your data set or  your data standard say about having these types  
of vulnerabilities is it okay to use right um what  does your encryption policy say is it okay to use  
ssl version two and version three right is that  still acceptable for your business right and i  
already know the argument for some organizations  be like well we have we have clients that have  
older machines and they might not be able  to connect our application right so that's  
always right the reason for leaving those opening  however i will tell you that all modern browsers  
okay anybody who's not using a windows xp machine  from 2002 is on a modern browser chrome firefox  
doesn't matter right the safari they're all going  to prefer a pki modernized approved security  
connection over they're always going to prove  that over something weird you know what i mean so  
again we have that conversation with with older  versions of tls and ssl and older versions of um  
uh older versions of your connection oriented  methodologies too and it's like it shouldn't  
a great example 80 and 443 people leave 80 open  and they're like well we would redirect to 443  
what browser in the last five years is ever  kind of going to not warn you when you connect  
to a web application over port 80 it's going to  say you can't go here this is bad like there's  
no security like you can't go to this website and  so i don't understand the argument right based on  
modern technology well isn't it that microsoft  came out with an edge because they send down  
i.e that their edge won't even connect to port 80  anymore or they're moving to that in some way sure  
literally i mean even if port 80 is open traffic  is allowed it still won't allow it to connect  
sometimes you do hear the um the company's  justification is oh well we have three clients  
that are still using aol you know whatever they're  they're worried about i'm one of them i'm one of  
them i mean it's almost at that point it's almost  like we'll send them all new laptops and get them  
on a new browser and it would uh be cheaper  to do that than suffering you know our clients  
what's that i'm guessing they're not your high  dollar clients it's kind of like music you can  
use a war analogy because north korea still uses  biplanes we shouldn't send f-22s to south korea  
so it's crazy but that's you know you you hear it  i think a little too often um that oh well we you  
know our clients have these old old browsers  we're worried about them being able to connect  
well let's bring it we ran into a place  where there was a lot of customer facing  
websites right where they would be able  to get quotes for different things and um  
you know it's like well we don't know what the  end users are using you know it's then we can't  
we can't control people and it's like well at a  certain point you have to protect the house right  
you know your 16 year old keeps forgiving  his keys it doesn't mean you leave the  
door unlocked every night that's a great  analogy man that's that's perfect actually
that's perfect if we all join the same team  and everybody gets their systems right and  
then all the these people on deprecated systems  will be forced to upgrade because they won't be  
able to do anything they're trying to do  and then the world would be a better place
for a while for about five minutes yeah five  minutes so to circle back right i i hate i hate to  
i hate to ruin the uh their rhetoric because  i'm having a great time but i know everybody was  
like what's the point well the point is is that  ongoing is going to be point time any day a week  
our recommendations you should have a third party  doing you know um we have manual and automated  
penetration testing at least quarterly right  you should you should have a third party looking  
at yourself quarterly if you if you really want  to understand what your like risk period is and  
mike's going to jump in any minute and say and  internal penetration testing too because we see  
everybody skip out on that yep and everybody skips  on that and after every major release yep and of  
all the you know you just gave that headline  of all those billions of records that have been  
leaked and i want to tell you something okay all  of the information that everybody needs to break  
into your organization is probably out there in  pieces okay what what what you're up against is  
the time window okay that data is so vast to look  through right now that's what takes time it's like  
the proverbial needle in a haystack the  needle's there we know it's there it's  
a matter of the time it takes to dig for it  right and so that is what you're up against  
with all of these records that are now out there  in mass they are there for everybody to look at  
everybody to gain information on that's going  to contain password hashes common passwords  
email addresses right they're going to have a lot  of stuff about your organization that's there it  
might be old but you need to ask yourself is you  know is that time window that you're up against  
worth the you know worth having the the uptick  in in cyber security's you know related tasks and  
you don't have to hire that third party okay just  there's a book that i know a couple smart guys and  
one dude with a beard wrote it's the least of the  three but in any case you can get that book and  
it'll tell you about all the stuff that you can  put in your everyday operations processes to like  
simulate a lot of this work and reduce  your risk quite efficiently and cheaply  
well you know another thing too that i think is  forgotten a lot is as people start to go back  
to offices uh don't forget about your wireless  environment too and that seems to be one of the  
one of the weakest points of an office environment  at least from what we've seen in testing that you  
know we've we've done with clients so don't  forget about that if you're still running wep  
it's timed upgrade yeah because you might have  you might have clients with older laptops that  
you know don't use you're still  running the web you need to just  
grab your waps put them in a box  send them back to the manufacturer  
go practice the phrase do you want fries with that  yep how to work off only the only fax machines  
moving forward yeah well well this has been good  you know i hope everybody learned a lot there's a  
lot to unpack when it comes to pen testing and a  lot of misconceptions and confusion out there so  
our goal was to help remediate some of that if  you're looking for for some more information  
check out nist uh 800 115 is the is the  penetration test methodology recommended  
so it doesn't have to get too fancy i mean you can  kind of see the fundamentals of these activities  
and how they work um through a industry accepted  standard with nist and um that'll help you gain  
a better understanding but um yeah just know  that penetration testing is important if you  
you learned one thing out of this uh little series  here it's something that that everybody should be  
doing it's critical to understand your real risk  right so you might do a cyber risk assessment  
against a framework like nist or css cis  controls but uh that's not gonna do it right  
i mean just because it's in writing and people  say that they're doing something doesn't mean  
that the attacker from the outside uh doesn't see  things entirely different so that said any final  
thoughts just remember that the attacker  out there has nothing better to do all day  
but figure out how to attack you so if you get to  a static point you think all right we're secure  
you're not you're secure at that point but  unless you continue to evolve remember the  
enemy's always going to evolve so you have to  keep that in mind as you move forward there's no  
all right we've achieved everything  it doesn't happen so keep that happen
sand castles man it's sand castles on the beach  right the waves are coming you better you gotta  
have to reinforce the castle the way the water's  gonna take it away yeah well good stuff well  
thanks for listening everyone please subscribe  to the podcast rate us let us know your comments  
reach out i would love to hear from you we're  always looking for new topics that you care about  
and that's what's most important so hope this  helped you out and check out if  
you haven't already there's some information about  the book and such and uh podcast episodes are also  
on our website at have a great  day pick up your copy of the cyber rants book on  
amazon today and if you're looking to take your  cyber security program to the next level visit us  
online at join us next time  for another edition of the cyber rants podcast