Hiep Hinh is a Principal MDR Analyst at Malwarebytes, where he supports 24/7/365 Managed Detection and Response (MDR) efforts. Hiep has over 16 years of experience in the cybersecurity and intelligence fields, including for the US Army as an intelligence analyst and for the Airforce Computer Emergency Response Team (AFCERT/33NWS). Hiep is an expert user of Endpoint Detection and Response (EDR) platforms and is highly-skilled in incident response, DLP (data loss prevention), data mining, and threat hunting, among other things. In this post, Hiep breaks down his threat hunting career and shares tips and best practices for those looking to become a cyber threat hunter (or who are just interested to listen!).
When I first heard the words “cyber threat hunter”, I imagined a sort of Holmesian figure with an upturned collar sitting at a desk, scouring a network for signs of intruders. And when I talked to Hiep Hinh, Principal MDR Analyst at Malwarebytes, I found out I was more or less right in my guess—minus the trenchcoat, maybe.
Threat hunting is all about nipping stealthy attackers (and malware) in the bud. It’s plain to see why this is such important work—just consider that the median number of days between system compromise and detection is 21. The earlier cyber threat hunters can find threats, the earlier they can send them off to the remediation team.
Hiep has been threat hunting for a while—since 2007, in fact. According to Hiep, threat hunting is a natural part of incident response, SOC work, and network monitoring in general.
“I’ve been doing threat hunting for a decent amount of time. I got my start in cybersecurity at the Air Force Computer Emergency Response Team (AFCERT) in ‘07, where we monitored and defended the Air Force network,” said Hiep. “I did a lot of the forensics work back then, but we were still very deeply involved with just the network monitoring aspect as well.”
For Hiep, effective threat hunting starts with really understanding the network.
“I think to be an effective cyber threat hunter, you have to have a good understanding of what ‘normal’ behavior is,” he says. “For example, you should be able to answer questions like, ‘What are common activities seen in the environment? What are the users usually using? When are they usually online? What are they usually connecting from?’, and so on.”
“All of this information gets put under your belt, you take that knowledge, and now look for things that stand out. Using this understanding of normal will make certain activity stand out such as users that are on way too late, or are logging in from a different country than usual.”
“A threat hunter is most effective when they know the network well.” – Hiep Hinh
Hiep’s advice? If a cyber threat hunter isn’t a part of the company or used to seeing the environment, take some time to learn what is normal. It can be very overwhelming to jump into an environment with thousands of endpoints and separate malicious and benign activity.
“Threat hunting is used to find threats that aren’t caught by antivirus or your other defenses. It’s literally looking for things that are unfound, advanced, and hidden, right? So the only way to do that is by knowing what’s normal, and trying to catch that weird stuff, keep catching those outliers.”
If worse comes to worse, however, and a cyber threat hunter doesn’t know the network well, Hiep says there are “low-hanging fruit” you can look out for.
“It’s easy to go after low hanging fruit. It’s easy to go after a bunch of indicators, like lists of hashes, looking for VPN and RDP tools, and looking for a lot of freeware stuff that generally is used during attacks, such as IP scanners.” says Hiep. “These are the really quick and dirty threat hunts, if you don’t have a lot of time, and you don’t have the ability to actually sit on the network for a while and find out. These findings can potentially lead you to more juicy activity.”
Of course, while threat hunting is undeniably an essential component to a security team, we want to prevent bad actors from accessing our systems in the first place. To that end, Hiep told me about some of the most common ways adversaries break into an environment.
“The most common thing is credentials being stolen or used for to get into these systems, things like phishing. That’s like, the quickest way to do it,” says Hiep. “Otherwise, there’s other ways such as vulnerabilities that people can exploit to access your network. That’s why it’s good to keep everything updated.“
One of the things I found most interesting about my conversation with Hiep was how much of a science and art threat hunting is. Just like how scientists form a hypothesis about something before setting off to prove or disprove it, so do threat hunters. If a cyber threat hunter notices an unusual spike in network traffic, for example, their hypothesis might be that there’s an attacker on the network doing data exfiltration.
Hiep’s cybersecurity “battle-station”
Back view. Hiep may or may not be a fan of Godzilla.
Hiep describes what hypotheses look like in threat hunting:
“Your hypothesis lets you target a specific problem so that you don’t get overwhelmed with all the different types of data at your disposal. As a threat hunter you hypothesize certain attack scenarios, one example could be data exfiltration.”
“Knowing that attackers may want to steal your data to ransom or sell to a third party. We could then focus on data coming out of your network. Here is where having a solid understanding of average traffic in and out of your network becomes extremely useful or if users in the environment actively use file sharing sites.”
Like any hypothesis, however, there is a chance that it’s wrong and the thing you’re investigating is totally normal. A big part of threat hunting is not necessarily trying to prove that an anomaly is bad, but rather just validating the activity.
“You’re not always gonna find something when threat hunting. There’s a lot of hit and miss. Whether or not my hypothesis for some potential malicious activity bears fruit, however, the act of finding or not finding something leaves the environment safer or validates activity seen.”
“Just because I determine that the system is downloading and uploading a ton of data doesn’t necessarily mean it’s bad. Maybe a user is just sending out their christmas pics from the last decade. It’s not bad, it just stands out.”
“There has to be like a very solid communication between the threat hunters and the IT and the security departments of the company so you can quickly go through all those validations and move on. Otherwise you will just kind of be spinning.” – Hiep Hinh
The uncertainty of whether or not an indicator of compromise (IOC) is a genuine threat or not is part of what makes threat hunting so difficult, especially when you consider the vast amount of data threat hunters have to take in from all of their endpoints. That’s why threat hunters need to rely on more than just their skills to help investigate IOCs—they also need the right Endpoint Detection and Response (EDR) platform.
“You’re gonna get an overwhelming amount of data, and will need to put it into segments, separate it, understand it, and then, potentially find something that stands out. So it’s tough. You need something that can dissect that data quickly, effectively, and present it to the threat hunter in a very clear and easy to manipulate tool, this way you spend more time finding baddies and not be bogged down in data prep.”
Like many cybersecurity professionals, Hiep’s career is full of twists and turns; he’s probably seen more sides of cybersecurity than you can count on one hand. That includes SOC work, forensics, malware analysis, and more, each of which Hiep feels has over the years given him a leg-up in the world of threat hunting.
“Working in a bunch of different positions throughout the years is helpful because threat hunting is all about knowing what’s normal, right?” Hiep says. “And at some point in your career, you’ve gone through the gamut and looked at tons of things. This experience helps you get through the noise and make determinations on actual malicious activity.”
If you’re an aspiring threat hunter, try to get as much experience as you can working in network monitoring roles. An experienced cyber professional can look at a wall of alerts and go, ‘I’ve seen this many times. This activity is normal. This is somebody just doing XY&Z’. They can then look at another and go, ‘That’s strange.’ But, according to Hiep, they can’t easily tell you why it’s strange.
“There’s nothing that really teaches you that,” Hiep says. “It just comes from working it for a long time, like any other job, I think.”
Dedicated experts, precise technology
Hiep is just one of many experienced cyber threat hunters on the Malwarebytes MDR team. Purpose-built for resource-constrained teams, Malwarebytes MDR provides alert monitoring and threat prioritization with flexible options for remediation—at a cost that makes sense. Our highly-effective, easy-to-deploy EDR technology coupled with our team of security experts creates the perfect one-two combo for fighting cybercrime.