In working on AboutDFIR the last couple months, I’ve come to learn that while digital forensics and incident response share some basic foundational knowledge, they are widely different in practice. I’ve taken SANS FOR500: Windows Forensic Analysis and have been reading the recent articles about vulnerabilities, and have to say it’s been a series of eye-openers, especially coming from a law enforcement digital forensic background, as to how evidence and analysis can differ depending on the case.
Noting these differences led to several conversations with fellow AboutDFIR contributor Andrew Rathbun about how that shift of mindset and skills may look on a career transition level. After letting me pick his brain for a while now, we thought it might be a good idea to share some of those questions and answers for anyone in my position in the future. While these are a little more focused on coming from law enforcement, I think anyone considering the field would benefit from a read.
To set the stage, here’s some quick background on both Andrew and me at the time of this blog post. I am Cassie Doemel, currently a Detective Sergeant for a rural Sheriff’s Office. I am also responsible for digital forensic analysis for our agency. I’ve been in law enforcement for 10.5 years now and have been considering a transition to the private sector, which is part of what prompted these conversations.
Andrew Rathbun formerly worked in law enforcement for seven years sworn on the local level and one year non-sworn at the federal level. Andrew currently works as a Senior Associate for Kroll Cyber Risk and has been conducting digital forensics and incident response (DFIR) investigations for approximately two years.
Q1: If someone is considering incident response in their 5- or 10-year plan, what do you think is the bare minimum they should have before considering a transition to private sector incident response? Obviously, a lot of law enforcement officers have two-year degrees, is that enough? Should they add certs?
A1: Certs will help the Human Resources team at any place you apply better understand and measure your skill level against current employees and other prospects, and will likely make a stronger case for you to have a higher starting salary than if you didn’t have those certs. Sometimes, it’s not what you know, but what you can prove, and given that HR may not know the ins and outs of the cyber security discipline, they likely will rely on certifications to gauge or justify your worth when you’re onboarding. Over time, you can prove your worth in other ways for raises, promotions, etc., but having certifications is better than not having certifications, I’ll say.
Going from law enforcement digital forensics work to incident response, I’d say basic networking knowledge and log analysis are two major pieces I found were missing from my arsenal going into Kroll. Understanding what a domain is, what static and dynamic IP addresses are, local admin accounts versus domain admin accounts, basic Active Directory skills. Those are invaluable to understand because from what I’ve seen, 99.9% of systems in the incident response world seem to run Windows in an Active Directory environment! But like I say, that’s just my perspective, of course, at Kroll we have experts who investigate everything from mobile devices and MacOS to OT/ICS (the industrial control systems used in operational technology) and beyond.
Understanding the backend of an AD environment PRIOR to starting in IR will be much easier than learning these terms and concepts on the job while actively responding to incidents the first time, much like I did! Luckily, I had strong support from my Kroll managers and colleagues, who were always available for questions, and the peer review process we also have at Kroll, where there’s always a second set of eyes to check your findings and make sure we covered all the bases for any investigation.
Log analysis, from both event logs and logs generated by various software suites, was something that had very little relevance for me when I was investigating crimes like homicides and robberies – logs generated from various network infrastructure, like firewalls, were rarely relevant in those types of investigations. But I’ve found that they’re critical and can provide very important leads in the IR cases I’ve been working these past couple years. Learning how to analyze a different set of data compared to when I was working criminal investigations involving phones and computers was quite the learning curve.
To make a long answer even longer, any familiarity one can gain with event logs and log analysis may not help on a resume, but when it comes to on-the-job training for a lot of IR cases, it’ll make the learning process a bit smoother and less painful.
Q2: Beyond education, what else can someone do to set themselves up for the transition?
A2: I think the first thing anyone needs to do is hang out where everyone else is hanging out. What that means is join the Digital Forensics Discord Server, create a Twitter account, and spruce up your LinkedIn account. Digitally mingle where others who are doing what you want to be doing. If you’re trying to catch a train, don’t hang out at a bus station. Go where the trains are! I love my analogies, so I had to throw that in there! It’s a simple concept, but it bears stating so plainly because I see many do it but get nothing out of it for a variety of reasons. In life, half of the battle is showing up. The other half is following through via participating where you’ve shown up to. I cover this in the latest podcast I did for Chewing the Fat. You can find that conversation here.
Some other resources that are very helpful for transitioning into the field regardless of professional background are the following:
Should I Start a Blog? The Answer is Yes – ThinkDFIR
Starting a Blog – This Week in 4n6
Getting Started in DFIR: Testing 1, 2, 3 – SANS DFIR Summit
Securing Your Future in DFIR – SANS DFIR Summit
Q3: Say I’ve found a job duty or role I’m interested in. In law enforcement, generally, jobs have industry-wide consistent job titles with consistent duties. In cyber security, the same role can be given vastly different titles by different companies, or the same title can do vastly different duties. How would you best determine what a job is or what jobs you might be interested in?
A3: I personally think the best way to get an understanding of what a particular job role, job title, etc., is at a given company in each field (cyber security or otherwise) is to rub elbows with people who are doing what you are looking to do. That was the nice thing about getting into law enforcement, being able to do ride-alongs to experience the job firsthand. Since a lot of jobs in cyber security are remote, you’re not able to do ride-alongs like someone who is looking to become a police officer. So, your next best option is to hang out where the professionals are. Places like Twitter, Discord, LinkedIn, and Reddit are going to be the best places to be present and observe what working professionals are discussing with each other. Chime in when you have questions or if you can contribute to help solve a problem. This is how you network with people in the remote workplace that has become so common nowadays. The internet makes the world very small, but you still must know where to go to get what will help you the most.
So, all that being said, observe what working professionals are talking about in these social circles and don’t hesitate to ask in public or in private for mentorship or guidance on how you can get from where you are now to where you want to be. Places like the Digital Forensics Discord Server have a very helpful and willing community of people who’ve not forgotten where they came from and are more than willing to help send the elevator back down to the next generation of cyber security professionals.
One last thing about mentorship. If you’re asking someone to be your mentor, understand the dynamic in that if someone is willing to mentor you, there’s a very strong chance they’re doing the same for others as well. Don’t expect a mentor to be able to maintain proactive communication with you as they very likely have others they’re mentoring as well as their day job, family life, and whatever else may be going on in their lives. If someone has extended their mentorship and expertise to you, that is basically permission to reach out and ask any question, regardless of what it may be, pertaining to whatever you need to move the ball forward in your career to get where you want to go.
If the mentor is actively checking in on you over a sustained period, you found an awesome mentor and be sure to buy them a beer if you ever meet in person! There are only so many hours in the day, and I speak from experience that mentoring 5+ people at a time is hard to keep up with, being the one initiating interactions between the mentee and myself, despite my best intentions. I try to make the interaction as fruitful as possible for the mentee when they reach out because they’re putting themselves out there to me to advance their life and career. The least I can do is give it my all with every response they ask of me.
Q4: Networking, networking, networking (human networking, that is!). It has been repeated to me, above and elsewhere, as being incredibly important in getting into this field. How do you recommend getting started or finding a mentor?
A4: Putting yourself out there in any capacity, whether it’s as simple as asking in a public forum like Discord, Reddit, etc., “Hey, does anyone have any interest in mentoring me?”. I think the biggest thing is that everyone nowadays has so many things competing for their attention, so you have to meet a mentor in the middle, so to speak. Just leaving that question as is isn’t a big draw for someone looking to be a mentor. Why should I mentor you? What do you want to do? What don’t you want to do? What experience do you have? What experience are you trying to obtain? What goal(s) do you have for yourself that you have not met yet? Why have you not met those goals? What do you need from a mentor that’ll help you achieve those goals?
All of that is incredibly important because given how deep and wide this field is, there’s a very good chance I may not be the right mentor for what you’re looking to achieve. For instance, if a DFIR student asked me to mentor them so they could become an analyst in a SOC, I’d be the wrong person to mentor them. I simply do not have the experience they are looking to gain. However, someone like yourself, Cassie, who is in law enforcement currently, having been in your position as recent as three years ago, I can serve as an effective resource to someone like you who may be looking for the next step in their career outside of law enforcement. That’s a language I’m very fluent in and I can be very helpful to someone looking to make that transition. But it all comes down to this. If you don’t ask, the answer is “no”.
Q5: What are the top resources for trying to keep up or get acclimated to the incident response news?
A5: Twitter is the best place overall to stay in touch with those who are boots on the ground in cyber security in general. In DFIR, Twitter is a great place, but also the Digital Forensics Discord Server is a great place where digital forensics practitioners as well as those who work in incident response mingle together with vendors, students, etc. It’s been mentioned a couple times so make sure to check out the guide here on AboutDFIR for how to join!
Q6: Can you give a “day in your life” of investigating breaches or in your position at Kroll?
A6: Well, first I should say that what is typical for me might not be the same for my colleagues investigating cases involving different events or client challenges. But for me, on a typical day, I likely will participate in scoping calls. The scoping call is the equivalent to interviewing the victim to see what happened when and what, if anything, can we do for the victim (aka our client). Many details are hashed out about the five Ws during this call and if the scoping call leads to a new case (aka engagement) for us, then a statement of work (SOW) is established. The SOW establishes the parameters in which we operate, i.e., budget, how many systems we are to analyze, if email inboxes are to be analyzed, etc. Once we have an agreed upon and signed SOW, then we get to work!
Ultimately, we want to identify multiple investigative milestones. To apply a law enforcement metaphor, often it ends up being the equivalent of responding to a 100-car pile-up on the freeway and you must figure out what happened from A-Z and provide a battle damage assessment of everything that happened when, why, where, by who, and how. Instead of on a freeway with 100 cars, it could be on a corporate network with 100 computers, or even 1000s of computers and servers! This is an oversimplified summary of the entire process, but it gives you an idea of what can be involved.
Regardless, when working a case in incident response, you want to identify as many investigative leads as possible. Investigative leads are often referred to as IOCs, or indicators of compromise. To apply another law enforcement metaphor, the IOC is the suspect description for things that are evil on a victim’s system or network. Instead of “white male, 6-foot tall, slender build, blue jeans, white hat, etc.”, you might have a situation where the threat actor leveraged things like batch files titled “1.bat” which disables antivirus and “2.bat” which executes C:\Users\Public\Music\abc.exe which serves as a ransomware binary to encrypt all files on a computer.
These filenames, folder paths, and techniques, tactics, and procedures (TTPs) are IOCs for a respective case. IOCs are what you can use to pivot on and find more IOCs that’ll further flesh out the overall narrative of what a threat actor did (or didn’t do). IOCs are developed by understanding what’s normal and what’s not on a system, as well as reviewing all the artifacts on a system during and around the time of reported and observed evil on a victim’s network.
Also, triage is the name of the game, and if you must carve in free space for IOCs, you likely need to reconsider what’s important in your investigation. Usually, I and others on our team will start grabbing data remotely from the client’s system or network. We pull logs using a variety of tools but ultimately rely on KAPE (see below) for at-scale remote collection and parsing of the data we pull. I also work with other resources here at Kroll, including but not limited to our lab, malware analysis team, threat intelligence teams, and telemetry teams to acquire the most data and develop the most complete picture of what happened, when it happened, and how it happened.
Naturally, it will all depend on the type of case you’re assigned. If it’s a deadbox forensics case, there’s no telemetry involved. In those cases, you’re typically provided with forensic image(s) that you’ll need to conduct a deep dive forensic analysis into in order to defensibly answer the question(s) posed to us by the client based on the forensic artifacts. If you’re working on a business email compromise (BEC) case, artifacts on disk are likely not going to be as relevant. Instead, we’ll be interested in the various logs provided within M365 environments or on-premises Exchange Servers. For a traditional IR case, the client networks we work with can contain thousands of endpoints – this is where triage saves the day and provides us with the most efficient way to uncover the most answers from each endpoint.
Q7: What tools are commonly used (or do you use) in incident response?
A7: At Kroll, we very commonly use KAPE, a terrific tool developed in-house here at Kroll and the other tools by Eric Zimmerman, which KAPE ships with and leverages with most common workflows. Frankly, KAPE, Arsenal Image Mounter, a tool to view CSVs (Excel, Timeline Explorer, Modern CSV, etc.), a feature-rich text editor (EditPad Pro, UltraEdit, Sublime Text, etc.), and a tool that allows you to GREP or search across disk (Everything and PowerGREP).
The flexibility you have to use your creativity to solve problems in engagements can be really rewarding and provide much fulfillment when your efforts pay off by being able to provide more thorough findings that reflect well on the efforts and abilities of our team.
Q8: Related to above, where can I find some practice files or challenges that are specific to incident response if I want to try practicing with those tools?
A8: AboutDFIR has a list of Challenges & CTFs that you can test your mettle on. Also, there’s a great Tool Testing page where you can find test images to test tools with. Additionally, I’ve put together a GitHub repository that contains output generated from KAPE for some well-known challenges that can be found on the internet. You can find that repository here.
Q9: I already have some experience in digital forensics through my law enforcement role, how would you explain the differences between digital forensics in incident response?
A9: In my personal experience with law enforcement cases, the main sources of artifacts were typically communications (SMS, iMessages, chats, etc.), images, videos, or browsing history. Very rarely did something critical to my case reside out of those main areas. It did happen, but 90% of the time this is where I made my cases. In incident response, the evidence types are very broad. As I said earlier, since most of my cases involve Windows operating systems, I am going to be focused on the common Windows artifacts.
This includes knowing what’s normal and what’s not on a Windows system. For example, with LOLBAS (living off the land binaries and scripts) serving as standard tradecraft for most ransomware gangs, you have to be able to notice and act upon a binary on the system called “scvhost.exe”. For one, you need to know that “svchost.exe” is the correct spelling of that binary and scvhost.exe is just trying to masquerade as svchost.exe. Knowing when svchost.exe is in a potentially weird folder path is a critical skill. That binary should only exist where Microsoft puts it. If it’s not in one of those spots, it’s likely bad, and you need to act on it, report it, and pivot on your findings to find more evil.
Q10: The biggest difference between law enforcement DF and IR seems to be that in LE, we typically know exactly what we’re looking for before we go looking but, in IR, the methods of intrusion are so varied that you can realistically be looking for anything. How else might you explain the differences?
A10: I think that’s a pretty good way of putting it. When I was in law enforcement, I knew that the phone I was receiving was related to a homicide where the suspect texted the victim leading up to the incident, or a robbery where the suspects were texting each other prior to the incident, or a child exploitation case where you know images and videos are going to be key. I pretty much knew where I was going to find most of what I needed to prove a case one way or another.
In IR, often the most information we have to go off of at the beginning is whatever the client tells us. Sometimes, let’s say if it’s a ransomware case, then we might have other clues, like the ransom note. Once we identify the type of ransomware, then we have a basic idea of what the typical modus operandi is for that particular strain of ransomware and we can use that as a potential lead to try and find evidence of what happened when. I say potential lead because just because it’s X ransomware group doesn’t mean you’re going to find the exact same thing every single time. It’s not that cookie cutter, but it’s a good starting point.
Q11: Do you find that, even though you’re not running and gunning any more, IR provides some of the same cyber challenges you faced in law enforcement?
A11: I feel like I am running and gunning much more than I was before in law enforcement. In general, I found that law enforcement and government cases moved at a much slower pace than incident response. IR investigations usually last two to six weeks, depending on many variables. That’s from zero to submitting the report. Maybe a law enforcement case took the same or less amount of time, but the difference in IR is often you’re triaging an entire network of 10 to 1000+ systems. Considering how much ground you cover in that amount of time; you move a lot quicker.
It really is a completely different world, and I and other coworkers of mine who come from law enforcement have had to go through the mindset shift that takes a few months to sink in. Cases move a lot faster in IR considering the larger scope of many of them plus the fact we are often juggling multiple cases at once. When I worked for the federal government for a year, I was working cases from three to eight years prior. IR cases tend to move faster because everyone involved in an IR case is usually under a lot of pressure to get the answers they need to move on to the next step, be it getting operations back online, meeting payroll, restoring services to customers, attempting to recoup money lost in a business email compromise, or meeting regulatory requirements, just to name a very, very few of those pressures.
