Steps for system administrators and technical staff who believe they have discovered an information security incident - a compromised computer or unauthorized access to digital data.
On this page:
Discovering a compromised host can be stressful but first and foremost, remember not to panic: The Information Security (IS) division is here to help guide you through the incident response process and determine whether sensitive data was accessed. In order to do so, we will need you to take some important first steps.
Throughout the entirety of this process, remember to take notes. Write down everything you observed and what you did, with timestamps. Here are some examples of what to record:
If you interacted with affected devices in any way, it may be critical for us to know what you did and when. The reason is that we’re frequently asked to determine with high confidence what an adversary did and what data they touched. It can sometimes be difficult or impossible to distinguish between your actions and the adversary’s, especially if you have interacted with the device since discovery of the incident.
It is equally important to document only information that has been substantiated. No speculation or inference should be included in your notes; only actions and observations.
We understand that circumstances vary and taking the following steps may not always be possible. These are best practices; if you need to modify them, remember to make a note of what you have done.
Disconnecting the device from the network will help prevent further spread of any malware or malicious activity from a compromised device. Remember that if your device has both wired and wireless capabilities, both should be disabled.
If the host is part of a critical service, it will be up to you and the service owner to determine whether or not it can be taken offline. If malware is involved, it is important to isolate the device as quickly as possible in order to prevent other systems from being infected.
Do not make any changes to the state of the computer such as powering it down, logging off, or otherwise using the computer. Turning the power off could destroy key pieces of forensic evidence that will later be needed to help determine root cause and the full extent of the compromise.
If we have to do a forensic analysis, it might be really important to know, for example, that the last time a file was accessed was before the initial compromise occurred. If you’ve copied, moved, restored, deleted, or otherwise touched that data, then we may not know whether it was you or the adversary that did the touching, and that reduces our confidence that it wasn’t compromised.
It’s nearly universal that, at the very least, we’re going to need to examine logs from every system that might be affected, but there are caveats.
In the best case, you’re sending all the right logs to a central server, so you can easily preserve them without having to even log into the affected systems. It is always safe to do this, and this is one of the few things we’d encourage you do to even before you talk with us.
If you don’t have central logging, then it’s a little trickier. You’ll have to collect logs from individual hosts, and this may be less reliable, not only because some actors will delete or modify logs, but also because it risks trampling evidence if we end up doing a forensic analysis. That still might be a good trade off, but it’s not always clear cut. For those reasons, if you don’t have central logs, it’s best to contact us before preserving them directly from affected hosts.
Also, note that “affected hosts” doesn’t just mean the one that is compromised; it means any host on which there may be evidence. For example, a lot of workstations connect to file servers, and a compromised workstation may have accessed files on the file server, even if the file server itself is not compromised. If your logging on the file server is sufficiently verbose, that may be where we find our best evidence. In extreme cases, we may even be able to answer the question “was this data compromised?” by doing disk forensics on the file server, even if it was not compromised, and that kind of evidence is easily trampled by disk accesses.
Finally, if you’re not very careful how you interact with untrusted computers, it may be trivial for an attacker who has compromised a computer to steal the credentials of the first administrator who logs in to investigate. Don’t do it! (And if you already did it, change your password immediately.) Sometimes, in fact, this is precisely what the attacker wants and expects; they may compromise a workstation, then just wait for a domain administrator to log in, and steal their credentials to gain new privileges. You don’t want that to happen.
If you have already shut down or disconnected computers or taken other remedial actions, that’s ok; just leave them as they are and do nothing else to them. Most importantly, record exactly what you did and exactly what time, as best you can remember. (See the previous section.)
To recap: talk to us before doing anything. However, one exception to the “don’t touch anything” rule is that it’s nearly always ok and desirable to preserve logs from a central logging server, as soon as you think there might be a compromise (unless, of course, the central logging server itself is believed to be compromised).
Depending on the type of data involved in the incident, there may be one or more delegated authorities to whom the incident needs to be reported. IS can help assist you in determining which delegated authority you need to work with or you can review Administrative Policy Statement 2.5: Information Security and Privacy Incident Reporting and Management Policy.
Even if IS is not the delegated authority for the type of information involved in the incident, we may still be involved from a forensics perspective. We would ideally like to get a technical person from your group in contact with an analyst in our office as quickly as possible. Delay can affect our ability to understand what happened and respond effectively.
We also have threat intelligence analysts who have insight into current attack trends which can help guide our response and help us know what to look for. Again, the best course of action may be situational and fluid, so getting technical heads together as quickly as possible helps a lot.
When reporting an incident, we encourage the use of the telephone over email to convey sensitive information. Also, limit information sharing to only those individuals directly involved in the incident management process.
First and most importantly, attribution (the process of determining who is responsible for an attack) is not nearly as easy or straightforward as some people believe. Attackers can and do take a number of measures to implicate others in a crime, either to avoid accountability or to instigate conflict with their own adversaries.
Any contact with, or action against, the attacker must be left to law enforcement. For any UW employee or representative to do so may not only be counterproductive by increasing risks of all types, but may in some cases be a crime. It is absolutely critical to not do this.
Forensic analysis itself involves risk. Evidence is easily corrupted if not handled correctly, and incorrect or premature conclusions may have unforeseen and damaging consequences. It is critical to understand the process of preserving evidence and correctly scoping and conducting an investigation, and what will be done with the results of the investigation.
If you have staff trained in forensics who understand the process and risks, it may be fine for them to assist or conduct additional analysis after consultation with us; however, this should never be attempted at the outset. Preserving evidence is time-critical, but analysis can usually wait until enough facts are understood to scope the work appropriately.
In many cases, we are dealing not with black-and-white answers, but with degrees of confidence. Evidence is the key to increasing the confidence level of our findings, and that usually means:
After an incident, while we’re in reactive mode, the best way to minimize the impact is to collect quality evidence that can help us determine with high confidence what did and did not happen.
We hope these few steps can help further that goal.