When testing any control, regardless of type, there will ultimately be four types of results: true or false positives and true or false negatives.
If we look at those results from a Blue Team or defensive perspective, they will look a little like this:
True Positive: A legitimate attack that triggers an alarm. An alarm triggers and you investigate the alert and find out that somebody (or something) was trying to break into one of your systems via the method that triggered the alert – for example, known malware was executed.
False Positive: This could be when there was no malware dropped to disk, and the defensive product incorrectly raised an alert. Microsoft Defender is particularly sensitive to files that contain "Invoke-Mimikatz" for example.
False Negative: No alarm is raised when an attack has occurred. This can happen with newly discovered techniques not recognised by the defensive product, or an evasion technique was successfully applied.
True Negative: An event when no attack has occurred, and no detection is made. No attack occurred, and no alerts were raised.
The problem outcomes are the false outcomes. There is a balancing act that is determined by an organisation's appetite risk. Some companies who have a low-risk appetite are happy to get more false positives and triage them, with the associated workforce costs involved in reviewing those alerts to minimise the number of false negatives. Companies that don't have the workforce or resources to respond to a barrage of alerts may dial down the detection sensitivity to focus their response efforts on alerts that have a much higher level of severity/confidence.
False negatives are the worst of all the outcomes as it means that a control has been breached and no alert has been raised - leaving the target completely unaware that something is amiss. The inability for any (despite what the nice people in sales and marketing may want you to believe) defensive product to have 100% coverage is a primary driver of the defensive-in-depth philosophy and is easily explained using Swiss Cheese:
Part of the work we do at Cybaverse involves investigating ways to bypass these defensive products for our advanced offensive operations. Trying to generate false negatives also gives us an interesting view into the world of false positives. An example of the false-negative we engineered is shown below - executing Mimikatz (a commonly used malicious tool that is a password-stealing and manipulation tool) on a host protected by CrowdStrike with no alert being raised.
A false negative, as shown above, without adequate defence-in-depth, could compromise the target organisation.
But what about false positives? Why should we be concerned about these, and when do we cover Ninjas?
Let's look at the following alert:
Well, that does not look good! High severity, the process was killed, malicious module and a tactic & technique known as execution via shared modules.
Something that the Blue Team would want to take a closer look at.
The generated alert maps the technique ID to the MITRE ATT&CK framework, which (https://attack.mitre.org/techniques/T1129/) has this to say
Adversaries may execute malicious payloadsvia loading shared modules. The Windows module loader can be instructed to loadDLLs from arbitrary local paths and arbitrary Universal Naming Convention (UNC)network paths. This functionality resides in NTDLL.dll and is part of theWindows Native API which is called from functions like CreateProcess, LoadLibrary, etc. of the Win32 API.[1]
The module loader can load DLLs:
· via specification of the (fully-qualified or relative) DLLpathname in the IMPORT directory;
· via EXPORT forwarded to another DLL, specified with(fully-qualified or relative) pathname (but without extension);
· via an NTFS junction or symlink program.exe.local with thefully-qualified or relative pathname of a directory containing the DLLsspecified in the IMPORT directory or forwarded EXPORTs;
· via <filename="filename.extension" loadFrom="fully-qualified or relativepathname"> in an embedded or external "applicationmanifest". The file name refers to an entry in the IMPORT directory or aforwarded EXPORT.
It also contains the following detection recommendations:
So far, so good. It looks like we have an executable being run that may use this functionality to execute arbitrary payloads on a victim system. For example, malware may execute shared modules to load additional components or features.
The SOC (Security Operation Centre) would want to look at this and determine who is attempting to run malicious software on the endpoint.
But - this is a false positive and is why you need Ninjas. Below is the code (as the author of this code, I use the word code very lightly, and for any programmers out there reading this, please do not send me hate mail about my coding style, I like to think of it as artisanal code!) that generates that false positive:
Which compiles to (and is much more pleasing on the eye for anyone who does code):
Ultimately, it does not load any other (non-Microsoft) DLLs and does not contain any of the standard Windows APIs for loading DLLs. In fact, it does nothing other than pick a substantial number and then find the next prime number after that:
There is no malicious code here and nothing that should trigger any of the reported indicators. No shared modules. No execution of any other code. Nothing. It is utterly benign.
Why did we get an alert, and why were the indicators so far off from what the code does? Let's just say machine learning is not a solved problem for malware analysis/identification now.
This is where the SOC Ninjas come in. Companies that have the risk slider all the way over to High severity alerts and above, are typically those that pass the SOC function over to IT. When a high severity alert is generated, a simple playbook may be followed where perhaps the host is quarantined, potentially reimaged, and then redeployed – anywhere from a couple of hours upwards depending on the current workload on the response team. This can cause a substantial loss of productivity for the affected user - exponentially more so when a similar false positive is generated across multiple endpoints (in the case that a widely deployed application suddenly is flagged).
Having an experienced analyst responding to the alert means the file can be quickly disassembled or run in a sandbox. In addition, other Blue Team witchcraft can be applied, analysing, and identifying it as a false positive which can result in a machine being back online and operational in much shorter periods.
There have been many instances where an antivirus signature update has resulted in false positives being reported across an organisation (Microsoft Defender for Endpoint had this issue in March, https://www.itpro.co.uk/operating-systems/microsoft-windows/367018/microsoft-defender-false-positive-ransomware-alerts.
Worse still, sometimes these false positives incorrectly identify a benign file that is critical to the operation and either block access to or delete the file in question which breaks functionality (Sophos last year).
With the responsibilities of resolving these issues in smaller organisations falling to non-dedicated personnel, normally with a full workload already, sometimes the default playbook response can be detrimental. For example, in the event of a false positive alerting on benign software (Immunet AV flagged Chrome.exe as malicious this year), the playbook outlined above will not help - once reimaged, the benign software will still be there, triggering another alert.
An experienced analyst will be able to determine the root cause, correlating those indicators across known tactics, techniques, and procedures (TTPs) as well as determining if they are in fact valid. They can also advise on how to adjust the protection controls accordingly and help prevent a similar alert occurring again. Trying to respond to multiple simultaneous alerts across an organisation without dedicated response personnel can also mean that "business as usual" IT operations take a back seat for a while.
Most importantly, any extra time taken to respond to a false positive is time that is being taken away from investigating a true positive that may have negative implications on your business' security.
• It is difficult to determine where the false positives vs false negatives slider should be - there is always a trade-off.
• No single defensive solution is 100.
• A single line of defence is a Bad Idea™.
• Multiple defensive solutions, even if they are only each 80% effective, are good (Swiss Cheese).
• False positives cost time and money - if you do not have dedicated defensive personnel who can correctly triage alerts, downtime can also be longer.
• A response playbook that is rigid and inflexible can cause unnecessary loss of productivity and may fail to resolve the initial issue.
• A well-trained, experienced SOC analyst is indistinguishable from an actual ninja (and will save you time and money) – show them some love!
If you don't have dedicated personnel - perhaps consider outsourcing this requirement to a company with experience in this field. Why not take a look at our services to see how we can support you and your business?
If you do – let them know they are important and that the work they carry out is appreciated. They are a vital part of your organisation whose skills and talents often go overlooked.