{"id":9030,"date":"2018-08-09T20:00:05","date_gmt":"2018-08-09T20:00:05","guid":{"rendered":"https:\/\/cloudblogs.microsoft.com\/microsoftsecure\/?p=84787"},"modified":"2018-08-09T20:00:05","modified_gmt":"2018-08-09T20:00:05","slug":"protecting-the-protector-hardening-machine-learning-defenses-against-adversarial-attacks","status":"publish","type":"post","link":"https:\/\/www.threatshub.org\/blog\/protecting-the-protector-hardening-machine-learning-defenses-against-adversarial-attacks\/","title":{"rendered":"Protecting the protector: Hardening machine learning defenses against adversarial attacks"},"content":{"rendered":"

Harnessing the power of machine learning and artificial intelligence has enabled Windows Defender Advanced Threat Protection (Windows Defender ATP<\/a>) next-generation protection<\/a> to stop new malware attacks before they can get started \u2013 often within milliseconds. These predictive technologies are central to scaling protection and delivering effective threat prevention in the face of unrelenting attacker activity.<\/p>\n

Consider this: On a recent typical day, 2.6 million people encountered newly discovered malware in 232 different countries (Figure 1). These attacks were comprised of 1.7 million distinct, first-seen malware and 60% of these campaigns were finished within the hour.<\/p>\n

\"\"<\/em><\/p>\n

Figure 1. A single day of malware attacks: 2.6M people from 232 countries encountering malware<\/em><\/p>\n

While intelligent, cloud-based approaches represent a sea change in the fight against malware, attackers are not sitting idly by and letting advanced ML and AI systems eat their Bitcoin-funded lunch. If they can find a way to defeat machine learning models at the heart of next-gen AV solutions, even for a moment, they\u2019ll gain the breathing room to launch a successful campaign.<\/p>\n

Today at Black Hat USA 2018, in our talk \u201cProtecting the Protector: Hardening Machine Learning Defenses Against Adversarial Attacks<\/a>\u201d, we presented a series of lessons learned from our experience investigating attackers attempting to defeat our ML and AI protections. We share these lessons in this blog post; we use a case study to demonstrate how these same lessons have hardened Microsoft\u2019s defensive solutions in the real world. We hope these lessons will help provide defensive strategies on deploying ML in the fight against emerging threats.<\/p>\n

Lesson: Use a multi-layered approach<\/h2>\n

In our layered ML approach<\/a>, defeating one layer does not mean evading detection, as there are still opportunities to detect the attack at the next layer, albeit with an increase in time to detect. To prevent detection of first-seen malware, an attacker would need to find a way to defeat each of the first three layers in our ML-based protection stack.<\/p>\n

\"\"Figure 2. Layered ML protection<\/em><\/p>\n

Even if the first three layers were circumvented, leading to \u201cpatient zero\u201d being infected by the malware, the next layers can still uncover the threat and start protecting other users as soon as these layers reach a malware verdict.<\/p>\n

Lesson: Leverage the power of the cloud<\/h2>\n

ML models trained on the backend and shipped to the client are the first (and fastest) layer in our ML-based stack. They come with some drawbacks, not least of which is that an attacker can take the model and apply pressure until it gives up its secrets. This is a very old trick in the malware author\u2019s playbook: iteratively tweak prospective threats and keep scanning it until it\u2019s no longer detected, then unleash it.<\/p>\n

\"\"<\/em><\/p>\n

Figure 3. Client vs. cloud models<\/em><\/p>\n

With models hosted in the cloud, it becomes more challenging to brute-force the model. Because the only way to understand what the models may be doing is to keep sending requests to the cloud protection system, such attempts to game the system are \u201cout in the open\u201d and can be detected and mitigated in the cloud.<\/p>\n

Lesson: Use a diverse set of models<\/h2>\n

In addition to having multiple layers of ML-based protection, within each layer we run numerous individual ML models trained to recognize new and emerging threats. Each model has its own focus, or \u201carea of expertise.\u201d Some may focus on a specific file type (for example, PE files, VBA macros, JavaScript, etc.) while others may focus on attributes of a potential threat (for example, behavioral signals, fuzzy hash\/distance to known malware, etc.). Different models use different ML algorithms and train on their own unique set of features.<\/p>\n

\"\"Figure 4. Diversity of machine learning models<\/em><\/p>\n

Each stand-alone model gives its own independent verdict about the likelihood that a potential threat is malware. The diversity, in addition to providing a robust and multi-faceted look at potential threats, offers stronger protection against attackers finding some underlying weakness in any single algorithm or feature set.<\/p>\n

Lesson: Use stacked ensemble models<\/h2>\n

Another effective approach we\u2019ve found to add resilience against adversarial attacks is to use ensemble models. While individual models provide a prediction scoped to a particular area of expertise, we can treat those individual predictions as features to additional \u201censemble\u201d machine learning models, combining the results from our diverse set of \u201cbase classifiers\u201d to create even stronger predictions that are more resilient to attacks.<\/p>\n

In particular, we\u2019ve found that logistic stacking, where we include the individual probability scores from each \u201cbase classifier\u201d in the ensemble feature set provides increased effectiveness of malware prediction.<\/p>\n

\"\"Figure 5. Ensemble machine learning model with individual model probabilities as feature inputs<\/em><\/p>\n

As discussed in detail in our Black Hat talk, experimental verification and real-world performance shows this approach helps us resist adversarial attacks. In June, the ensemble models represented nearly 12% of our total malware blocks from cloud protection, which translates into tens of thousands of computers protected by these new models every day.<\/p>\n

\"\"Figure 6. Blocks by ensemble models vs. other cloud blocks<\/em><\/p>\n

Case study: Ensemble models vs. regional banking Trojan<\/h2>\n

\u201cThe idea of ensemble learning is to build a prediction model by combining the strengths of a collection of simpler base models.\u201d<\/em>
\u2014 Trevor Hastie, Robert Tibshirani, Jerome Friedman<\/em><\/p>\n

One of the key advantages of ensemble models is the ability to make a high-fidelity prediction from a series of lower-fidelity inputs. This can sometimes seem a little spooky and counter-intuitive to researchers, but uses cases we\u2019ve studied show this approach can catch malware that the singular models cannot. That\u2019s what happened in early June when a new banking trojan (detected by Windows Defender ATP as TrojanDownloader:VBS\/Bancos) targeting users in Brazil was unleashed.<\/p>\n

The attack<\/h3>\n

The attack started with spam e-mail sent to users in Brazil, directing them to download an important document with a name like \u201cDoc062108.zip\u201d inside of which was a \u201cdocument\u201d that is really a highly obfuscated .vbs script.<\/p>\n

\"\"Figure 7. Initial infection chain<\/em><\/p>\n

\"\"Figure 8. Obfuscated malicious .vbs script<\/em><\/p>\n

While the script contains several Base64-encoded Brazilian poems, its true purpose is to:<\/p>\n