{"id":40709,"date":"2021-05-03T16:00:52","date_gmt":"2021-05-03T16:00:52","guid":{"rendered":"https:\/\/www.microsoft.com\/security\/blog\/?p=93404"},"modified":"2021-05-03T16:00:52","modified_gmt":"2021-05-03T16:00:52","slug":"ai-security-risk-assessment-using-counterfit","status":"publish","type":"post","link":"https:\/\/www.threatshub.org\/blog\/ai-security-risk-assessment-using-counterfit\/","title":{"rendered":"AI security risk assessment using Counterfit"},"content":{"rendered":"

Today, we are releasing Counterfit<\/a>, an automation tool for security testing AI systems as an open-source project. Counterfit helps organizations conduct AI security risk assessments to ensure that the algorithms used in their businesses are robust, reliable, and trustworthy.<\/p>\n

AI systems are increasingly used in critical areas such as healthcare, finance, and defense. Consumers must have confidence that the AI systems powering these important domains are secure from adversarial manipulation. For instance, one of the recommendations from Gartner\u2019s Top 5 Priorities for Managing AI Risk Within Gartner\u2019s MOST Framework<\/a> published in Jan 20211<\/sup> is that organizations \u201cAdopt specific AI security measures against adversarial attacks to ensure resistance and resilience,\u201d noting that \u201cBy 2024, organizations that implement dedicated AI risk management controls will successfully avoid negative AI outcomes twice as often as those that do not.\u201d<\/p>\n

However, performing security assessments of production AI systems is nontrivial. Microsoft surveyed 28 organizations<\/a>, spanning Fortune 500 companies, governments, non-profits, and small and medium sized businesses (SMBs), to understand the current processes in place to secure AI systems. We found that 25 out of 28 businesses indicated they don\u2019t have the right tools in place to secure their AI systems and that security professionals are looking for specific guidance in this space.<\/p>\n

This tool was born out of our own need to assess Microsoft\u2019s AI systems for vulnerabilities with the goal of proactively securing AI services, in accordance with Microsoft\u2019s responsible AI principles<\/a> and Responsible AI Strategy in Engineering (RAISE) initiative. Counterfit started as a corpus of attack scripts written specifically to target individual AI models, and then morphed into a generic automation tool to attack multiple AI systems at scale.<\/p>\n

Today, we routinely use Counterfit as part of our AI red team operations. We have found it helpful to automate techniques in MITRE\u2019s Adversarial ML Threat Matrix<\/a> and replay them against Microsoft\u2019s own production AI services to proactively scan for AI-specific vulnerabilities. Counterfit is also being piloted in the AI development phase to catch vulnerabilities in AI systems before they hit production.<\/p>\n

To ensure that Counterfit addresses a broader set of security professionals\u2019 needs, we engaged with a diverse profile of partners spanning large organizations, SMBs, and governmental organizations to test the tool against their ML models in their environments.<\/p>\n

\n

\u201cAI is increasingly used in industry; it is vital to look ahead to securing this technology particularly to understand where feature space attacks can be realized in the problem space. The release of open-source tools from an organization such as Microsoft for security practitioners to evaluate the security of AI systems is both welcome and a clear indication that the industry is taking this problem seriously.\u201d<\/em><\/p>\n

\u2014<\/em>Matilda Rhode, Senior Cybersecurity Researcher, Airbus<\/p>\n<\/blockquote>\n

Three key ways Counterfit is flexible<\/h2>\n

As a result of internal and external engagements, Counterfit is flexible in three key ways:<\/p>\n

    \n
  1. Counterfit is environment agnostic\u2014it can help assess AI models hosted in any cloud environment, on-premises, or on the edge.<\/li>\n
  2. Counterfit is model agnostic\u2014the tool abstracts the internal workings of their AI models so that security professionals can focus on security assessment.<\/li>\n
  3. Counterfit strives to be data agnostic\u2014it works on AI models using text, images, or generic input.<\/li>\n<\/ol>\n

    Under the hood, Counterfit is a command-line tool that provides a generic automation layer for adversarial AI frameworks such as Adversarial Robustness Toolbox<\/a> and TextAttack<\/a>. Our tool makes published attack algorithms accessible to the security community and helps to provide an extensible interface from which to build, manage, and launch attacks on AI models.<\/p>\n

    Designed for security professionals<\/h2>\n

    Counterfit uses workflows and terminology similar to popular offensive tools that security professionals are already familiar with, such as Metasploit or PowerShell Empyre. Security professionals can benefit from the tool in the following ways:<\/p>\n