Adversarial Machine Learning: A Looming Threat to AI Security

Machine learning is great for niched-down problems that require an artificial intelligence (AI) algorithm to pay attention to very specific details. For that reason, it’s becoming a very popular technology. But machine learning applications used in essential fields like healthcare, transportation, and finance carry a lot of responsibility. If one of these algorithms were to become adversarial through a hack or a malicious developer, they could cause a lot of damage to people, companies, and the public’s trust in technology.

A research collaboration between 13 organizations focuses on finding ways to compromise machine learning algorithms. One of the biggest outcomes of this partnership is a framework called Adversarial ML Threat Matrix (AMLTM) to detect and respond to various types of adversarial attacks against machine learning algorithms. The researchers have shown that the threat to machine learning is real and here to stay and AI systems need to be secured immediately.

Protecting Machine Learning

The AMLTM follows the convention and layout of ATT&CK, which is a tried-and-true framework developed by MITRE, one of the companies in the collaboration. ATT&CK was created to handle security threats in enterprise networks. Similar to the AMLTM table, ATT&CK also utilizes a matrix that lists various adversarial tactics and what malicious actors typically do in each tactic. This helps cybersecurity experts and threat analysts find patterns and warning signs of potential attacks.

The ATT&CK table is well-known in the cybersecurity industry, so it made sense for the AMLTM to mimic the layout of the popular table, making it easier to understand and accessible for both cybersecurity engineers and machine learning engineers. Each column represents a tactic, while each cell contains specific techniques to look for. Pin-Yu Chen is an AI researcher at IBM, another company that joined the collaboration. He describes the matrix as one that “bridges the gap by offering a holistic view of security in emerging ML-based systems, as well as illustrating their causes from traditional means and new risks induce by ML.”

Chen says that machine learning is going to become a mainstay as it expands into other industries that are experiencing a digital transformation. These machine learning applications could even include offering high-stakes decision-making. In fact, he says, “The notion of ‘system’ has evolved and become more complicated with the adoption of machine learning and deep learning.” For companies that change from a transparent rule-based system to a black-boxed machine learning- or AI-enhanced system, the new “smarter” system would be at a considerably higher risk of attack and infiltration.

Complexities of Securing Machine Learning

With every new emerging technology comes a unique set of security and privacy problems. Web apps with a database backend created SQL injection threats. Improved JavaScript for websites’ frontend and backend created cross-site scripting threats. The Internet of Things created botnet threats (like the Mirai botnet) and proliferated the strength of DDoS attacks. Mobile phones introduced the threat of spying without permission. Although we have developed a host of protective measures, built-in security protocols, and ongoing research groups for these threats, it takes a lot of time, testing, and loss of revenue to create a robust cybersecurity solution.

For machine learning algorithms, there are vulnerabilities that are embedded within the thousands or millions of parameters of deep neural networks. It’s outside the scope of today’s security tools, and the vulnerabilities are extremely difficult to find manually. Chen agrees that machine learning is so new that current software security doesn’t fit the bill yet, but he adds, adding machine learning into today’s security protocols and landscape helps us develop new insights and improve risk assessment.

The AMLTM doesn’t skip a beat: it comes with case studies that involve adversarial machine learning, traditional security vulnerabilities, and combinations of both. It shows that adversarial attacks on machine learning systems aren’t just limited to the testing phase of an algorithm; they’re found in live systems as well. This really hits home that any machine learning system is vulnerable to malicious attacks, which raises the seriousness of the problem for all developers and engineers involved in the development, testing, and implementation of each system.

In one case study, the Seattle-based Microsoft Azure security team (another company in the AMLTM partnership) researched and consolidated information about a machine learning model. Then, they got access to the model using a valid account on the server. Using the information they’d found, they were able to detect adversarial vulnerabilities and create attacks against the model. Using these case studies, the research group hopes that security vendors will create new tools to secure and find weaknesses within machine learning systems in the future. Chen says, “Security is only as strong as its weakest link.”

Watching for Adversarial Threats

Without the AMLTM table, machine learning and AI developers and companies using these emerging technologies were creating algorithms blindly and without enough security. But the new matrix should give engineers the power to enhance their system’s security.

Chen says that he wants machine learning engineers to not only test-drive their algorithms but also perform crash-dummy collision tests to bring out the most vulnerable parts of the algorithm’s design. His ultimate hope for the AMLTM table is that “the model developers and machine learning researchers can pay more attention to the security (robustness) aspect of the model and looking beyond a single performance metric such as accuracy.”