What is MLSecOps?

The machines are coming, but who watches the watchers?

MLSecOps is to MLOps what DevSecOps is to DevOps - perhaps the most intuitive analogy for any software and cloud practitioner.

However, the scope of MLSecOps - short for Machine Learning Security Operations- is far more cross-disciplinary than DevSecOps and, more importantly, its implications and risks are profoundly more consequential: in this article I will explain why I am convinced that’s the case and why you should pay close attention to the burgeoning field of AI Safety.

But first, let’s get it over with the definitions: MLOps (Machine Learning Operations) is the engineering discipline that manages the end-to-end lifecycle of a machine learning model - while MLSecOps is the field that attempts to secure each stage of the ML lifecycle.

The final cybersecurity frontier

There's considerable buzz about AI as a transformative tool enhancing every segment of the security tooling ecosystem - and, occasionally, inching closer to end-to-end automation. However, there is not quite as much focus on AI itself as a critical attack surface.

The above statement is a post I shared not too long ago on a LinkedIn.

In hindsight, I realize my claim was rather conservative: AI is not just a critical attack surface, but it’s THE attack surface of the coming decades in the tech industry.

Lately, the AI security landscape evolved at breakneck speed. New threats targeting Artificial Intelligence systems appear daily, and as a result the MLSecOps security tooling ecosystem has grown accordingly. These tools are designed to protect against the widening attack surface outlined, for instance, in the OWASP Top 10 ML attack vectors and OWASP Top 10 LLM security risks.

At present, not only do ML security threats outpace the available tools to mitigate them, but there's also a stark imbalance between the rapid, widespread adoption of AI technologies across industries and the limited engineering resources allocated to ensure their proper security posture.

Vitruvian Man meets AI agent: Leonardo Da Vinci according to DALL-E 3 - OpenAI text-to-image model.

This brought us back to our initial claim: AI as the defining attack surface in our digital era - driven by a mixture of temporary weaknesses and fundamental technical reasons.

Temporary Security Weaknesses in the AI Industry

The current state of Artificial Intelligence security is impacted by a lack of ML security culture, standardized best practices and talent shortages.

The current shortage of security professionals with machine learning skills is a challenge that is likely to persist for the foreseeable future. Bridging the gap between InfoSec and the Machine Learning fields will require a long upskilling cycle, as these two industries have historically diverged until recently.

The safety shortcomings in this section are socio-economic rather than technical - as such, their impact will diminish over time as the AI industry adapts itself to the challenges that come with technological mass adoption.

While it’s true that the machine learning industry is lagging behind the rest of the software and cloud industry in terms of security, we are witnessing a more decisive shift towards the development of robust enterprise-grade AI safety frameworks.

Some notable private-sector examples are Google’s Secure AI Framework (SAIF) and IBM’s own internal AI security framework.

On the governmental and institutional front, the National Institute of Standards and Technology (NIST) has introduced the AI Risk Management Framework (AI RMF) and the AI RMF Playbook - a series of documents that map the organizational risks of AI adoption and the corresponding enterprise processes to reduce their threat profiles.

Although we are still far from achieving well-adopted industry standards, these initiatives effectively set the first successful quantitative and qualitative benchmarks to measure a company’s AI security posture.

Analogously, the industry has yet to develop a more mature security tooling ecosystem and a consensus on the best mitigation strategies to the the most exotic adversarial risks, but it is beginning to converge towards some general solutions against the most common attack vectors.

Some of these emergent best practices focus on restricting AI agents’ access to internal APIs only to what is strictly necessary via guardrails, enforcing sensible access-control policies and tracking the provenance of ML models.

Fundamental Technical Challenges

A Schrodinger’s cat black-box - courtesy of DALL-E 3

Complexity

AI systems are the most complex software components ever designed. The complexity of their architectures lies as much in the mathematical design of their models as it does in their infrastructure challenges.

Explainability - Or Lack Thereof

This inherent complexity is further amplified by the lack of explainability of Artificial Intelligence systems.

Explainability is the ability to model an AI agent's decision-making process by establishing causal or logical relationships between its inputs and outputs. While we can explain - debug - traditional software, AI systems largely operate as black boxes.

The opaque nature of AI has been a key focus of academic and industry research aimed at making their inner workings more human-digestible. Today, the field of Explainable AI (XAI) offers a range of increasingly sophisticated strategies to enhance the interpretability of AI outputs.

However, ML models are still far from being fully explainable and there’s a trade-off between a model’s performance and its interpretability.

This partial lack of transparency raises trust concerns for businesses and complicates the task of ensuring the security and regulatory compliance of ML models.

High-Value Targets

Machine Learning models are increasingly often the most valuable component of a software stack.

The main reasons are:

• The high capital requirements necessary to cover the costs of the engineering talent and the infrastructure where the ML models are trained and run.

• The intellectual property value of the datasets used to train the ML models - which often are highly confidential data such as financial or healthcare information.

• The increasingly critical function that AI components play in the application architecture and its business logic.

These properties make ML models compelling targets to attackers.

Imagine a scenario where a large tech company deployed a gargantuan R&D budget for the development of a new machine learning model. Then, once in production, an attacker is able to reverse-engineer an approximate replica of the company’s ML model.

This attack vector is called model theft. It is performed against Machine Learning models that expose an unprotected public-facing API.

High-level walk-through of a model theft exploit against a public-facing ML Model API

Attacker Scenario

Let's say there are two leading skincare apps in the digital cosmetic dermatology industry.

Company A recently released the latest version of its ML-powered skincare recommendation system, providing users with highly personalized skincare routines and product suggestions. This new feature has significantly boosted Company A's user growth and retention, far surpassing its competitor Company B. To further consolidate its new market lead, Company A opens up its ML model via a public API, allowing other skincare e-commerce shops to integrate the recommendation engine into their applications.

Its competitor Company B is nowhere near developing a similarly sophisticated machine learning model, so it decides to reverse-engineer Company A’s valuable AI intellectual property.

Essentially, the model theft exploit simplified in the above diagram boils down to the three main phases:

- After obtaining access to the A's skincare recommendation API, Company B designs several user persona templates which it uses for the automated generation of API queries with a wide range of user profiles such as different dermatological concerns (i.e.: aging, eczema, acne etc.), lifestyle factors and skin types. This systematic API probing is designed to model the recommendation system's ranking function.

- Company B iteratively starts pairing the prompt inputs and ML model's outputs & optimizing its reconnaissance queries. The API inputs selection strategy is designed to progressively refine the attacker’s payloads for the purpose of maximizing the knowledge gained on the underlying target model’s behavior.

- Eventually, Company B collects enough input-output pairs to approximate the output distribution of Company A’s skincare recommendation engine. It then uses this dataset to train or fine-tune its own replica model. As a result, Company B’s recommendation system is able to mimic quite closely the decision-making capabilities of Company A’s proprietary ML model - although never with absolute fidelity.

What’s the outcome? The digital skincare Company A partially lost its competitive advantage - the proprietary nature of its recommendation engine's ML model - into which it had invested substantial engineering and infrastructure resources. For Company B, on the other hand, the attack was relatively inexpensive.

What makes model theft even more concerning is, in fact, that it’s a relatively cost-effective attack compared to the financial damage that it can inflict on its targets.

Luckily, this AI attack is still mostly theoretical as its real-world exploit implementation are rare and usually not as effective. Nevertheless, in the near future we can expect more exploit attempts bridging the gap between academic theory and real-world model stealing attacks: we will go through their main mitigation strategies in a future article.

Model stealing is just a relatively uncommon example within the growing taxonomy of AI security threats: there are many other AI attack vectors that are far more production-ready and pose a similarly devastating intellectual property loss and reputational damage to their targets.

Verdict

In conclusion, ML models are often the most capital intensive, the most valuable and the most obscure component in modern software architectures. At the same time, they are also one of the areas that is the most neglected from a cybersecurity perspective.

Securing them is, therefore, the most daunting challenge in the cybersecurity industry - as well as the most impactful.

Would you like us to cover a specific AI Security topic? Let us know in the comments, on Twitter or Linkedin .

If you are interested in knowing more about AI security tools and resources, check out the awesome-MLSecOps repository - contributions are welcome!

Lastly, share this article with your network & subscribe to the newsletter.

Help us build the AI Security culture of the future.