Unlike the previous examples, this article focuses on a familiar attack vector: the execution of untrusted code. This long-standing issue has plagued application security for decades, and the AI field is no exception. In fact, the problem of dealing with untrusted code is just as relevant in AI as it is in other areas of cybersecurity.

A vulnerable Flask API that feeds unsanitized user input to the infamous Python `eval`:

One of the core principles in InfoSec is to never run untrusted code without first thoroughly validating and sanitizing its inputs, strictly limiting its execution environment through sandboxing or virtualization, and applying the principle of least privilege to minimize the blast radius in case of a compromise.

As we mentioned in our first article, the machine learning industry is at a less mature stage in terms of security culture than the cloud and application development ecosystems. This is partly because, until not too long ago, the AI field was partially decoupled from customer-facing commercial applications as a substantial portion of its practitioners were scholars.

Understanding Model Serialization Attacks

Model Serialization Attacks are one of the main security blind-spots in the ML industry. They are, essentially, a form of arbitrary code execution: the only novelty is that they target the MLOps lifecycle rather than Web API endpoints.

Serialization is the process of converting an object into a format that is more portable and that can be more easily stored and shared. Conversely, deserialization reconstructs the original artifact from its serialized format.

In the machine learning field, ML models are often serialized for storage and distribution and then deserialized before their production deployment. As complex and valuable digital artifacts, machine learning models move across the software supply chain and they are shared across many different systems, making them vulnerable to supply chain attacks.

Model Serialization Attacks are a form of supply chain attack.

When you deserialize an ML model, you are running untrusted code in your machine. Unsurprisingly, this is one of the main gateways that attackers use to hack the AI supply chain.

Platforms like Hugging Face - which enables users to share ML models and datasets- are especially exposed to this security risk: there have already been a few occurrences where MLSecOps researchers detected malicious models in public hubs.

Downloading one of these seemingly harmless ML models could potentially enable an attacker to spawn a shell in the victim’s machine.

More generally, compromising a machine learning model with remote code execution (RCE) can weaponize the AI supply chain into a distribution channel for malwares.

But what specific serialization weaknesses do RCE exploits leverage to compromise ML assets?

All the most popular ML development libraries - such as Tensorflow or PyTorch- come with built-in serialization utils, but only few are reasonably safe. The most widely used serialization library happens to be also the most unsafe: Pickle.

Pickle is THE Python serialization module in terms of popularity - image courtesy of DALL-E

In the Pickle library jargon, serialization and deserialization are referred to as, respectively, pickling and unpickling: when you pickle an object, you transform it into a byte stream with either .pkl or .pickle as file extensions - whereas unpickling reconstructs the binary file into its original format.

Some other ML serialization utils are built on top of Pickle, most notably PyTorch’ saving/loading model methods: PyTorch code has, in fact, already been found vulnerable of a few RCE exploits.

Exploiting Python Pickle

The main downside of Pickle is its lack of safety: it grants unrestricted execution to untrusted code, which makes it easily exploitable.

unsafe_pickle.py file in the damn-vulnerable-ML-serialization repository

In the above example, the attacker defines a MaliciousPayload class which, when deserialized, executes arbitrary shell commands on the victim's machine. The malicious instructions are embedded at the pickling stage, which are then triggered when pickle.dumps() deserializes the MaliciousPayload class instance.

A brief anatomy of the Pickle lifecycle

When the victim runs the unsafe_pickle.py file, the pickle.dump operation converts the MaliciousPayload instance into a byte-stream, which is then saved as the file damn_vuln_pickle_model.pkl. Furthermore, upon pickling, the `__reduce__()` method of the MaliciousPayload class is automatically invoked.

Let’s review the return signature of the `__reduce__()` method:

where:

• `callable_object` is a Python callable: namely, an object that can be called like a function.

• `tuple_args` is a tuple of arguments that will be passed to the `callable_object`

The reason why `__reduce__()` is such a security liability is because it serializes the callable_object and its arguments: - in our proof-of-concept, the callable_object is os_system and tuple_args is a crafted payload of bash commands.

high-level walk-through of the serialization exploit

Finally, during unpickling, the callable and tuple are reconstructed into an executable and its arguments.

This means that when the code reaches the pickle.loads() instruction, the exploit runs the malicious shell commands.

In a real-world scenario, this type of exploit could be used to mount a backdoor on the victim’s machine, exfiltrate sensitive data or conduct a poisoning attack.

In the case of Pickle vulnerabilities, the exploit is embedded during the serialization stage - however, there are other variants of the exploit which target a different stages of a model’s lifecycle.

Mitigation Strategies

The three key action items for an effective prevention strategy against serialization attacks and enforce secure ML practices are:

• Replacing Pickle with safer alternatives: Hugging Face, as one of the main targets of model serialization attacks, allocated its engineering resources to develop a much safer alternative to pickle called safetensors.

• Set up a model registry to have greater visibility on the data provenance of your ML artifacts: supply chain security is even more critical in AI than it is in application development - and just as code repositories have tools like Backstage for better inventory management, the machine learning world has MLFlow.

• Implement MLSecOps CI/CD in your model lifecycle: the two main security tools specialized in ML serialization attacks are ProtectAI’s ModelScan and Trail of Bits’ Fickling.

  The companies that developed the aforementioned scanners are already established in the MLSecOps space in general and, in particular, they are not new to model serialization exploits: ProtectAI unconvered a few vulnerabilities in the MLFlow platform, whereas Trail of Bits (ToB) audited Hugging Face’s safetensors.

  Both scanners are developer-friendly open-source Python packages that can be easily run from CLI, but they come with their own design trade-offs:

  • ModelScan is more of a blue team tool, which can be easily integrated with CI/CD in your MLOps lifecycle and helps your organization to identify dangerous serialization patterns. While it has less functionalities than Fickling, it supports more serialization utils and formats than the ToB’s scanner.

  • Fickling has a more red-team design, which includes a decompiler and the ability to perform code injection - as such, Fickling has a richer feature set than ModelScan, . On the downside, unlike the ProtectAI’s counterpart, ToB’s tool supports only Pickle or Pickle-based serialization utils - such as PyTorch.

  Let’s now have a look at how we can run them against the vulnerable unsafe_pickle.py’s output:

  • ModelScan CLI command and output:

  

  

  • Fickling CLI command ’s safety-check flag and output:

  

Bingo! Both scanners detected the security issue. If you want to check other vulnerable ML serialization examples with tensorflow and pytorch, then clone the damn-vulnerable-ML-serialization repository.

This is the end, my only friend

Time to wrap up our second MLSecOps article - in summary, understanding the risks associated with AI supply chain in general and model serialization in particular is critical to secure MLOps pipelines.

However, the scope of MLSecOps - short for Machine Learning Security Operations- is far more cross-disciplinary than DevSecOps and, more importantly, its implications and risks are profoundly more consequential: in this article I will explain why I am convinced that’s the case and why you should pay close attention to the burgeoning field of AI Safety.

But first, let’s get it over with the definitions: MLOps (Machine Learning Operations) is the engineering discipline that manages the end-to-end lifecycle of a machine learning model - while MLSecOps is the field that attempts to secure each stage of the ML lifecycle.

The final cybersecurity frontier

There's considerable buzz about AI as a transformative tool enhancing every segment of the security tooling ecosystem - and, occasionally, inching closer to end-to-end automation. However, there is not quite as much focus on AI itself as a critical attack surface.

The above statement is a post I shared not too long ago on a LinkedIn.

In hindsight, I realize my claim was rather conservative: AI is not just a critical attack surface, but it’s THE attack surface of the coming decades in the tech industry.

Lately, the AI security landscape evolved at breakneck speed. New threats targeting Artificial Intelligence systems appear daily, and as a result the MLSecOps security tooling ecosystem has grown accordingly. These tools are designed to protect against the widening attack surface outlined, for instance, in the OWASP Top 10 ML attack vectors and OWASP Top 10 LLM security risks.

At present, not only do ML security threats outpace the available tools to mitigate them, but there's also a stark imbalance between the rapid, widespread adoption of AI technologies across industries and the limited engineering resources allocated to ensure their proper security posture.

Vitruvian Man meets AI agent: Leonardo Da Vinci according to DALL-E 3 - OpenAI text-to-image model.

This brought us back to our initial claim: AI as the defining attack surface in our digital era - driven by a mixture of temporary weaknesses and fundamental technical reasons.

Temporary Security Weaknesses in the AI Industry

The current state of Artificial Intelligence security is impacted by a lack of ML security culture, standardized best practices and talent shortages.

The current shortage of security professionals with machine learning skills is a challenge that is likely to persist for the foreseeable future. Bridging the gap between InfoSec and the Machine Learning fields will require a long upskilling cycle, as these two industries have historically diverged until recently.

The safety shortcomings in this section are socio-economic rather than technical - as such, their impact will diminish over time as the AI industry adapts itself to the challenges that come with technological mass adoption.

While it’s true that the machine learning industry is lagging behind the rest of the software and cloud industry in terms of security, we are witnessing a more decisive shift towards the development of robust enterprise-grade AI safety frameworks.

Some notable private-sector examples are Google’s Secure AI Framework (SAIF) and IBM’s own internal AI security framework.

On the governmental and institutional front, the National Institute of Standards and Technology (NIST) has introduced the AI Risk Management Framework (AI RMF) and the AI RMF Playbook - a series of documents that map the organizational risks of AI adoption and the corresponding enterprise processes to reduce their threat profiles.

Although we are still far from achieving well-adopted industry standards, these initiatives effectively set the first successful quantitative and qualitative benchmarks to measure a company’s AI security posture.

Analogously, the industry has yet to develop a more mature security tooling ecosystem and a consensus on the best mitigation strategies to the the most exotic adversarial risks, but it is beginning to converge towards some general solutions against the most common attack vectors.

Some of these emergent best practices focus on restricting AI agents’ access to internal APIs only to what is strictly necessary via guardrails, enforcing sensible access-control policies and tracking the provenance of ML models.

Fundamental Technical Challenges

A Schrodinger’s cat black-box - courtesy of DALL-E 3

Complexity

AI systems are the most complex software components ever designed. The complexity of their architectures lies as much in the mathematical design of their models as it does in their infrastructure challenges.

Explainability - Or Lack Thereof

This inherent complexity is further amplified by the lack of explainability of Artificial Intelligence systems.

Explainability is the ability to model an AI agent's decision-making process by establishing causal or logical relationships between its inputs and outputs. While we can explain - debug - traditional software, AI systems largely operate as black boxes.

The opaque nature of AI has been a key focus of academic and industry research aimed at making their inner workings more human-digestible. Today, the field of Explainable AI (XAI) offers a range of increasingly sophisticated strategies to enhance the interpretability of AI outputs.

However, ML models are still far from being fully explainable and there’s a trade-off between a model’s performance and its interpretability.

This partial lack of transparency raises trust concerns for businesses and complicates the task of ensuring the security and regulatory compliance of ML models.

High-Value Targets

Machine Learning models are increasingly often the most valuable component of a software stack.

The main reasons are:

• The high capital requirements necessary to cover the costs of the engineering talent and the infrastructure where the ML models are trained and run.

• The intellectual property value of the datasets used to train the ML models - which often are highly confidential data such as financial or healthcare information.

• The increasingly critical function that AI components play in the application architecture and its business logic.

These properties make ML models compelling targets to attackers.

Imagine a scenario where a large tech company deployed a gargantuan R&D budget for the development of a new machine learning model. Then, once in production, an attacker is able to reverse-engineer an approximate replica of the company’s ML model.

This attack vector is called model theft. It is performed against Machine Learning models that expose an unprotected public-facing API.

High-level walk-through of a model theft exploit against a public-facing ML Model API

Attacker Scenario

Let's say there are two leading skincare apps in the digital cosmetic dermatology industry.

Company A recently released the latest version of its ML-powered skincare recommendation system, providing users with highly personalized skincare routines and product suggestions. This new feature has significantly boosted Company A's user growth and retention, far surpassing its competitor Company B. To further consolidate its new market lead, Company A opens up its ML model via a public API, allowing other skincare e-commerce shops to integrate the recommendation engine into their applications.

Its competitor Company B is nowhere near developing a similarly sophisticated machine learning model, so it decides to reverse-engineer Company A’s valuable AI intellectual property.

Essentially, the model theft exploit simplified in the above diagram boils down to the three main phases:

- After obtaining access to the A's skincare recommendation API, Company B designs several user persona templates which it uses for the automated generation of API queries with a wide range of user profiles such as different dermatological concerns (i.e.: aging, eczema, acne etc.), lifestyle factors and skin types. This systematic API probing is designed to model the recommendation system's ranking function.

- Company B iteratively starts pairing the prompt inputs and ML model's outputs & optimizing its reconnaissance queries. The API inputs selection strategy is designed to progressively refine the attacker’s payloads for the purpose of maximizing the knowledge gained on the underlying target model’s behavior.

- Eventually, Company B collects enough input-output pairs to approximate the output distribution of Company A’s skincare recommendation engine. It then uses this dataset to train or fine-tune its own replica model. As a result, Company B’s recommendation system is able to mimic quite closely the decision-making capabilities of Company A’s proprietary ML model - although never with absolute fidelity.

What’s the outcome? The digital skincare Company A partially lost its competitive advantage - the proprietary nature of its recommendation engine's ML model - into which it had invested substantial engineering and infrastructure resources. For Company B, on the other hand, the attack was relatively inexpensive.

What makes model theft even more concerning is, in fact, that it’s a relatively cost-effective attack compared to the financial damage that it can inflict on its targets.

Luckily, this AI attack is still mostly theoretical as its real-world exploit implementation are rare and usually not as effective. Nevertheless, in the near future we can expect more exploit attempts bridging the gap between academic theory and real-world model stealing attacks: we will go through their main mitigation strategies in a future article.

Model stealing is just a relatively uncommon example within the growing taxonomy of AI security threats: there are many other AI attack vectors that are far more production-ready and pose a similarly devastating intellectual property loss and reputational damage to their targets.

Verdict

In conclusion, ML models are often the most capital intensive, the most valuable and the most obscure component in modern software architectures. At the same time, they are also one of the areas that is the most neglected from a cybersecurity perspective.

Securing them is, therefore, the most daunting challenge in the cybersecurity industry - as well as the most impactful.