EU AI Act Article 15: What Adversarial Testing Actually Means

The EU AI Act (Regulation 2024/1689) entered into force on August 1, 2024. It is the first comprehensive horizontal AI regulation anywhere in the world, and the cybersecurity obligations buried inside it are stricter than most security teams realize. Article 15 is where the teeth are.

This guide walks through what Article 15 actually requires for high-risk AI systems, who's in scope (including non-EU companies), what adversarial testing looks like in practice, and how to prepare in the time you have left before the August 2, 2026 deadline. No regulatory hand-waving — just the specific obligations and what they mean operationally.

What Article 15 Actually Says

Article 15 is the EU AI Act's accuracy, robustness, and cybersecurity requirements clause for high-risk AI systems. The cybersecurity-specific obligations are concentrated in three sub-paragraphs:

Article 15(1) — Design Requirement

High-risk AI systems must be "designed and developed in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle." This is the high-level mandate; the specifics come later.

Article 15(4) — Resilience to Errors and Faults

The system "shall be resilient against errors, faults or inconsistencies that may occur within the system or the environment in which the system operates." Read narrowly, this is fault tolerance. Read in context with Article 15(5), it's the start of the cybersecurity story.

Article 15(5) — Resilience Against Unauthorised Third Parties

Here's the operative cybersecurity clause: high-risk AI systems "shall be resilient as regards attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities." This is where security testing becomes mandatory in practice.

Recital 77 — The Technical Specifics

If Article 15 is the legal mandate, Recital 77 is the technical brief. It calls out the specific attack categories that the technical solutions must address:

• Data poisoning — manipulating the training dataset
• Model poisoning — manipulating pre-trained components used in training
• Adversarial examples / model evasion — inputs designed to cause the AI to make a mistake
• Confidentiality attacks — including model extraction and inversion
• Model flaws — exploitable defects in the model itself

This list is the AI-security adaptation of what penetration testers already do for traditional applications, with three categories that don't have direct analogues outside AI: data poisoning, model poisoning, and adversarial examples. We cover each below.

What "Adversarial Testing" Actually Requires

The term "adversarial testing" doesn't appear in the regulation in those exact words — it shows up in the official guidance and in industry shorthand. What Article 15 and Recital 77 effectively mandate is a cybersecurity testing program that exercises the AI model and its training/inference pipeline against the five threat categories they enumerate, plus the robustness obligation in Article 15(4). The closest existing reference points are the NIST AI Risk Management Framework and ENISA's Multilayer Framework for Good Cybersecurity Practices for AI.

In operational terms, an Article 15 testing engagement exercises the model and its training/inference pipeline across the five Recital 77 categories plus Article 15(4) robustness:

1. Adversarial Examples and Out-of-Distribution Robustness

Inputs crafted specifically to cause the model to produce a wrong output. For computer-vision systems, this is the canonical "stop sign with stickers" attack. For text models, it's input perturbation that flips classifications or breaks decision boundaries. Out-of-distribution boundary testing assesses how the model behaves under unexpected inputs. Testing here exercises whether the model degrades safely under crafted or anomalous inputs.

2. Data Poisoning Resilience

Reviewing training-data integrity controls and supply-chain provenance. If your model retrains on user inputs or scraped data, can an attacker inject poisoned samples that bias the model's future outputs? Pipeline mapping, access-control rating on each ingestion pathway, and integrity verification using a proprietary injection methodology.

3. Model Poisoning and Supply Chain Integrity

If you fine-tune on top of a foundation model, who controls that base model and the fine-tuning data? Pre-trained components can carry backdoors — triggers that activate malicious behavior on specific inputs. Testing involves SHA-256 hash verification on imported model weights against authoritative reference hashes, version pinning audit across the ML dependency graph, and pre-trained component inventory and source URL verification.

4. Confidentiality Attacks

Black-box membership inference attacks (determining whether a specific record was in the training set) and, for LLM-based systems, training-data extraction probes. These are particularly relevant for systems trained on regulated data — health records, financial records, biometric templates. Note that model inversion (reconstructing training records from outputs) is a related but distinct attack class with its own methodology, typically scoped separately.

5. Model Flaws and Robustness (Article 15(4))

Article 15(4) requires resilience against errors, faults, and inconsistencies — adjacent to but distinct from 15(5)'s adversarial categories. OOD failure-mode classification, decision-boundary stability analysis, and edge-case behavior assessment fall here. Continuously learning systems also get a feedback-loop adversarial-detection review.

Who Article 15 Applies To

Article 15 obligations attach to providers and deployers of high-risk AI systems. The categories are listed in Annex III:

1. Biometrics — including remote biometric identification and emotion recognition
2. Critical infrastructure — AI used as a safety component in road, rail, water, gas, electricity, etc.
3. Education and vocational training — admissions, evaluation, monitoring
4. Employment and worker management — recruitment, performance evaluation, task allocation
5. Access to essential private and public services — credit scoring, public benefits, emergency dispatch, insurance underwriting
6. Law enforcement — risk assessment, polygraph-like systems, evidence evaluation
7. Migration, asylum, and border control — visa eligibility, polygraphs, document verification
8. Administration of justice and democratic processes — judicial decision support, election influence systems

Plus: AI systems used as a safety component of products covered by the Union harmonisation legislation in Annex I (medical devices, machinery, toys, lifts, radio equipment, etc.).

Providers vs. Deployers vs. Distributors

The obligations differ slightly by role:

• Providers develop or have developed an AI system and place it on the market under their name. They carry the bulk of the testing and conformity-assessment obligations.
• Deployers use an AI system under their own authority (other than in the course of a personal non-professional activity). Deployers have monitoring, human oversight, and incident-reporting obligations, plus a fundamental rights impact assessment for certain deployments.
• Distributors and importers have verification duties — confirming that the provider has done the conformity work.

Extraterritorial Reach

Article 2 extends the regulation to providers and deployers established outside the EU when the output produced by the AI system is used in the EU. This is the clause most US-based companies underestimate. A US fintech using an AI model to underwrite loans for EU customers is in scope. A US SaaS embedding an AI feature that processes EU user data is in scope. A US healthcare-tech vendor whose AI is used by an EU clinic is in scope.

The Compliance Timeline

The EU AI Act phases in over several years. Here's what's already in force and what's coming:

• February 2, 2025 — Prohibited AI practices banned (social scoring, exploitative manipulation, real-time remote biometric ID in public, etc.). Already in effect.
• August 2, 2025 — General-purpose AI (GPAI) governance and the codes of practice took effect. AI literacy obligations apply. Already in effect.
• August 2, 2026 — High-risk AI systems listed in Annex III must be fully compliant. This is the deadline you're racing.
• August 2, 2027 — High-risk AI systems already on the market that have not undergone substantial modification become subject to the full requirements. Also: the date when AI systems used as safety components of products covered by Annex I harmonisation legislation become subject.

What "Substantial Modification" Means

If you're relying on the 2027 grace period, watch for substantial modifications that pull you back to the 2026 deadline. A substantial modification is, roughly, a change that alters the intended purpose or that affects compliance with the regulation. Adding new training data is generally considered a substantial modification. Retraining the model is generally considered substantial. So is deploying the model in a new high-risk use case.

Penalties

Article 99 sets the administrative fines:

• Up to €35 million or 7% of total worldwide annual turnover (whichever is higher) for prohibited AI practices
• Up to €15 million or 3% for non-compliance with operator and provider obligations (this is where Article 15 violations land)
• Up to €7.5 million or 1% for incorrect, incomplete, or misleading information to authorities

For SMEs and startups the fine is the lower of the two amounts rather than the higher — small consolation if you're a Series B fintech.

What the Testing Looks Like in Practice

An Article 15 conformity-supporting engagement runs longer than a standard pentest because the AI-specific layers add work that traditional testing doesn't cover. A compressed but still-defensible scope for a high-risk AI system used in financial underwriting looks like this:

• Week 1 — Recon, threat modeling, and AI lifecycle mapping. Map the AI lifecycle (data sources, training pipeline, model-supply chain, deployment topology, inference endpoints) and confirm the in-scope endpoints, training pipeline ownership, and pre-trained component inventory. Architecture review and Metric Pre-Registration Document countersigning happens in this window.
• Week 2 — Adversarial testing and data/model integrity review. Adversarial perturbation campaigns against representative production inputs, with toolchain calibrated to data type. Out-of-distribution boundary testing. Plus training-data provenance, pipeline access-control rating, model-checkpoint integrity verification, and fine-tuning supply-chain review.
• Week 3 — Confidentiality testing, Article 12 logging verification, and reporting. Black-box membership inference attacks plus training-data extraction probes for LLM-based systems. Article 12 logging verification submits a structured set of inference requests and verifies the required fields are captured server-side. Findings written for both technical and conformity-assessment audiences with detailed remediation recommendations. Stakeholder reporting tailored to drop directly into your Annex IV technical file, paired with a 30-minute debrief call.

Three weeks is the floor for a real Article 15 engagement on a non-trivial system. The exact scope varies — systems using a frozen foundation model with no fine-tuning collapse the data-integrity review, while continuous-learning pipelines require an entirely separate review track for the training pipeline itself.

Common Misconceptions

"We're not in the EU, so this doesn't apply."

Article 2 disagrees. If your AI's output is used in the EU, you're a provider or deployer in scope. The regulation explicitly anticipated this and pulled non-EU companies in.

"We use a foundation model from OpenAI/Anthropic/Google. They handle compliance."

They handle their compliance for the foundation model. You handle yours for the system you build on top. The foundation-model GPAI obligations took effect August 2025 — and those obligations don't transfer their downstream compliance to your high-risk system. You still have to test how your system uses the model, the prompts and tool integrations, and whether your application introduces new attack surface.

"A regular pentest will cover this."

A good pentest covers the application around the AI. It doesn't cover adversarial input testing against the model, training-data poisoning resilience, model-supply-chain integrity, or confidentiality attacks against the model itself. The OWASP LLM Top 10 categories are not in a typical web pentest scope.

"Our AI isn't 'high-risk.'"

Read Annex III carefully before concluding this. Credit scoring, hiring algorithms, performance evaluation, insurance underwriting, and biometric verification are all in scope. Many SaaS companies have at least one feature that lands in a high-risk category they didn't realize was regulated.

"We have a NIST AI RMF assessment, that should cover it."

The NIST AI RMF is a good starting point and many of the controls map across, but it isn't a conformity assessment for the EU AI Act. Annex IV requires specific technical documentation that goes beyond NIST's voluntary framework. Use NIST AI RMF as scaffolding; use the Annex IV checklist as the actual deliverable.

What Voke Delivers: Kickoff, Testing, Artifacts

The Voke Cyber engagement is structured around three things you'll see and sign: the Metric Pre-Registration Document at kickoff, a Signed Test Log entry per testing phase, and an Annex IV-aligned cybersecurity documentation support package at delivery. The calendar from kickoff to final delivery is typically three to four weeks.

Kickoff — Article 9(8) Metric Pre-Registration

The engagement opens with an architecture review and the Metric Pre-Registration Document. We sign and you countersign the pre-defined pass/fail thresholds for each Article 15(5) cybersecurity threat category — before any testing begins. The dating is structurally non-backdateable; that's the point. Article 9(8) requires testing against pre-defined metrics, and this is how the engagement produces a record that the metrics existed before the results.

Active Testing — Phases 2 through 7

The active testing window covers six article-mapped phases, each producing a Signed Test Log entry tied to its pre-registered threshold:

• Phase 2 — Data Poisoning Assessment (Article 15(5))
• Phase 3 — Model Poisoning / Supply Chain Integrity (Article 15(5))
• Phase 4 — Adversarial Examples + Out-of-Distribution Testing (Article 15(5))
• Phase 5 — Confidentiality Attacks / Membership Inference (Article 15(5))
• Phase 6 — Model Flaws & Robustness (Article 15(4))
• Phase 7 — Article 12 Logging Verification (Article 12)

Phases 2 and 3 are conditional. Phase 2 scopes to zero hours if you're a pure API wrapper without an owned training pipeline; Phase 3 scopes to zero if no pre-trained components are imported. Either way, the scope-out is documented in a signed justification and the engagement fee adjusts accordingly — the assessment focuses on the phases where you actually own the attack surface.

Final Artifacts — Annex IV-Aligned Cybersecurity Documentation Support Package

Phase 8 compiles the support package: security assessment summary, testing records and assessment log, findings register with risk ratings, description of identified cybersecurity weaknesses and observed failure modes, remediation recommendations and prioritization guidance, and supporting cybersecurity assessment evidence. Plus the final assessment report (the narrative wrapper) and a 30-minute debrief call with technical and compliance personnel. The package supports — does not replace — the customer's internal Annex IV technical documentation effort.

Frequently Asked Questions

Does the EU AI Act actually require penetration testing?

Yes, in operational terms, for high-risk AI systems. Article 15(5) requires resilience against attempts by unauthorised third parties to alter the system's use, outputs, or performance. Recital 77 specifies the technical solutions must address data poisoning, model poisoning, adversarial examples, confidentiality attacks, and model flaws. Article 9(8) further requires testing against pre-defined metrics before market placement. There's no other practical way to demonstrate resilience to those threats than to test against them.

How is Article 15 testing different from a regular penetration test?

A traditional pentest covers the application, infrastructure, and authentication around the AI. Article 15 cybersecurity testing focuses specifically on the model itself: adversarial inputs, training-data integrity, model-supply-chain controls, and confidentiality boundaries. The two engagements use different methodologies and tooling. The Voke Cyber Article 15 engagement is structured around the model-layer cybersecurity requirements named in Recital 77 and Article 15(4).

Does Article 15 apply if I'm not based in the EU?

Yes, if your AI's output is used in the EU. Article 2 explicitly extends the regulation to non-EU providers and deployers when the output is used inside the EU. This pulls in most US-based companies serving EU customers.

What's the penalty for non-compliance?

Up to €15 million or 3% of total worldwide annual turnover (whichever is higher) for non-compliance with operator and provider obligations. Higher fines (up to €35 million or 7%) for prohibited AI practices.

Can your team test our system for prompt injection?

Not under the Article 15 cybersecurity assessment. Prompt injection isn't named in Recital 77 as a required test category — it's a real LLM-specific threat addressed under our LLM/AI Penetration Testing service line, which covers OWASP LLM Top 10 categories explicitly. We can quote prompt injection separately as a Change Order if you want it bundled with your Article 15 engagement.

Can your team test for model inversion?

Not under the base Article 15 engagement. Phase 5 covers black-box membership inference and, for LLMs, training-data extraction probes — both confidentiality-attack categories named in Recital 77. Model inversion is a related but distinct attack class with separate methodology and tooling, scoped as a separate engagement.

Can our internal team do the adversarial testing?

Technically yes, but most regulators and notified bodies expect testing performed by an independent function or third party. Internal testing also tends to suffer from the "we built it, we know how it works" bias that adversarial testing specifically tries to defeat. An independent assessment produces evidence that's stronger for Annex IV documentation purposes.

What about general-purpose AI (GPAI) models?

GPAI obligations are separate (Articles 51-55) and took effect August 2, 2025. If you provide a GPAI model with systemic risk, you have your own testing, documentation, and incident-reporting obligations on top of Article 15 if you also deploy it as a high-risk system. GPAI red-teaming is a separate service track from this engagement.