It examines how an adversary can influence data, models, inputs, outputs or surrounding workflows to violate confidentiality, integrity, availability or acceptable-use objectives. It is a body of attack and defense knowledge, not the name of one specific exploit.
NIST’s current taxonomy organizes attacks using dimensions such as the lifecycle stage, the adversary’s goal, objective, capability and knowledge. Relevant families include evasion at inference time, poisoning during training, privacy attacks and abuse of generative systems through prompting or misuse. Real systems also inherit conventional weaknesses in identities, APIs, software, cloud services and supply chains, so an adversarial-ML assessment should connect model behavior to the complete application.
Key points
Attack stageDetermine whether the adversary can affect data collection, training, model distribution, deployment, inference inputs, retrieval, feedback or connected actions.
Adversary conditionsState what access, knowledge, control, resources and query opportunities an attack assumes; a result under full model access may not transfer to a remote service.
Security objectivesEvaluate availability failures, untargeted or targeted integrity violations, privacy loss, model theft and misuse against consequences meaningful to the deployed context.
Defensive practiceCombine model-level evaluation with provenance, access control, secure engineering, monitoring, incident response and constraints on downstream decisions or actions.
Important limitationRobustness against a known benchmark or attack method is not general security. Attackers adapt, research assumptions vary, and controls that improve one property may leave application or human trust boundaries exposed.