Which practice involves injecting prompts to cause the AI to reveal restricted outputs or behave unexpectedly?

Which practice involves injecting prompts to cause the AI to reveal restricted outputs or behave unexpectedly?

• A. Data governance
• B. Adversarial Inference
• C. Prompt injection
• D. Data poisoning

Answer: (C)

Prompt injection is a manipulation technique that exploits how prompts and system instructions guide an AI’s responses. By weaving or embedding prompts in the user input, an attacker can override safeguards, prompting the model to reveal restricted outputs or behave in unintended ways. This highlights a risk where the model’s safety boundaries can be bypassed through the interaction itself, underscoring the need for robust guardrails, input sanitization, and resilient system prompts that cannot be easily overridden by user-provided text. This differs from data governance, which focuses on policies and practices for managing data quality, privacy, and access across an organization; data poisoning, which targets training data to degrade model performance or behavior; and adversarial inference, which seeks to extract sensitive information from the model or its training data through cleverly crafted queries.

Prompt injection is a manipulation technique that exploits how prompts and system instructions guide an AI’s responses. By weaving or embedding prompts in the user input, an attacker can override safeguards, prompting the model to reveal restricted outputs or behave in unintended ways. This highlights a risk where the model’s safety boundaries can be bypassed through the interaction itself, underscoring the need for robust guardrails, input sanitization, and resilient system prompts that cannot be easily overridden by user-provided text.

This differs from data governance, which focuses on policies and practices for managing data quality, privacy, and access across an organization; data poisoning, which targets training data to degrade model performance or behavior; and adversarial inference, which seeks to extract sensitive information from the model or its training data through cleverly crafted queries.