Who is responsible when an AI system causes harm

An AI system does not need intentions to cause harm. A support assistant can disclose customer records. A coding agent can weaken an access check. A purchasing agent can place an unauthorized order. These are failures of a deployed system, whether or not anyone describes the model as going rogue.

No developer can guarantee that a generative model will never produce an unsafe response. Organizations can, however, limit what that response can cause. Ethical development starts with that distinction, then assigns responsibility to the people who design, authorize, sell, and operate the system.

Define the authority before choosing the model

Start with a written operating boundary. Specify what the system may read, recommend, change, and commit. Treat those as separate permissions. An assistant that drafts a refund recommendation has a different risk profile from one that transfers money.

For each action, identify the affected person, the plausible harm, and the available remedy. An incorrect internal summary may be easy to replace. A disclosed medical record cannot be made private again. Reversibility should influence how much autonomy you allow, not just how quickly you deploy.

Include misuse and unequal outcomes. A recruiting assistant that consistently discounts nontraditional work histories can harm applicants while operating exactly as configured. Preventing unauthorized actions is necessary, but ethical development also requires examining whether authorized behavior is acceptable.

Put enforcement outside the model

A prompt saying “never reveal confidential information” is an instruction, not an access control. Retrieved documents, emails, and web pages can contain malicious instructions. Treat that content as untrusted input. Safety training and careful prompts help, but neither should determine what an agent is technically permitted to do.

• Restrict access: give each agent only the records, tools, and credentials required for its task.
• Validate actions: enforce recipients, amounts, file paths, and other limits in application code.
• Require approval: route consequential actions to an authorized person before execution.
• Bound execution: cap spending, tool calls, runtime, and the scope of changes.
• Support recovery: retain audit trails, rollback options, and a way to revoke access immediately.

Human approval only helps when the reviewer can understand the proposed action and reject it. Present the evidence, expected effect, and relevant uncertainty. A queue that pressures someone to approve hundreds of opaque requests is not meaningful oversight.

Match responsibility to control

Developers should be responsible for their engineering decisions, foreseeable risks they ignore, and claims they make. That does not mean an individual programmer should bear sole responsibility for every output. Responsibility should follow control, knowledge, and the ability to prevent harm.

A model provider controls training choices, baseline safeguards, and disclosures about known limitations. An application developer controls integration, permissions, testing, and interface design. The deploying organization controls the business purpose, operating environment, staffing, and acceptance of residual risk. One organization may hold several of these roles.

Name an accountable business owner and a technical owner before launch. Record who approves access, accepts unresolved risks, handles incidents, and can suspend operation. Legal liability depends on jurisdiction, contracts, sector rules, and facts. Contracts can allocate duties, but they do not necessarily remove obligations to affected people. Get legal advice for consequential uses rather than assuming a vendor disclaimer settles responsibility.

Test consequences, not convincing answers

Build evaluations around concrete failures. For a coding agent, test whether it exposes secrets, bypasses authorization, or changes files outside its assigned repository. For a customer assistant, test disclosure across accounts, fabricated policy claims, and hostile instructions embedded in retrieved material.

Include ordinary mistakes alongside deliberate attacks. Use representative workflows and, where appropriate and lawful, assess performance across affected groups. Measure completed actions and downstream effects, not just whether the final answer sounds safe. A polite response can accompany an unauthorized tool call.

Set release criteria before reviewing results. Repeat tests when models, prompts, tools, or permissions change. Keep testing in production through proportionate monitoring and incident review, while limiting sensitive data in logs. Passing a finite test suite is evidence about tested conditions, not proof of universal safety.

Make evidence part of the purchase

Whether you build or buy, request a permissions map, evaluation results, known failure modes, data retention terms, and an incident procedure. Ask what changes when the supplier updates its model. Require notification and revalidation arrangements proportionate to the risk.

Before granting production access, run a contained pilot with explicit limits and a named decision maker. Demonstrate that access can be revoked and pending actions stopped. For systems affecting customers, employees, or applicants, establish a usable route to challenge an outcome and reach a person authorized to correct it.