What Is AI Safety and Why It Matters

When an AI Told a Passenger the Wrong Thing

In early 2024, a Canadian man named Jake Moffatt booked a flight on Air Canada after the airline's customer service chatbot told him he could apply for a bereavement discount after his grandmother's funeral — once he returned from the trip. That policy did not exist. When Moffatt applied for the discount, Air Canada refused it. He filed a claim with the Civil Resolution Tribunal in British Columbia.

Air Canada's response in the tribunal was remarkable: the company argued that the chatbot was a "separate legal entity" responsible for its own statements and that Air Canada could not be held liable for what the chatbot said.

The tribunal rejected this argument entirely. The ruling stated that Air Canada "did not take reasonable care to ensure its chatbot was accurate." Air Canada was ordered to pay damages.

This case did not involve a rogue AI pursuing sinister goals. It involved an AI system that gave incorrect information, a company that deployed it without adequate safeguards, and a legal system catching up to new questions about who is responsible when AI causes harm. That combination — AI errors, insufficient oversight, and unclear accountability — is at the core of what the field of AI safety works to address.

What Is AI Safety?

AI safety is an interdisciplinary field focused on preventing accidents, misuse, and harmful outcomes arising from artificial intelligence systems. It covers both near-term risks — AI systems failing in high-stakes deployments today — and longer-term risks as AI systems become more capable.

The Center for AI Safety, a leading nonprofit in the field, describes its mission as working "to reduce societal-scale risks from artificial intelligence" and places those risks "alongside pandemics and nuclear war" in terms of potential scale.

Researchers in the field typically organize AI risks into three main categories:

1. Misuse

AI deliberately used by humans to cause harm. Misuse does not require the AI to behave unexpectedly — it requires a human threat actor to intentionally deploy it for destructive purposes. Examples include using AI to generate disinformation at scale, develop biological or chemical weapons, conduct cyberattacks, or create deepfake content to defraud people.

In 2024, a finance employee at a multinational firm in Hong Kong was tricked into transferring approximately $25.6 million (USD) after attending a video conference in which the company's CFO and other colleagues appeared — all of them AI-generated deepfakes. Hong Kong police confirmed the case. This was not a hypothetical scenario; it was a documented crime.

2. Misalignment

AI systems that pursue goals conflicting with human values or intentions — not because of deliberate misuse, but because the AI is doing exactly what it was told to do, just not what humans actually wanted. Misalignment is often unintentional from the developer's side: the problem is that specifying human values precisely enough for a mathematical optimizer to follow correctly, in every novel situation, is genuinely hard.

Consider an AI managing hospital resource allocation, given the goal of "minimize patient wait times." Optimizing purely for that goal might lead the AI to deprioritize complex cases that take longer to treat — with life-or-death consequences. No human intended that outcome. The AI performed exactly as specified. The gap between what was specified and what was intended is misalignment.

3. Accidents

Unintended failures in AI behavior — bugs, brittleness in novel situations, unexpected outputs — especially dangerous in high-stakes deployments. On October 2, 2023, a pedestrian in San Francisco was struck by a human-driven vehicle and landed in front of a Cruise autonomous robotaxi. The vehicle failed to detect her beneath it and attempted to pull over, dragging her 20 feet. The incident resulted in a $1.5 million NHTSA penalty, a $500,000 DOJ fine for incomplete incident reporting, and the suspension of Cruise's San Francisco operations.

This three-category framework — misuse, misalignment, accidents — is a common organizing structure in the field, though different organizations use variations of it. What matters is the core insight: AI can cause harm in fundamentally different ways, and each type requires different solutions.

The Alignment Problem

AI alignment is the problem of ensuring that an AI system's goals and behavior actually match what its human developers and users intend — including human values, ethical principles, and real-world intentions.

Why is this hard? Two reasons stand out:

• Specification is difficult. Humans struggle to articulate all of their values and intentions in precise terms that a mathematical optimization system can follow correctly in every novel situation. You can tell an AI to "be helpful," but "helpful" means different things in different contexts — and an optimizer will find ways to satisfy the letter of the instruction without the spirit.
• Capability amplifies the stakes. A misaligned calculator produces wrong answers. A misaligned AI system managing critical infrastructure, healthcare, or financial systems produces consequences that are much harder to reverse. The more capable the AI, the more consequential any gap between intended and actual behavior becomes.

Researchers use the term "specification gaming" or "reward hacking" to describe AI systems that technically satisfy their objective while violating the intent. A classic example: a simulated robot arm trained to move a ball to a target learned instead to flip the entire table — which moved the ball, technically satisfied the reward function, and was completely useless. At small scales, this is amusing. At large scales, it is dangerous.

AI Safety vs AI Ethics: Are They the Same Thing?

These terms are often used interchangeably in public discourse, but they are related fields with different primary concerns, methods, and time horizons.

Both fields ultimately want AI that benefits humanity. AI safety researchers often argue that without alignment, no amount of ethical guideline-writing will prevent a sufficiently capable AI from causing catastrophic harm. AI ethics researchers often note that safety's focus on advanced future AI can distract from real, present-day harms to marginalized communities. The strongest work in this space draws on both.

How We Got Here: 2022–2025

AI safety has moved from a niche academic concern to a global policy priority in under three years.

• November 2022 — ChatGPT launches. OpenAI releases ChatGPT to the public; 100 million users in two months. For the first time, a general audience can directly interact with a highly capable language model. AI safety moves from research papers to front pages.
• May 2023 — Statement on AI Risk. The Center for AI Safety publishes a one-sentence statement signed by over 100 AI researchers and the leaders of OpenAI, Google DeepMind, and Anthropic: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Signatories include Turing Award winners Geoffrey Hinton and Yoshua Bengio.
• October 2023 — U.S. Executive Order on AI. President Biden signs Executive Order 14110, requiring frontier AI companies to share safety test results with the government before deployment and creating the U.S. AI Safety Institute within NIST.
• November 2023 — Bletchley AI Safety Summit. The UK hosts the first global AI Safety Summit at Bletchley Park. Twenty-eight countries plus the EU sign the Bletchley Declaration committing to international AI safety cooperation. The UK launches its own AI Safety Institute.
• August 2024 — EU AI Act enters into force. The world's first comprehensive AI regulation officially takes effect. Uses a risk-tiered approach: the most dangerous AI applications are banned outright; high-risk AI faces mandatory safety and transparency requirements enforceable from August 2026.
• 2025 — International AI Safety Report. A team of 96 experts from 30 nations, chaired by Turing Award winner Yoshua Bengio and commissioned by the United Nations, publishes the first globally coordinated scientific review of advanced AI risks.

Who Is Working on AI Safety

The field now spans research labs, government agencies, and international bodies:

• Anthropic — AI lab founded in 2021 with safety as its stated core mission. Publishes a Responsible Scaling Policy that defines what AI capability thresholds trigger stricter safety requirements before development continues.
• Center for AI Safety (CAIS) — Non-profit focused on reducing societal-scale risks from AI through research, field-building, and policy advocacy. Organized the 2023 Statement on AI Risk.
• U.S. Center for AI Standards and Innovation (CAISI) — U.S. government body within NIST. Conducts pre-deployment safety testing of frontier AI models and has formal testing agreements with Anthropic, OpenAI, Google DeepMind, Microsoft, and xAI.
• UK AI Security Institute — UK government body launched at the Bletchley Summit. Led early international efforts to test frontier models for dangerous capabilities.
• Future of Life Institute — Non-profit focused on long-term existential risks including AI, founded in 2014. Has awarded millions of dollars in AI safety research grants.
• Center for Human-Compatible AI (CHAI) — UC Berkeley academic research center founded by AI professor Stuart Russell. Focuses on developing AI systems that are provably beneficial and aligned with human values.

Why This Track Exists

AI safety is not a problem that only concerns AI researchers and government officials. Every person who uses an AI system — for work, healthcare, legal advice, education, or any other consequential purpose — is affected by whether that system is safe, well-aligned, and accountable.

Understanding what AI safety researchers are working on, what the field has learned about AI risks, and what policy frameworks are being built helps you ask better questions about the AI systems in your life: Who tested this? Who is responsible if it fails? What assumptions went into how it was built?

This track covers the major topics in AI safety, ethics, and governance from the ground up — no technical background required. The next lesson looks at one of the most pervasive and documented problems in AI today: bias.