Five Case Studies: Full Diagnosis and Fix

Five Prompts. Five Failures. Five Fixes.

The previous lessons covered the nine failure modes individually. This lesson puts the diagnostic process together across five complete worked examples, each covering a different failure mode. The goal is not to memorize these specific fixes — it is to see the diagnostic sequence applied end-to-end, so the pattern becomes recognizable in your own work.

Each case follows the same structure: original prompt, output received, failure mode identified, root cause, fixed prompt, what changed in the output. Read each case as a detective reading evidence — identify the failure mode before you read the diagnosis.

Case 1: Vague and Generic Output

Original prompt: "Write a welcome email for new customers."

Output received: A generic three-paragraph welcome email with the company name left as [Company Name], standard phrases about being excited to have the customer join, a list of generic resources, and a closing invitation to reach out with questions. The email could have come from any company selling any product to any customer.

Failure mode: Vague and generic output (Mode 1).

Root cause: No audience, no product context, no purpose beyond a generic welcome, no tone specification. The model had nothing to anchor to except the category "welcome email," so it produced the statistical center of all welcome emails it had seen.

Fixed prompt: "Write a welcome email for new customers of a B2B project management tool. The customer is a team lead at a 10- to 50-person company who just activated a free trial. They are technically comfortable but time-constrained. Goal: get them to complete the three-step setup checklist within the first 48 hours. Tone: efficient and warm — like a knowledgeable colleague, not a support ticket. Under 120 words. No hollow phrases like 'we're excited you're here.' Open with the specific next action they should take."

What changed: The output now names the specific next action in the first sentence, references the setup checklist explicitly, calibrates tone to the audience (busy team lead, not a consumer), and comes in under 120 words with no generic filler.

What the fix added: audience (team lead at a small company), product context (B2B project management, free trial), purpose (complete setup checklist within 48 hours), tone descriptor (knowledgeable colleague), length, and a negative constraint on hollow phrases.

Case 2: Hallucination and Fabrication

Original prompt: "What does the research say about the impact of flexible work arrangements on employee retention?"

Output received: A confident response citing "a 2024 Gallup study showing that 68% of employees with flexible work arrangements reported higher job satisfaction" and "research from the Society for Human Resource Management indicating a 23% reduction in voluntary turnover at companies with fully flexible policies." Both statistics sounded authoritative and were perfectly matched to the argument.

Failure mode: Hallucination and fabrication (Mode 5).

Root cause: The prompt asked for specific research findings on a topic where the model does not have reliable, verifiable statistics. The model produced plausible-sounding numbers because the prompt invited statistical specificity without providing a grounding document.

Fixed prompt: "I'm going to paste summaries of three recent articles on flexible work and retention. Based only on what I provide, identify the consistent findings across sources and note where they disagree. Do not add any statistics or study citations beyond what appears in my summaries. Flag any claims you make that go beyond the provided material. [paste article summaries]"

What changed: The output now cites only what was provided, flags where its analysis goes beyond the source material, and does not invent supporting statistics. The grounding document replaced the invitation to fabricate.

What the fix changed: replaced an open-ended retrieval request with a grounded synthesis task, added explicit scope limitation (only from provided material), and added a flag-for-verification instruction.

Case 3: Wrong Format and Structure

Original prompt: "Explain the difference between machine learning and deep learning for a non-technical audience."

Output received: A response with a header "Key Differences," followed by five bullet points comparing the two, each point containing technical terminology that contradicted the non-technical audience specification. The bullets were accurate but unsuitable for the audience and used the wrong format for an explanation.

Failure mode: Wrong format or structure (Mode 6), with a secondary audience mismatch (Mode 8).

Root cause: No output specification meant the model applied its default for a comparison task (bullet list with headers). The audience specification was present but too brief to overcome the format default, and the technical level was not constrained explicitly.

Fixed prompt: "Explain the difference between machine learning and deep learning for a non-technical product manager who needs to explain the distinction to business stakeholders. Two flowing paragraphs — no bullet points, no headers. Use an analogy, not technical definitions. Under 150 words. The goal is that a reader with no technical background could repeat the explanation in a meeting."

What changed: The output now uses two paragraphs of plain prose, opens with an analogy, uses no technical terminology, and comes in under 150 words — directly usable for a non-technical explanation.

What the fix added: explicit format (two paragraphs, no bullets, no headers), a length limit, a technique instruction (analogy), and a purpose clause that calibrated both the technical level and the length.

Case 4: Ignored Instructions

Original prompt: "Write a product description for our noise-canceling headphones. Key features: 40-hour battery life, folds flat for travel, works with any device via Bluetooth or cable. Keep it under 80 words. Tone: casual and fun, targeting Gen Z shoppers. Make sure to mention the 5-year warranty."

Output received: A decent product description, casual in tone, that ran 97 words and did not mention the warranty.

Failure mode: Ignored instructions (Mode 7) — two instructions dropped: the word count and the warranty mention.

Root cause: The word count instruction was mid-prompt, in a list of requirements where later items received less attention. The warranty instruction was last in the list, where the model's attention was weakest. Both were dropped in favor of the model's default behavior (write what feels complete, not what fits a count).

Fixed prompt: "Write a casual, fun product description for noise-canceling headphones targeting Gen Z shoppers. Tone: energetic, not corporate. Cover: 40-hour battery, folds flat for travel, works with any device. You must mention the 5-year warranty — this is required. [On its own line:] Hard limit: 80 words maximum."

What changed: The output now includes the warranty, stays under 80 words, and maintains the casual tone — because the two critical instructions were moved to positions of maximum visibility (warranty flagged as required inline; word count isolated at the end as a hard limit).

What the fix changed: moved the word count to the final line as an isolated hard limit; added "you must mention" and "this is required" to the warranty instruction to elevate its priority signal.

Case 5: Over-Refusal

Original prompt: "Explain how social engineering attacks work."

Output received: The model declined to provide a detailed explanation, citing concern that the information could be used to harm others, and offered instead to discuss cybersecurity best practices at a general level.

Failure mode: Over-refusal (Mode 9).

Root cause: The phrase "social engineering attacks" matched surface patterns associated with malicious intent in the model's training, triggering a safety default. The absence of professional context left the model with no signal that the purpose was defensive.

Fixed prompt: "I'm developing a security awareness training curriculum for employees at a financial services firm. I need to explain how social engineering attacks work so that employees can recognize and report them. Please describe the three most common social engineering techniques used against employees, focusing on the warning signs a non-technical employee should look for. This is for a defensive education purpose; the audience is bank employees, not attackers."

What changed: The output now provides a clear, specific explanation of social engineering techniques, framed for employee education, with practical warning signs that serve the defensive purpose.

What the fix added: professional context (security awareness curriculum), organization type (financial services firm), audience (non-technical bank employees), purpose (recognize and report), and an explicit statement of the defensive framing. None of these changed what was being asked — they changed the model's ability to evaluate who was asking and why.