Skip to content

Prompt Refinement & Stress-Testing

When to use

You have a prompt that mostly works but sometimes produces off-target output — wrong format, missed instructions, hallucinated content, or inconsistent quality. You want to diagnose the failure modes and produce a tighter version.

Prompt

I have a prompt that I use regularly but that produces inconsistent results.
I want you to analyze it, identify the failure modes, and produce an improved
version.

Here is the current prompt, followed by my description of what goes wrong.

## Step 1: Diagnosis
Analyze the prompt and identify:

**Ambiguities** — instructions that a model could reasonably interpret in more
than one way. For each, quote the ambiguous language and describe the two (or
more) plausible readings.

**Missing constraints** — things the prompt assumes but doesn't say. Common
gaps:
- Output format not specified (model guesses between prose, bullets, headers).
- Length not specified (model defaults vary).
- What to do when information is missing (guess vs. flag vs. skip).
- Priority ordering when instructions conflict.
- Whether to include or exclude certain content categories.

**Overloaded instructions** — single sentences that ask the model to do
multiple things, making it likely to drop one. Quote each overloaded
instruction and suggest how to decompose it.

**Competing objectives** — places where two instructions push in opposite
directions without a tiebreaker (e.g., "be thorough" vs. "be concise").

**Hallucination risk** — any instruction that invites the model to generate
content it cannot verify (e.g., "cite relevant cases" without a verification
requirement).

## Step 2: Failure Mode Inventory
Based on the diagnosis and my description of problems, produce a numbered list
of distinct failure modes, each in this format:
- **Failure mode:** one-sentence description.
- **Root cause:** which aspect of the prompt (or absence of instruction)
  causes this.
- **Fix:** the specific language change that would prevent it.

## Step 3: Revised Prompt
Produce the full revised prompt with all fixes incorporated. Do not just
describe the changes — write the complete prompt I can copy and paste.

Mark every change with a bracketed comment on the line above it:
[FIX: failure mode #N — description of change]

## Step 4: Edge Case Test Plan
List 3-5 edge case inputs I should test the revised prompt against to verify
the fixes work. For each, describe:
- The test input (briefly).
- What the old prompt would likely produce.
- What the revised prompt should produce.

Current prompt:
{{paste your current prompt here}}

Problems I've observed:
{{describe what goes wrong — specific examples of bad output, inconsistent
formatting, missed instructions, hallucinations, etc.}}

Notes / variations

  • For prompt creation from scratch (rather than refinement), replace the current prompt and problems section with: "I want to create a prompt for the following task: [describe task]. Start with Step 2 (anticipated failure modes for this type of task), then go directly to Step 3 (draft prompt with preventive measures built in), then Step 4 (test plan)."
  • For A/B testing, add: "Produce two alternative versions of the revised prompt — one optimized for precision (minimize false positives / wrong output) and one optimized for recall (minimize missed content / incomplete output). Label each with its optimization target."
  • For system prompt refinement (as opposed to a one-shot prompt), add: "The prompt I'm refining is a system prompt / project instruction that governs many conversations. Focus the diagnosis on instructions that are likely to drift or be forgotten over long conversations, and add anchoring mechanisms (e.g., periodic self-checks, explicit priority ordering) where needed."