Chat Customizations Evaluations

An extension for analyzing and improving AI prompt files. Works with .prompt.md, .agent.md, SKILL.md, and .instructions.md files — providing LLM-powered semantic analysis directly in VS Code.

Features

LLM-Powered Analysis (via GitHub Copilot)

• Contradiction Detection — Finds logical, behavioral, and format conflicts
• Semantic Ambiguity — Ambiguity analysis with rewrite suggestions
• Persona Consistency — Detects conflicting personality traits and tone drift
• Cognitive Load Assessment — Warns about overly complex prompts with too many nested conditions
• Semantic Coverage — Identifies gaps in intent handling and missing error paths
• Composition Conflict Analysis — Detects conflicts between a prompt and other prompt files it imports via markdown links

Waza Integration

• Create Eval Scaffold — Generates eval files for a skill via waza new eval <skill-name>
• Run Evaluation — Executes skill evaluation via waza run <eval.yaml> --context-dir <skill-dir>
• Automatic Local Fallback — If waza is not on PATH, commands attempt a local fallback via go run ./cmd/waza when a sibling waza repo is available

Editor Integration

• Editor Title Bar — Analyze Prompt button appears when editing prompt files
• Command Palette — Chat Customizations Evaluations: Analyze Prompt command
• Problems Panel — All diagnostics appear in the standard VS Code Problems panel with precise line and column locations

Supported File Types

Pattern	Type
*.prompt.md	Prompt
*.agent.md	Agent
*.instructions.md	Instructions

Usage

1. Open any supported prompt file in VS Code
2. Run Chat Customizations Evaluations: Analyze Prompt from the command palette or click the beaker icon in the editor title bar
3. View results in the Problems panel (Ctrl+Shift+M / Cmd+Shift+M)

LLM analysis requires GitHub Copilot — no API keys needed. Just sign in to GitHub Copilot in VS Code.

Commands

Command	Description
Chat Customizations Evaluations: Analyze Prompt	Run full LLM-powered analysis on the active file
Chat Customizations Evaluations: Create Waza Eval Scaffold	Create eval.yaml and task files for the active skill
Chat Customizations Evaluations: Run Waza Evaluation	Run the skill's eval suite
Chat Customizations Evaluations: Download Waza Binary	Download the latest platform-specific waza binary and configure the extension to use it

Configuration

Setting	Default	Description
chatCustomizationsEvaluations.enable	true	Enable/disable the extension
chatCustomizationsEvaluations.trace.server	off	Trace communication between VS Code and the language server
chatCustomizationsEvaluations.customDiagnostics	[]	Array of custom diagnostic objects with name and description fields
chatCustomizationsEvaluations.waza.command	waza	Command used to run waza (for example /usr/local/bin/waza)

License

MIT