Test and improve a skill beyond a single successful example
Translated from Korean. View original
One successful example does not guarantee the next task will succeed. Test a skill with ordinary input, missing information and a request where it should not be used. Record selection separately from output quality and change one thing at a time to find problems without endlessly expanding the instructions.
Should I define success first?
Write observable criteria instead of ‘good meeting minutes’: distinguish decisions from proposals, do not invent owners, and show a deadline or ‘undecided’ for each action item. Even three criteria are easier to compare than a vague impression.
You do not need to combine accuracy, format and tone into one score. If dates are correct but the tone is awkward, record them separately so you know what to fix.
What inputs should I prepare?
For a meeting-note skill, prepare a note with clear decisions and owners, one missing deadlines and one containing only proposals. You may use real work material, but replace personal information if sharing the test record.
Also include a request where the skill should not be used. For example, ask a meeting-note skill to research a company's competitors and check that it does not force the answer into an unrelated minutes template.
How do I separate selection from output quality?
Explicitly naming a skill helps test the procedure in its body. Asking naturally without naming it also tests automatic selection. Success when invoked directly does not prove automatic selection works.
Record the request, whether the skill was used, the actual result and deviations from the criteria. Previous responses left in a conversation can make comparison harder, so test under the same conditions in a new conversation.
How much should I record about a failure?
A short text file or table is enough. Connecting the observed problem to a candidate fix makes the next revision easier.
Input: ‘We will decide the launch date at the next meeting.’ Expected: Mark the launch date undecided. Actual: Used the next meeting date as the launch date. Problem: Confused the discussion date with the execution date. Candidate fix: Classify unconfirmed dates as items to check.
Keep successful cases too. They help you check whether an edit changed behavior that previously worked.
How much should I change at once?
Start with one rule that caused a problem. If you change the description, workflow and output format together, it is hard to know what helped. Before adding instructions, also look for conflicting sentences.
After editing, rerun both the failed input and ordinary inputs. The same input can produce different results, so check important criteria more than once, but do not present a few successes as overall accuracy.
Frequently asked questions
Q. Must I build an automated evaluator first? A. You can start with a few real requests and explicit review criteria. Consider automation when repeated comparisons increase.
Q. Should I add a rule for every failure? A. First check whether an existing rule is ambiguous or conflicts with another. A more specific example may be better than adding another sentence with the same meaning.