This GitHub Copilot Evals tutorial starts with a question Nigerian coding learners can test: does one clearer instruction improve invalid-input handling without breaking normal behaviour? A convincing answer needs fixed tasks, repeat runs and a grading contract that the agent cannot quietly change.
Our starter kit separates two kinds of evidence. We executed 24 local Python checks against a reference and three deliberately defective implementations. Separately, we designed 36 Copilot runs; all remain unrun. The local checks validate the teaching grader, not an AI assistant’s performance.
What is new in this GitHub Copilot Evals tutorial?
GitHub’s 17 November announcement introduces Copilot Evals in public preview for skills, agents and MCP tools. It lists VS Code, Copilot CLI and the Copilot app, with approval before each run. The tool supports repeatable checks and comparison after a customization changes.
The official repository distinguishes saving or validating an evaluation from running it. Runs can execute code and incur token or credit costs. Therefore, this report does not install the preview plugin or start a paid run without a stated budget. It supplies neutral fixtures and an execution ledger instead of pretending an offline script is Copilot Evals.
Freeze a small invoice contract
First, define six tasks for a fictional invoice-total function. It returns integer kobo from decimal-string unit prices and integer quantities. The contract covers a normal total, an empty invoice, negative quantity, nonnumeric price, fractional quantity and final-sum rounding. The three invalid-input cases must raise ValueError.
The normal example totals NGN 2,500.99, or 250,099 kobo. The rounding example uses a unit price of NGN 0.005 and quantity three. Its exact sum is NGN 0.015, which rounds half-up to two kobo. Rounding the unit price first would produce a different answer. This is a deliberately chosen test boundary, not financial advice or a production billing rule.
| Task | Expected result | Why include it? |
|---|---|---|
| Normal invoice | 250099 kobo | Preserve ordinary arithmetic |
| Empty invoice | 0 kobo | Define the empty boundary |
| Negative quantity | ValueError | Reject a disallowed quantity |
| Nonnumeric price | ValueError | Reject invalid price text |
| Fractional quantity | ValueError | Enforce integer quantities |
| Round final sum | 2 kobo | Expose premature rounding |
Check whether the grader can reject known defects
Next, run the local Python validation script. It checks the reference and three authored mutations: allowing negative quantities, ignoring quantities and rounding unit prices early. Each implementation receives the same six tasks. No model generated these implementations, and repeating the script would not create independent AI samples.

The execution record also demonstrates a changed-copy hash check. Adding a newline to a copy produces a different digest, so the guard rejects it while retaining the original task file. That proves this simple integrity check detects that alteration. It does not make the grading environment tamper-proof or replace review of changed files.
Design the 36-run comparison
Then, cross six tasks with three repeats and two instruction variants. That gives 36 planned runs, with eighteen allocated to each variant. The second instruction adds only an explicit invalid-input requirement. Keep the model, repository, fixtures, tests and budget fixed so that the intended difference stays interpretable.

Use a fresh workspace and session for each run where the tool permits it. Also counterbalance run order rather than always evaluating the new instruction last. Record residual shared state and other limitations. Repeats can reveal variation, but three attempts do not establish independence or a universal success probability.
Keep instructions and grading independent

Reject a run that edits the tests to obtain a pass. Save test hashes before and after execution, inspect the file diff and record unrelated changes. A valid final number is not enough if the agent weakens the assertion or removes a failing case. The ledger keeps correctness and run eligibility separate.
Similarly, a failed setup, tool error or interrupted run needs its own status. Do not silently omit it from the denominator. Decide how to count those outcomes before execution, then retain the reason. Keep the original task result visible even if a separate reporting step fails.
Measure outcomes before claiming improvement
Record per-task passes out of three for each variant, with the complete run ledger alongside any summary. Preserve duration, observed usage and file changes. If the tool does not expose a measure, mark it unavailable rather than estimating it. A local zero-credit grader check cannot predict the cost of 36 model runs.
This report makes no claim that either instruction improves Copilot. All model-run fields remain unfilled. Our observed result is narrower: the local reference passes all six tasks, the grader catches three chosen defects and a changed-copy integrity check detects an alteration. Other defects can still escape these tasks.
Download the GitHub Copilot Evals tutorial kit
- Twenty-four observed local grader checks (CSV)
- Thirty-six proposed Copilot runs (CSV)
- Six-task frozen invoice contract (CSV)
- Blank budget and run-audit ledger (CSV)
- Python grader, instructions and starter kit (ZIP)
Finally, read the official setup and approval flow before adapting the neutral files to the preview tool. Our ZIP is not a validated Vally configuration or a complete Copilot integration. For related evidence boundaries, see our safe review-scope lab and planned Android diagnostics comparison.
Unlock Your Unique Tech Path
Get expert tech consulting tailored just for you. Receive personalized advice and solutions within 1-3 business days.
Get Started