Few-Shot Classification Prompts: Label Design, Examples, and Accuracy Tuning

Overview

Few-shot classification is an LLM workflow in which the prompt includes a task description, a set of possible labels, and several labeled examples. The model uses those examples as guidance for classifying new text. Unlike a simple zero-shot prompt, the model does not have to infer the desired boundaries from definitions alone.

The examples are not a replacement for clear label definitions. They work best when the taxonomy, instructions, examples, and output format all reinforce one another. A support team, for example, might classify incoming tickets as billing, technical issue, or account access. If the examples use “I was charged twice” for billing and “I cannot sign in” for account access, the model has concrete decision patterns to follow.

Start by deciding whether each input should receive one label or several. In a single-label task, categories should be mutually exclusive where possible. In a multi-label task, labels may coexist, such as billing and urgent. Define what each label means, what it does not mean, and how to handle text that does not fit.

Few-shot prompting is often a practical middle ground for LLM application development. It is also easier to revise when categories change, but every revision should be evaluated rather than accepted based on a few successful outputs.

Checklist by scenario

Single-label classification

Use a single-label prompt when each text must receive exactly one category. Tell the model whether it must choose from the supplied labels and what to do when no category is appropriate. An explicit fallback such as other or needs_review can be safer than forcing a poor match.

System: You classify customer messages into exactly one label.

Labels:
- billing: charges, invoices, refunds, or payment questions
- technical_issue: a product feature is not working as expected
- account_access: sign-in, password, verification, or account lockout
- needs_review: insufficient information or no suitable label

Rules:
1. Choose exactly one label.
2. Use the label definitions, not keywords alone.
3. If two labels seem plausible, select the one that best describes the customer's main request.
4. Return valid JSON with keys "label" and "reason".

Examples:
Input: I was charged for the same order twice.
Output: {"label":"billing","reason":"The message concerns a duplicate charge."}

Input: The dashboard stays blank after I sign in.
Output: {"label":"technical_issue","reason":"The user can sign in but a product feature is not working."}

Input: My verification code never arrives.
Output: {"label":"account_access","reason":"The user cannot complete account verification."}

New input: {{text}}
Output:

Keep the reason short if it is returned to an application. If a downstream system only needs the label, omit the reason to reduce parsing and token overhead. For reliable machine handling, pair the prompt with a schema or structured-output mechanism where your chosen API supports one. See the guide to structured output prompting for a broader implementation checklist.

Multi-label classification

Use a multi-label prompt when more than one category can be true. Define whether the labels describe topics, intent, urgency, sentiment, or another dimension. Do not mix dimensions in one unexplained list. “Billing” and “technical issue” may be topics, while “urgent” describes priority; combining them can be valid, but the prompt must state the rule.

System: Assign every applicable label to the message.

Labels:
- billing: payment, invoice, refund, or charge issue
- technical_issue: a product behavior or feature is failing
- urgent: the message indicates immediate business impact or a time-sensitive risk

Rules:
1. Return an array of zero or more labels.
2. Include a label only when the text provides evidence for it.
3. Do not infer urgency from an upset tone alone.
4. Sort labels alphabetically for stable comparison.
5. Return JSON: {"labels": ["..."]}.

Examples:
Input: The payment failed and our checkout is blocked for every customer.
Output: {"labels":["billing","technical_issue","urgent"]}

Input: Can you send me a copy of last month's invoice?
Output: {"labels":["billing"]}

New input: {{text}}
Output:

Ambiguous or overlapping categories

Include at least one example for each important boundary, not just one obvious example per label. If “technical issue” includes a failed export but “billing” includes an incorrect invoice total, show both cases. Add a tie-breaking rule that identifies the primary user intent. For genuinely uncertain inputs, use a review label or return a confidence flag for human triage rather than pretending the distinction is clear.

Choosing examples for the prompt

1. List the real categories and write a one-sentence inclusion rule for each.
2. Collect representative inputs from different users, channels, lengths, and writing styles.
3. Remove examples that contain unnecessary identifying information or irrelevant context.
4. Choose examples that demonstrate both ordinary cases and category boundaries.
5. Balance the examples so one label does not dominate the prompt unless that reflects the actual task.
6. Keep a separate evaluation set; do not use every labeled example in the prompt.

What to double-check

Before relying on a few-shot classifier, build a test set that reflects the traffic it will receive. A useful minimum workflow is:

1. Define the unit of classification. Decide whether the input is a sentence, a complete ticket, a conversation turn, or a document segment.
2. Freeze the taxonomy. Record label names, definitions, exclusions, and fallback behavior before measuring results.
3. Stratify the test set. Include each label, common wording variations, short and long inputs, misspellings, and ambiguous cases. If inputs come from several channels, include each channel.
4. Separate prompt examples from test data. Do not judge the prompt on examples it was shown.
5. Run repeatable evaluations. Keep the model version, prompt version, settings, input set, and output format together so changes can be compared.
6. Review confusion pairs. Group errors such as billing versus account access rather than looking only at an overall score. The most important improvement may be a clearer boundary, not a new example.
7. Inspect abstentions and malformed outputs. A review label, missing field, or invalid JSON can matter operationally even when category assignments look accurate.

When reviewing a misclassification, ask four questions: Was the label definition ambiguous? Was the correct distinction represented in the examples? Did irrelevant context distract the model? Did the output format permit an answer that the application cannot safely use? Fix one issue at a time and rerun the same test set.

Keep prompts concise enough to leave room for the input and output within the selected model's context limits. If examples become numerous, select a small curated set or retrieve examples by similarity. A text similarity checker can help compare candidate examples, but similarity alone should not decide which examples represent a business category.

Common mistakes

• Using vague labels: “issue,” “request,” and “other” do not provide useful boundaries without definitions.
• Showing only easy examples: Obvious examples do not teach the model how to resolve overlaps.
• Inconsistent output instructions: Asking for a label in one place and a list in another creates avoidable parsing failures.
• Letting examples contradict the rules: A labeled example can silently override a definition in the model's interpretation.
• Forcing every input into a category: Add a review or unknown path when the taxonomy cannot cover every case.
• Changing several variables at once: If the model, labels, examples, and output schema all change together, error analysis becomes difficult.
• Measuring only aggregate accuracy: Frequent labels can hide poor performance on rare or operationally important categories.
• Ignoring sensitive data: Remove unnecessary personal or confidential information from examples and test fixtures.

For workflows with multiple classification stages, separate routing from detailed classification. A short first pass can identify the relevant department, followed by a narrower prompt for intent. This can make label definitions clearer, although it adds another step to evaluate. The article on prompt chaining patterns provides a framework for testing such multi-step designs.

When to revisit

Treat a few-shot classification prompt as a maintained production component, not a one-time instruction. Revisit it before seasonal planning cycles when message volume, products, or customer language may change. Review it whenever workflows or tools change, including a new model version, API integration, output schema, routing step, or context-window constraint.

Schedule a taxonomy review when a new category is added, two labels are repeatedly confused, the fallback category grows, or support and product teams change how cases are handled. Add newly observed edge cases to a labeled review queue rather than inserting every unusual input directly into the prompt. Promote an example only after a human confirms its label and its value as a general pattern.

A practical maintenance checklist is:

• Version the taxonomy, prompt, examples, and evaluation set together.
• Record the reason for each label or example change.
• Run the regression set after every material change.
• Compare confusion pairs, fallback rates, parsing failures, and operational impact.
• Check token usage and latency after adding examples or explanations; the LLM token cost optimization guide covers ways to keep prompts efficient.
• Document the model and API behavior assumed by the prompt, then recheck it when those dependencies change.
• Use a staged rollout or human review for high-impact classification changes.

Finally, keep an evaluation record that another developer can reproduce. The LLM prompt testing framework can help organize accuracy, consistency, safety, and reliability checks. A few-shot prompt is ready for broader use when its labels are understandable, its edge cases have an explicit path, its output is machine-readable, and its performance has been tested on data the prompt did not see.