assessment-validity-checker

The saved excerpt is a snapshot from review. The external source remains the complete and most current version.

---
# AGENT SKILLS STANDARD FIELDS (v2)
name: assessment-validity-checker
description: "Audit a proposed assessment for construct validity, reliability, and alignment to learning objectives. Use when reviewing or quality-assuring assessments before deployment."
disable-model-invocation: false
user-invocable: true
effort: medium

# EXISTING FIELDS

skill_id: "curriculum-assessment/assessment-validity-checker"
skill_name: "Assessment Validity Checker"
domain: "curriculum-assessment"
version: "1.0"
evidence_strength: "strong"
evidence_sources:
  - "Wiliam (2011) — Embedded Formative Assessment"
  - "Messick (1989) — Validity in educational measurement: a unified validity framework"
  - "Kane (2006) — Validation as argument-based approach"
  - "Brookhart (2003) — Developing measurement theory for classroom assessment purposes and uses"
  - "Stobart (2008) — Testing Times: the uses and abuses of assessment"
input_schema:
  required:
    - field: "assessment_description"
      type: "string"
      description: "Description of the proposed assessment — what students do, how it is marked"
    - field: "intended_learning"
      type: "string"
      description: "What the assessment claims to measure"
    - field: "student_level"
      type: "string"
      description: "Age/year group"
  optional:
    - field: "subject_area"
      type: "string"
      description: "The curriculum subject"
    - field: "assessment_purpose"
      type: "string"
      description: "Formative, summative, diagnostic, or evaluative"
    - field: "marking_approach"
      type: "string"
      description: "How the assessment will be marked — rubric, mark scheme, holistic judgement"
    - field: "stakes"
      type: "string"
      description: "The consequences of the assessment — low stakes (informing teaching), high stakes (grading, reporting)"
output_schema:
  type: "object"
  fields:
    - field: "validity_analysis"
      type: "object"
      description: "Analysis of construct validity, content validity, and consequential validity"
    - field: "reliability_analysis"
      type: "string"
      description: "Analysis of inter-rater reliability and internal consistency"
    - field: "threats_identified"
      type: "array"
      description: "Specific threats to validity with explanations"
    - field: "recommendations"
      type: "array"
      description: "Specific modifications to address identified threats"
chains_well_with:
  - "criterion-referenced-rubric-generator"
  - "formative-assessment-technique-selector"
  - "competency-unpacker"
  - "backwards-design-unit-planner"
  - "curriculum-knowledge-architecture-designer"
  - "kud-knowledge-type-mapper"
teacher_time: "3 minutes"
tags: ["validity", "reliability", "assessment-design", "quality-assurance", "measurement"]
---

# Assessment Validity Checker

## What This Skill Does

Evaluates a proposed assessment against three dimensions: validity (does it measure what it claims to measure?), reliability (would different markers agree on the score?), and authenticity (is the task meaningful and does it require genuine demonstration of the intended learning?). The output identifies specific threats to validity — construct-irrelevant variance (the assessment measures something other than what it claims), construct underrepresentation (the assessment doesn't cover enough of what it claims to measure), and consequential validity problems (unintended negative effects of the assessment) — and provides specific, actionable recommendations for each threat. AI is specifically valuable here because most teacher-designed assessments contain validity threats that are invisible without explicit analytical frameworks — a teacher designing a "reading comprehension" test may inadvertently create a writing test, or a "science understanding" assessment may actually test literacy.

## Evidence Foundation

Messick (1989) unified the concept of validity into a single framework: validity is not a property of a test but of the int