How does NAEP reliably score and process millions of student-composed responses?

While multiple-choice questions allow students to select an answer from a list of options, constructed-response questions require students to provide their own answers. Qualified and trained raters score constructed-response questions.
Scoring a large number of constructed responses with a high level of reliability and within a limited time frame is essential to NAEP's success. (In a typical year, over three million constructed responses are scored.) To ensure reliable, quick scoring, NAEP takes the following steps:

• develops focused, explicit scoring guides that match the criteria delineated in the assessment frameworks;
• recruits qualified and experienced scorers, trains them, and verifies their ability to score particular questions through qualifying tests;
• employs an image-processing and scoring system that routes images of student responses directly to the scorers so they can focus on scoring rather than paper routing;
• monitors scorer consistency through ongoing reliability checks;
• assesses the quality of scorer decision-making through frequent monitoring by NAEP assessment experts; and
• documents all training, scoring, and quality control procedures in the technical reports.

NAEP assessments generally contain both constructed-response and multiple-choice questions. The constructed responses are scored using the image-processing system, whereas the responses to the multiple-choice questions are scored by scanning the test booklets.