Over the past several years, second-opinion consultations and reassessments have repeatedly put reports with Clinical Assessment of Pragmatics (CAPs) results on my desk that simply did not make clinical sense. I am not talking about one borderline score or a child who happened to perform better than expected on one task. I am talking about profiles showing broadly average pragmatic performance in students whose actual communication was characterized by markedly flat prosody, limited facial affect, disorganized explanations, difficulty interpreting irony or sarcasm, weak perspective taking, or difficulty producing a socially appropriate first-person response when the task required the student to perform the interaction.
Sometimes I knew the student directly and had extensive longitudinal data. In other cases, the inconsistency appeared across the rest of the evaluation. A student might earn a very weak score on another social-communication measure, produce a highly disorganized narrative with obvious perspective-taking and inferential breakdowns, and still be reported as completely average on the CAPs. A discrepancy does not prove that the CAPs was scored incorrectly. It does, however, require reconciliation. When the examiner’s score cannot be reconciled with the student’s documented response and the applicable scoring criteria, it raises a legitimate scoring-fidelity question.
The CAPs itself is not the problem
I reviewed the CAPs favorably when it became available because its design addresses a real weakness in pragmatic assessment: social communication is dynamic. The CAPs uses video-based situations and separates pragmatic judgment from pragmatic performance, including social context, nonverbal decoding, affective expression, prosody, and other paralinguistic features (Elleseff, 2018; Lavi & Mainess, 2019). That is more ecologically informative than asking a child to recite rules of politeness from static pictures or decontextualized vignettes.
The initial published validation study reported strong test-retest reliability and strong interrater reliability across the CAPs subtests in a small adolescent sample (Lavi, Mainess, & Daher, 2016). Those data are encouraging, but they do not mean that every real-world administration is automatically reliable. Standardized scores are only interpretable when examiners follow standardized administration and scoring procedures. The major professional testing standards explicitly require adherence to the developer’s administration and scoring directions (AERA, APA, & NCME, 2014).
What makes me question CAPs scores
I become concerned when the CAPs profile conflicts sharply with other direct evidence and the report does not explain the discrepancy. Examples include:
- A student is described across settings as monotone with minimal facial affect, yet earns average or above-average scores on expressive CAPs tasks that require the student to actually demonstrate prosodic and nonverbal modulation.
- A student demonstrates clear difficulty with sarcasm, implied meaning, perspective taking, or nonverbal cue interpretation in interview, discourse, or therapy, yet receives an entirely average CAPs profile without qualitative explanation.
- A narrative or conversational sample shows severe problems with organization, inference, contextual interpretation, reciprocity, or perspective taking, yet the CAPs is reported as normal. Pragmatic assessment should not be reduced to one standardized score, and discourse/narrative data are part of a comprehensive social-communication evaluation (Adams, 2002).
- The report contains few or no examples of the responses the child actually produced, with little or no verbatim response documentation. If nobody recorded what the child actually said and did, later review becomes guesswork.
None of these patterns, by itself, proves examiner error. Together, however, they are exactly the kind of internal inconsistency that should trigger a scoring audit.
Where CAPs scoring can go wrong
1. The child’s response is not written down verbatim.
If the examiner paraphrases, cleans up, or reconstructs the response after the fact, an independent reviewer cannot determine whether the item-level rubric was actually met. Pragmatic responses are especially vulnerable to “close enough” scoring because a socially reasonable answer is not always the answer required for a particular score level.
2. A correct yes/no judgment is treated as full credit.
On the appraisal and decoding portions of the CAPs, recognizing that something went wrong is only part of the task. Depending on the subtest and item, higher credit also depends on the explanation, the relevant actions of the people in the interaction, or the nonverbal evidence used to justify the judgment. A student who senses that a scene is awkward but cannot explain why has not demonstrated the same skill as a student who accurately identifies the social mechanism.
3. The examiner gives credit for the words but ignores how the child actually says them.
This is the error I find most concerning on expressive tasks. Pragmatic performance is not scored from the transcript alone. Prosody, facial affect, and the quality of the communicative performance are part of the construct. A child can produce socially appropriate words in a flat, mismatched, or emotionally incongruent manner. If the scoring target includes prosody or facial expression, the examiner has to score the behavior that was actually demonstrated, not the behavior the examiner imagines the child intended.
4. The examiner overprompts.
Standardized assessment is not treatment. Neutral clarification is different from cueing the answer. As with any standardized assessment, clinicians should avoid adding content-leading support that changes the task being administered. Additional explanations, choices, models, or cues may alter what is being measured. Clinicians should follow the CAPs examiner materials exactly regarding any permitted clarification or additional prompting. Once prompting changes the task demands, the standard score becomes harder to defend.
5. The examiner applies a global rule instead of the item-level rubric.
The opposite problem also occurs. A broad word such as “mean,” “rude,” “nice,” or “sad” should not be automatically accepted or rejected across every item. The official item-level scoring criteria control. A response can be adequate on one item and insufficient on another because the required pragmatic explanation differs.
6. First-person requirements are treated inconsistently.
Some expressive CAPs tasks require the examinee to answer as the speaker rather than describe what another person should say. The safest practice is to follow the official item-level directions and any authorized repair prompt exactly. Do not create a blanket first-person rule across all items, and do not ignore an item-specific requirement when it is part of the scoring rule.
Why response capture matters
If a CAPs score is questioned after the fact, a protocol containing only circled item scores may be impossible to audit. This is why I believe response capture needs to become routine. At minimum, examiners should record the student’s verbal response verbatim and document the observed prosody, facial affect, gesture, and other performance features that affected scoring.
When policy, consent requirements, and test-security rules permit it, a response-only audio recording can add an important layer of scoring fidelity. Audio allows a second qualified examiner to hear the exact wording, latency, tone, stress, intonation, and other prosodic features. Audio does not capture facial expression or gesture, so those features still require contemporaneous examiner documentation. If video of the examinee is permitted, it can preserve more of the performance, but it also creates greater privacy and test-security obligations.
There is an important test-security limitation here. WPS currently emphasizes that proprietary test materials must be protected from reproduction and broad dissemination, and its current online terms prohibit entering WPS test items, prompts, responses, scoring, or other assessment content into generative AI systems (WPS, 2026a, 2026b). A recording used for scoring fidelity should therefore be structured, where feasible, to capture the examinee’s response rather than reproduce secure test stimuli. Recordings should never be posted publicly or distributed as a substitute for the official test materials. Any recording procedure should be designed so that proprietary test stimuli, video content, examiner scripts, and other secure assessment materials are not captured or reproduced.
What parents can ask for before testing
Parents can ask the school or evaluator, in advance, whether the child’s CAPs responses can be audio-recorded for scoring fidelity and retained with the evaluation record. That is a request, not an automatic federal entitlement to force the school to create a recording. IDEA and FERPA provide parents rights to inspect and review education records that a school collects, maintains, or uses, including records connected with evaluation, but those laws generally do not require a school to create a new record simply because a parent wants one (34 C.F.R. § 300.613; U.S. Department of Education, 2026). If a recording is created and maintained as part of the evaluation record, access rights may apply, subject to privacy, state law, district policy, and test-security restrictions.
| Possible parent request: “Before the CAPs is administered, I am requesting that my child’s spoken responses and prosody be captured in a response-only audio recording, if permitted by district policy, applicable consent requirements, and publisher test-security rules, so that scoring can be reviewed if there is later disagreement. If recording is not permitted, please document the child’s responses verbatim and record the prosodic and nonverbal behaviors used in scoring.” |
The larger point
The CAPs can provide clinically useful information precisely because it asks students to do more than identify a social rule. It asks them to integrate context, intention, language, prosody, facial expression, and nonverbal information. But that strength disappears if the examiner scores only the words, overprompts, fails to document the response, or assigns credit based on what the child probably meant.
If the CAPs score does not fit the child’s observed communication, do not discard the score automatically, and do not accept it automatically. Go back to the administration. Review the exact response. Review the prompt that was given. Review the item-level rubric. Review the actual prosody and nonverbal performance. Then reconcile the CAPs with narrative, conversational, language, and functional data. That is what defensible assessment requires.
Note: This post is educational and is not legal advice. School recording rules, consent requirements, FERPA/IDEA implementation, state law, and publisher test-security policies should be checked before recording an assessment.
References
- Adams, C. (2002). Practitioner review: The assessment of language pragmatics. Journal of Child Psychology and Psychiatry, 43(8), 973-987.
- American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
- Elleseff, T. (2018, December 10). Test review: Clinical Assessment of Pragmatics (CAPs). Smart Speech Therapy. https://www.smartspeechtherapy.com/test-review-clinical-assessment-of-pragmatics-cap/
- Lavi, A., Mainess, K. J., & Daher, N. S. (2016). Clinical Assessment of Pragmatics (CAPs): A validation study of a video-based test of pragmatic language in adolescent students. Autism Open Access, 6(2), 172.
- Lavi, A., & Mainess, K. J. (2019). A comparative analysis of social (pragmatic) communication skills in students with high-functioning autism and pragmatic language impairment using video-based assessments. Autism Open Access, 9(2), 239.
- U.S. Department of Education, Office of Special Education Programs. (n.d.). 34 C.F.R. § 300.613, Access rights. https://sites.ed.gov/idea/regs/b/f/300.613
- U.S. Department of Education, Student Privacy Policy Office. (2026). Inspect and review. https://studentprivacy.ed.gov/resources/inspect-and-review
- Western Psychological Services. (2026a). WPS position statement regarding test security. https://www.wpspublish.com/media/content-wps/WPS_Test_Security_Position_Statement_03.12.2026.pdf
- Western Psychological Services. (2026b). Terms of use, WPS Online Evaluation System. https://platform.wpspublish.com/Home/TermsOfUse