Retrieval-Conditioned Counterfactual Consistency for Classroom Discourse Analysis

-- A controlled study of sparse evidence selection and prediction stability

Abstract

Classroom discourse analysis depends on detecting pedagogically meaningful reasoning from turns whose relevant context may occur far earlier. This study evaluates a small-footprint retrieval-conditioned classifier augmented with an explicit counterfactual consistency penalty. We manually curated 136 publicly displayed, human-labeled student-response records from the TalkMoves corpus (47 labeled as providing evidence or reasoning). A controlled benchmark inserts each record’s original paired context among 8, 32, or 64 distractor turns sampled from the same public excerpt pool; these reconstructed histories are synthetic, whereas the classification labels are authentic. A character n-gram TF-IDF selector supplies one context to a word n-gram logistic classifier. We adapt the symmetric Kullback–Leibler consistency objective and Counterfactual Flipping Rate audit of the neuro-symbolic pedagogical alignment literature to a binary, CPU-feasible estimator. Across five grouped random splits, the 32-candidate setting attains Macro-F1 0.753 with consistency regularization and 0.753 without it, while mean flipping falls from 10.38% to 3.07%. Retrieval recall at one declines as context grows, and the utterance-only baseline remains competitive. Thus, the supported contribution is improved stability under specified orthographic perturbations, together with a reproducible measurement of retrieval failures—not a claim of superior full-lesson performance or demographic fairness.