Assessments through Computer Analyses of Language, Discourse, and Conversation

Arthur C. Graesser
Department of Psychology and Institute for Intelligent Systems, University of Memphis
Honorary Research Fellow, Department of Education and Oxford University Center of Educational Assessment, University of Oxford

Abstract
This presentation reports research on automated text analysis systems that analyze edited texts, student writing, speeches, and conversations. The primary focus will be on a system called Coh-Metrix (http://cohmetrix.com and tea.cohmetrix.com), which analyzes texts on multiple measures of language and discourse that are aligned with multilevel theoretical frameworks of discourse comprehension and production.  Several dozen measures funnel into five major factors that systematically vary as a function of types of texts (e.g., narrative versus informational) and grade level: narrativity, syntactic simplicity, word concreteness, referential cohesion, and deep (causal) cohesion. We have recently examined a composite measure called formality, which increases with low narrativity, syntactic complexity, word abstractness, and high cohesion.  The assignment of texts to students is one of the central concerns of teachers, principals, superintendents, and other experts in educational policy. Students sometimes need to be challenged by assigning texts on difficulty levels that aggressively push the envelope on what they can handle. At other times students need a self-confidence boost by receiving easy texts they can readily comprehend. An alternative text assignment policies would assign texts that are not too difficult or too easy, but at an intermediate zone of difficulty. Yet another policy is to have a balanced diet of texts on the difficulty dimension, with adequate scaffolding for the difficult texts.  Coh-Metrix has also been used to analyze student writing and conversation.  However, other automated systems have been developed that are more appropriate for these registers of discourse that have minimal or no editing and that tend to have many misspelled words, ungrammatical sentences, and low cohesion. These alternative systems include AutoTutor, Operation ARA, and Writing-Pal in addition the essay graders that are being used at Pearson Education and Educational Testing Service.  The presentation will discuss the grading of writing and conversations in addition to the grading of edited texts.