Instructional Technology, Evaluation of

The Evolution of School-Based it Evaluation

When desktop technology became affordable enough for K-12 schools to buy in bulk, the emphasis was on building school, district, and statewide technology infrastructures. As these networks emerged, implementation was the initial interest for evaluators and their stakeholders. Evaluations focused on technology implementation applied a variety of strategies (e.g., the Computer Aided Education and Training Initiative). These studies attempted to determine the tractability of technology implementation and document how challenges were addressed. “Adoption of Innovation” studies applied the instrumentation of proven models (e.g., Apple Computers of Tomorrow) to compare IT project development to the blueprint of efficient and comprehensive technology adoption in schools.

Eventually, evaluations focused more intensively on such IT components as professional development, equity of access, integrative capacity, learner control, and technical support. These evaluations provided important project-level information on the formative development of school-based IT systems. As useful as these evaluation data were to stakeholders at the time, however, little information was directly linked to student achievement. Because of the enormous capital invested in IT, demonstrating technology's vital effectiveness has become a political, economic, and public policy necessity.

In focusing on learning impact, evaluators are cautious about treating technology as a discrete and isolated entity. Because technological applications evolve so quickly and because these evolutions limit the utility of past evaluations, careful IT evaluation embeds technology use in the larger process of school change. Still, pinpointing IT-induced learning outcomes is problematic. Evaluators often find the effectiveness of IT fixed in the effectiveness of other school improvement efforts. Disentangling technology-induced learning outcomes from learning resulting from other school programs is difficult and requires evaluation methodologies and data sources unique from oft-used survey and observation. When standardized test scores are used as a measure of learning impact, evaluators find they provide limited formative information with which to drive the development of a school's technology program. Consequently, alternative forms of documenting learning impact are needed.

A hybrid form of school-based IT evaluation suggests that evaluation apply three levels of indicators.

[Page 205]Leading indicators emerge from the first stages of an evaluation and address implementation and curriculum development concerns. A next level of indicators shows impact at an intermediate point in the form of behavioral indicators and performance on curriculum-embedded measures. A trailing set of indicators speaks to policy makers and the public about standardized tests and graduation rates. This distributed evaluation approach places the responsibility for evaluation at multiple levels and uses local capacity (teachers, schools) to identify and collect data. It is an approach with the potential to get formative evaluation data quickly into client hands, be flexible to types of indicators and measures, and be more resistant to criticisms than were previous evaluations.

...