Evaluation Methods for IS Artifacts

Research seminar by Isabelle comyn-Wattiau at HEC Lausanne

Artifacts, such as software systems, pervade organizations and society. In the field of information systems (IS) they form the core of research. The evaluation of IS artifacts thus represents a major issue. Although IS research paradigms are increasingly intertwined, building and evaluating artifacts has traditionally been the purview of design science research (DSR). DSR in IS has not reached maturity yet. This is particularly true of artifact evaluation.

This paper investigates the “what” and the “how” of IS artifact evaluation: what are the objects and criteria of evaluation, the methods for evaluating the criteria, and the relationships between the “what” and the “how” of evaluation? To answer these questions, we developed a taxonomy of evaluation methods for IS artifacts. With this taxonomy, we analyze IS artifact evaluation practice, as reflected by ten years of DSR publications in the basket of journals of the Association for Information Systems (AIS). This research brings to light important relationships between the dimensions of IS artifact evaluation, and identifies seven typical evaluation patterns: demonstration; simulation- and metric based benchmarking of artifacts; practice-based evaluation of effectiveness; simulation- and metric-based absolute evaluation of artifacts; practice-based evaluation of usefulness or ease of use; laboratory, student-based evaluation of usefulness; and algorithmic complexity analysis. This study also reveals a focus of artifact evaluation practice on a few criteria. Beyond immediate usefulness, IS researchers are urged to investigate ways of evaluating the long-term organizational impact and the societal impact of artifacts.

ISABELLE COMYN-WATTIAU is a professor of information systems at ESSEC Business School. She received an M.S. and a Ph.D. in computer science from the University of Paris 6. Her research interests include information systems design, data warehouse design and redesign, and database integration. She has published more than seventy journal and conference papers on information and database systems. Her research has appeared in journals such as Decision Support Systems, Data and Knowledge Engineering, Expert Systems with Applications, and major academic conferences including the International Conference on Conceptual Modeling (ER) and the European Conference on Information Systems (ECIS).

Comments are closed.