Benjamin Dunford and Dr. Douglas Dean, Accountancy and Information Systems
Background
In system development, no other phase is as fraught with difficulties as determining what exactly the system “should do” for a proposed project. These difficulties arise through a divergence in how the end user envisions the system should operate and how the analyst or developer perceives the user would like the system to operate. The search for a common language that both analyst and end user could use to effectively communicate what a system “should do”, or the system requirements, gave rise to the use of scenarios. Scenarios are textual narratives of how the system should operate in a given situation. The analyst then uses the scenarios to define the requirements of the system under development. Although both analysts and users can communicate system requirements relatively easily using scenarios, the effectiveness of their communication is directly dependent upon the quality of the scenarios in question (3).
Current Research
Perhaps the most revealing aspect of the present body of research lies in how little understood scenarios are. Few individuals contest the intrinsic worth of scenarios, but little empirical evidence exists that suggests scenarios are more effective than other requirement elicitation methods, and little or no quantitative evidence exists that compares various methods of scenario elicitation (2). In hopes of gathering evidence in support of a particular methodology to elicit scenarios, this research takes a “back-door” approach—employ more precise methods to elicit scenarios, and measure the quality of those scenarios.
Experimental Design
The variety of scenario applications makes developing an experimental design difficult. In conjunction with my faculty mentor, we identified a web-based tool, ColD Spa, that prompts users to generate scenarios. The use of a computer-based application to elicit and record scenarios is an attempt to improve on the more traditional methods of elicitation—direct interviews and observing the environment for analysis (1). In testing elicitation methods, a computer-based tool provides control to test variables in the elicitation method. Additionally, if a computer-based scenario tool is found effective in producing high-quality scenarios, it could represent a significant improvement over the present labor-intensive process. With a web-based elicitation tool, the analyst doesn’t need to be present to allow the elicitation process to carry on.
Although the tool has been tested to adequately elicit scenarios, previous tests were performed with the users solely working with ColD Spa, with no outside interaction. In combination with the elicitation tool, the experimental variable of active facilitation is the next step of testing. This combination is a hybrid of traditional methods and technological advances. The first treatment will serve as a control and will consist of testing the tool in a structured environment with no further interaction, and the second treatment consists of introducing a tester to provide direct facilitation in the scenario elicitation process. This direct facilitation more closely mirrors the traditional interaction between analyst and user when eliciting scenarios.
Additional Research
Future research consists of marshalling the resources to carry out the experimental design. This would likely include the use of MBA students in a testing environment of the two abovementioned treatments. An additional area of research in the experimental design lies in the development of evaluation methods to measure the quality of the scenarios. For instance, one method may generate a great deal of textual description, which when measured quantitatively appears superior. However, such a scenario may also lack accuracy or completeness—a qualitative concern. This combination of factors compounds the difficulty of measuring the overall “effectiveness” of scenarios. The ideal scenario elicitation method should provide as much information as accurately as possible, in as little time as possible.
References
- Beyer, H. and Holtzblatt, K. Contextual Design: A Customer-Centered Approach to System Designs. Morgan Kaufman, NY, 1997.
- Jarke, Matthias, Scenarios for Modeling. Communications of the ACM (Jan. 1999), 47-48.
- McGraw, Karen & Harbison, Karan. User-Centered Requirements: The Scenario-Based Engineering Process. Lawrence Erlbaum Associates Publishers: Mahwah, New Jersey. 1997. 110