Impact of Experience and Team Size on the Quality of Scenarios for Architecture Evaluation

S. Biffl, M. Ali Babar,D. Winkler:
"Impact of Experience and Team Size on the Quality of Scenarios for Architecture Evaluation";
Vortrag: Conference on Evaluation and Assessment in Software Engineering (EASE), Bari, Italy; 26.06.2008 - 27.06.2008; in:"Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (EASE)", G. Visaggio, M. Baldassarre, S. Linkman, M. Turner (Hrg.); (2008), S. 1 - 10.

[ Publication Database ]

Abstract:


Software and systems architecture is a success-critical issue in software projects. Changing nonfunctional
quality requirements, e.g., performance, modifiability, and maintainability, can have strong
impact on software architecture and can result in a high rework effort in case of changes. Architecture
reviews help evaluating architectural design with scenarios in early stages of product development.
Quality-sensitive scenarios represent a set of software requirements (including non-functional quality
attributes). In this paper we empirically investigate the impact of potentially important factors on the
number and quality of scenarios elicited in an architecture evaluation workshop: (a) scoring schemes
for scenario quality, (b) workshop participant experience, and (c) team size for workshop group work.
We report data analysis results from an empirical study where 24 reviewers at different experience
levels identified over 100 scenarios. Main findings are: (a) results of different scoring approaches
(frequency-based and expert scoring) agree very well regarding critical scenarios, (b) the scenario
elicitation method was more important than individual experience, and (c) adding a new person to a
team of size 3 or more increases scenario coverage by less than 10%.