Scenarios are extensively used in software architecture evaluation.
These scenarios are elicited from stakeholders using either a topdown
or bottom-up approach. The former approach uses categorization
schemes to focus stakeholders on developing scenarios for each
required category. The latter approach uses brainstorming without
any explicit categories of scenarios. It is claimed that top-down
approach can result in improved quality of scenarios. However,
there has been no empirical evidence on the relative effectiveness of
the scenario elicitation techniques. In this paper we report on a controlled
experiment with 24 subjects (postgraduate and final year
undergraduate students with industry experience) in an academic
context with the goal to assess the relative effectiveness of the two
scenario elicitation approaches. Two groups developed scenarios to
characterize quality attributes: the treatment group was given software
change categories, the control group was not. The outcome
variable was the quality of the scenarios produced by each participant.
The average quality score for individual scenario profiles in
the treatment group was significantly greater than the control group.
All participants using the change categories reported that the knowledge
of change categories helped them develop better quality scenarios.
Our results support the claim that the provision of domainspecific
software change categories helps generate better quality