Tools to Support Systematic Literature Reviews in Software Engineering: Protocol for a Feature Analysis

Work in Progress
Chris Marshall; Pearl Brereton

Systematic literature reviews (SLRs) have become an established methodology in software engineering research (Kitchenham et al., 2009). SLRs are concerned with rigorously evaluating empirical evidence from a variety of relevant literature, in an attempt to address a particular issue or topic of interest (Kitchenham & Charters, 2007). SLRs differ from ordinary literature reviews in being formally planned and methodically executed (Staples & Niazi, 2007). Undertaking an SLR comprises several discrete stages that can be grouped into three core phases: planning, conducting and reporting.

Despite their usefulness, conducting an SLR remains a highly manual, error prone and labour intensive process (Riaz et al., 2010; Babar & Zhang, 2009; Brereton et al., 2007). Automated support to assist the various stages of the SLR process may help to overcome some issues and so we are currently investigating the use and effectiveness of tools to support the conduct of SLRs in software engineering.

A literature review, in the form of a mapping study, identified tools that have been developed or used to provide automated support for the SLR process and established the extent to which the tools have been evaluated. The mapping study, after applying inclusion/exclusion criteria, accepted 14 papers into the final set. A variety of approaches and support tools which had been developed to assist the conduct of an SLR were found. During the next phase of the research project, using the Feature Analysis approach, four existing tools identified by the mapping study will be evaluated.

Feature Analysis is a qualitative form of evaluation involving a subjective assessment of the relative importance of different features and how well features are implemented (Kitchenham et al., 1997). It is based on the requirements that users have for a particular task/activity and mapping those requirements to features that a tool aimed at supporting the task/activity should possess. (Kitchenham & Jones, 1997; Kitchenham et al., 1997). Building on the results of the mapping study, a Feature Analysis protocol (plan) is being developed. Four SLR tools will be subject to the Feature Analysis.

These tools have been developed from scratch, with the sole aim of supporting software engineering researchers throughout the entirety of a SLRs undertaking. Table 1 lists the four candidate tools for evaluation.

The Feature Analysis will be the first step toward the development of a rigorous evaluation framework for tools that support SLRs. The aims of this activity are to produce an initial set of features and user requirements for a tool of this type. Results generated will act as a foundation for more complex evaluation exercises to be undertaken. The Feature Analysis will be organised as an initial screening, and will only focus on evaluating simple features. Simple features relate to aspects that are either present or absent and are assessed by a simple YES/PARTLY/NO nominal scale (Kitchenham et al., 1997).

PPIG-WIP, Keele University, 2013

A protocol for the Feature Analysis is currently being developed and will be presented at PPIG-WIP 2013. The presentation will include an initial feature list, score sheet and judgment scales and a discussion of future research aims and objectives.





An open source web enabled database that supports the management of SLRs. (Bowes et al., 2012)


A freely-available tool for performing SLRs (Fernández-Sáez et al., 2010)


A tool that provides support to each of the activities, except the automated support of primary studies. (Hernandes et al., 2012)


An open-source, web-based, multi-user tool that supports the SLR process. (

Table 1. SLR support tools selected for the Feature Analysis 

Type of Publication: Paper
Conference: PPIG WIP Workshop 2013
Publication Year: 2013
Paper #: 10
TitleTools to Support Systematic Literature Reviews in Software Engineering: Protocol for a Feature Analysis
Publication TypePaper
AuthorsMarshall, C, Brereton, P
PPIG Workshop: