Precision is the only metric that can be measured in a typical literature review
A highly precise search strategy should be suspicious because the search may not be comprehensive enough
Based on the SYNERGY dataset, average precision is 2% - 4% in medicine, chemistry, and computer science
Question: Would you expect more precise searches in disciplines like Information Systems, Management, or the Social Sciences?
Literature Review Seminar
Literature search: Outlook
Open challenge:
How can we iterate efficiently?
How do we justify the decision to terminate a search?
How can we use evidence to search effectively?
How can we make progress without database providers?
Literature Review Seminar
Screen
The screen is typically completed in two parts:
A pre-screen based on metadata ("include if in doubt")
A screen based on full-text documents, resulting in the final sample
The screen is often based on explicit inclusion and exclusion criteria
Literature Review Seminar
Screening reliability
Screening tasks are often split among the review team to complete the process more quickly, and to ensure reliable decisions.
Process:
Definition of criteria, training, and pilot test
Parallel-independent screen (partially or fully overlapping sample)
Independent screen of the remaining papers (if any)
Reconciliation: in case of disagreements, final decisions are made by selected team members (often more senior researchers)
Calculate inter-rater agreement (e.g., Cohen's Kappa) and report the process
Literature Review Seminar
Interpretation of Kappa Values
Kappa Value Range
Interpretation
≤ 0
No agreement
0.01 – 0.20
None to slight
0.21 – 0.40
Fair
0.41 – 0.60
Moderate
0.61 – 0.80
Substantial
0.81 – 1.00
Almost perfect agreement
Note: When data is skewed—meaning one category is much more common than others—the Kappa statistic can be artificially low even if there is a high level of agreement. This occurs because Kappa adjusts for the level of agreement that would be expected purely by chance. In skewed distributions, the expected chance agreement tends to be high, which lowers the Kappa score. Essentially, in skewed distributions, even a relatively high observed agreement may not lead to a high Kappa, as the metric accounts for the imbalance.
Literature Review Seminar
Reporting the search and screen
The PRISMA flow chart (updated version by Tricco et al. 2018)
The reading activities can be organized strategically at two levels:
The overall corpus level: In which order should papers be read or skimmed?
The individual paper level: How should the different parts of a paper be read?
Question: Assume you have 300 papers to cover, how would you organize the reading activities?
Literature Review Seminar
Data analysis
Key differences with regard to data extraction and analysis:
Focus on metadata vs content
Inductive vs deductive reasoning
Note: The distinction between inductive and deductive modes of reasoning has critical implications. For instance, it would be incoherent to present an inductive analysis with inter-coder reliability assessment, or a deductive analysis without a pre-defined coding schema.
Literature Review Seminar
Data analysis example: Metadata profiling (example)
Literature Review Seminar
Data analysis example: Co-citation analysis
Note: Information Systems journals do not publish many scientometric papers. MIS Quarterly had an explicit policy against these types of analyses, but Information Systems Research has published co-citation analyses.
Note: For non-experimental studies, other domains of bias may apply (such as the use of fixed-effects for years as a control for omitted time-varying confounders/endogeneity).
Note: It is good practice to analyze whether results differ between high and low quality studies (e.g., through subgroup analyses) instead of excluding low-quality studies.
Literature Review Seminar
Quality appraisal / Risk of bias assessment (II)
Literature Review Seminar
Data analysis: Data extraction
Research objective: "to assess the effects of Fitbit-based interventions, compared with non-wearable control groups, on healthy lifestyle outcomes." (Ringeval et al. 2020)
Type of primary studies: Randomized clinical trials (RCTs), as illustrated in the CONSORT flow diagram
Data analysis: Forest plot of standardized mean differences
Literature Review Seminar
Data analysis: Meta-analysis
We extract or calculate Standardized Mean Differences (SMD):
Pooled standard deviation:
SMD is also known as Cohen's d. For small sample sizes, the corrections of Hedge's g should be used.
Note: For research models, we will typically rely on correlations as effect sizes (beta coefficients depend on the other variables of the model).
Literature Review Seminar
Random Effects Meta-Analysis
We assume the true effect size varies between studies:
Weighted average of effects:
Weights (account for variance):
: between-study variance
: standard error of each SMD
Interpretation: Larger = more influence on pooled estimate. Output: Overall effect size with 95% CI shown in forest plot.
The Doing Meta-Analysis in R book by Harrer et al. offers a good overview of meta-analysis methods.
Literature Review Seminar
Discussion of the data analysis section
Task: Create a quick draft for the data extraction and analysis section.
Would you follow an inductive or deductive approach (why)?
Task: Select an exemplary review and fill out the PRISMA checklist to assess the transparency of reporting.
Literature Review Seminar
References
Generic steps
Okoli, C. (2015). A guide to conducting a standalone systematic literature review. Communications of the Association for Information Systems, 37. doi:10.17705/1CAIS.03743
Boell, S. K., & Cecez-Kecmanovic, D. (2014). A hermeneutic approach for conducting literature reviews and literature searches. Communications of the Association for information Systems, 34, 12. doi:10.17705/1CAIS.03412
Templier, M., & Pare, G. (2018). Transparency in literature reviews: an assessment of reporting practices across review types and genres in top IS journals. European Journal of Information Systems, 27(5), 503-550. doi:10.1080/0960085X.2017.1398880
Literature Review Seminar
Problem formulation
Alvesson, M., & Sandberg, J. (2011). Generating research questions through problematization. Academy of Management Review, 36(2), 247-271. doi:10.5465/amr.2009.0188
Search
Gusenbauer, M., & Haddaway, N. R. (2021). What every researcher should know about searching–clarified concepts, search advice, and an agenda to improve finding in academia. Research Synthesis Methods, 12(2), 136-147. doi:10.1002/jrsm.1457
Hiebl, M. R. (2023). Sample selection in systematic literature reviews of management research. Organizational Research MNethods, 26(2), 229-261. doi:10.1177/109442812098685
Knackstedt, R., & Winkelmann, A. (2006). Online-Literaturdatenbanken im Bereich der Wirtschaftsinformatik: Bereitstellung wissenschaftlicher Literatur und Analyse von Interaktionen der Wissensteilung. Wirtschaftsinformatik, 1(48), 47-59. doi:10.1007/s11576-006-0006-1
Literature Review Seminar
Wagner, G., Prester, J., & Paré, G. (2021). Exploring the boundaries and processes of digital platforms for knowledge work: A review of information systems research. The Journal of Strategic Information Systems, 30(4), 101694. doi:10.1016/j.jsis.2021.101694
Screen
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., ... & Straus, S. E. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Annals of Internal Medicine, 169(7), 467-473. doi:10.7326/M18-0850
Literature Review Seminar
Data analysis
Wolfswinkel, J. F., Furtmueller, E., & Wilderom, C. P. (2013). Using grounded theory as a method for rigorously reviewing literature. European Journal of Information Systems, 22(1), 45-55. doi:10.1057/ejis.2011.51
Higgins J, Savovic J, Page MJ, Elbers RG, Sterne JA. Chapter 8: Assessing risk of bias in a randomized trial. In: Cochrane Handbook for Systematic Reviews of Interventions. London: Cochrane; 2019. link
Lacity, M. C., Solomon, S., Yan, A., & Willcocks, L. P. (2011). Business process outsourcing studies: a critical review and research directions. Journal of Information Technology, 26, 221-258. doi:10.1057/jit.2011.25
Ringeval, M., Wagner, G., Denford, J., Paré, G., & Kitsiou, S. (2020). Fitbit-based interventions for healthy lifestyle outcomes: systematic review and meta-analysis. Journal of Medical Internet Research, 22(10), e23954. doi:10.2196/23954
TODO : present a clear Gioia structure as a result
**Small-sample correction (Hedges’ *g*):**
$$
g = d \times \left(1 - \frac{3}{4(n_1 + n_2) - 9} \right)
$$
Use **Hedges' g** when sample sizes are small.
Also calculate **SE** to determine study weights.