Learning objectives

• Familiarize with the scientific discourse on sourcing knowledge-intensive services from platform markets.
• Outline different ways in which literature review methods can contribute to scientific progress.

Digital markets for knowledge-intensive services

Platforms: Upwork, Toptal, Topcoder

Examples of tasks: Programming, design, app development, data science, customer services, translation, and digital marketing.

Economic potential:

• 72 million jobs by 2025 (Manyika et al. 2015)
• Effects of reducing restrictions on emigration: Trillion-dollar bills on the sidewalk? (Clemens, 2011)

Open questions

There are many open questions in the context of digital markets for knowledge-intensive services, including:

• How can platforms for knowledge-intensive services be distinguished?
• Which steps does the process of working on these platform involve?
• What are the determinants of success from a worker perspective (e.g., being selected for a project)?

In the following, we will see how literature reviews help to address these questions.

Literature searches

Different search techniques are available: Database searches, citation searches, table-of-content searches

For database searches, the building-blocks approach is the typical query structure:

• Corresponding search query: (digital OR online OR ...) AND (work OR labor OR ...) AND (productivity OR effectiveness OR ...)

Note: Ideally, literature searches rely on controlled vocabulary (e.g., the Medical Subject Headings, MeSH).

Challenges in the search

In the social sciences (including Information Systems and the Management disciplines), there is no controlled vocabulary, and researchers may try to introduce new terms.

Examples of terms used for our context:

• online labor markets, digital labor markets,
• microsourcing, online outsourcing,
• macro-tasks, human computing, skilled labor,
• online freelancing, IT service work, gig work.

Note : Example based on Wagner, Prester, Paré (2021)

Effective searches with heterogeneous terminology

Initial guidelines:

1. Report whether controlled vocabulary is available (raise awareness of the challenge)
2. Systematically collect prior knowledge, including keywords used in prior literature reviews, and consult with colleagues
3. Rely on search techniques that are robust to heterogeneous terminology, including citation searches, tables-of-contents searches
4. Complete several search iterations and look for new search terms
5. Use evidence from previous iterations to identify relevant keywords and construct queries *

* Focus of our current research projects.

Screening

The screening process is often split into two stages:

• A prescreen, which is based on meta data (titles and abstracts) and tends to be more inclusive
• A screen, which is based on full-text documents and tends to be more restrictive

The screen often involves specific inclusion and exclusion criteria.

How would you screen if there is no established definition for our phenomenon (digital platforms for knowledge-intensive services)?

Development of boundary conditions

Exercise

• Form groups of 3 students.
• Read through the extracts of process steps, sequences, and models.
• Develop a process model to summarize prior research.

Process model

Note : Example based on Wagner, Prester, Paré (2021)

Contracting decisions

• Contracting rates are low, especially for high-volume projects: Snir and Hitt (2003) report contracting rates of 60%, or even less than 10% for larger projects.
• Some antecedents (explaining variables) are contested. For example, some expect ratings to be the most important variable, while others disagree because ratings have minimal variance.
• Literature reviews can provide an overview of the evidence across prior studies.

Prior research on contracting decisions

We identified approx. 20 studies providing empirical evidence on contracting decisions

Challenge: Every empirical study has errors (Card, 2015, Lin et al. 2013):

• Sampling error (random selection from the population)
• Measurement error (random error and systematic errors caused by measurement instruments)
• Inflated p-values (large samples have excessive test power and detect negligible effects)

In addition, research with non-significant findings rarely gets published, leading to publication bias (the file-drawer problem).

Literature reviews: Aggregating evidence

Specific types of literature reviews, aimed at theory testing, can be used to aggregate the empirical evidence. These include (qualitative) systematic reviews, vote-counting studies, and meta-analyses.

They rely on rigorous methods, including

• Comprehensive searches (including unpublished research)
• Systematic screening (conducted in parallel independent mode and based on explicit criteria)
• Quality appraisal (to identify risk of bias)
• Data extraction (conducted in parallel independent mode and based on a predefined data extraction schema)
• Meta-analytic regressions (or vote-counting)

Taken together, these steps can address shortcomings of empirical studies.

Model for selection decisions (preliminary)

Prester and Wagner (2021)

Principal findings

Workers are more likely to receive contracts when:

• the price is lower
• they have earned a badge (platform certification)
• they have a prior working relationship with the client
• they are located in the same country

It is important to recognize that despite many studies (and many significant findings), very few antecedents have a substantial effect.

Conclusion

• Digital platforms for knowledge-intensive services: Boundary conditions, process, and model of contracting decisions.
• Literature review contributions: Synthesizing prior work across heterogeneous terminology and seemingly incompatible process descriptions. Aggregating evidence to overcome the shortcomings of individual empirical studies.

Materials

Card, N. A. (2015). Applied meta-analysis for social science research. Guilford Publications.

Clemens, M.A., 2011. Economics and Emigration: Trillion-dollar Bills on the Sidewalk? Journal of Economic Perspectives 25 (3), 83–106.

Lin, M., Lucas Jr, H. C., & Shmueli, G. (2013). Research commentary—too big to fail: large samples and the p-value problem. Information Systems Research, 24(4), 906-917.

Manyika, J., Lund, S., Robinson, K., Valentino, J., Dobbs, R., 2015. A Labor Market that Works: Connecting Talent with Opportunity in the Digital Age. McKinsey Global Institute.

Prester, J., & Wagner, G. (2021). Contracting Decisions on Digital Markets for Knowledge Work Services: A Qualitative Systematic Review. In: Proceedings of the International Conference on Information Systems.

Snir, E. M., and Hitt, L. M. 2003. “Costly Bidding in Online Markets for IT Services,” Management Science (49:11), pp. 1504–1520.

Wagner, G., Prester, J., & Paré, G. (2021). Exploring the boundaries and processes of digital platforms for knowledge work: A review of information systems research. The Journal of Strategic Information Systems, 30(4), 101694.