Session 2: Tools (teaching notes)

Start with a question on students’ current thoughts about tool selection in their review protocol.

Warm-up: have you worked with research software, what should the ideal literature review tools offer?

Colrev

  • OpenSource, extensible, validated, cost-efficient (open research software, e.g., R/Tidyverse, Machine Learning in Python, Visualization)
  • Code environment (low-code)

Explicitly show the colrev status

Show the data (that is what CoLRev relies on)

Invite students to contribute to CoLRev (documentation, testing, etc.)

LLMs, current challenges, and promises

How would you use genAI-tools in a literature review?

  • Follow-up: any application areas for genAI (beyond LLMs) - image/audio/video? - e.g., Googles NotebookLM

  • LLM like ChatGPT are seemingly easy to operate (simple interface), but generating useful output is surprisingly hard (a metaphorical Norman door)
  • litmaps

Which developments can be anticipated?

(not formally part of the main review steps) e.g., tabulating - give examples

effectively excluding over 90% of the information and only considering a few words of each PDF

-> we may even illustrate this with a whole paper and the title highlighted for screening

Philosophical questions

  • makes researchers obsolete
  • danger that it reduces deep engagement with prior literature (opportunity to preserve that ability)

Outlook

Create a reminder for the presentation session

Feedback

  • Ask students for issues that are good/should be improved
  • Mention 5-star ratings, and ask for feedback in the final evaluations

Thank you!

The nice thing about literature reviews is that there are many roads that may connect us (or colleagues)

  • Deep engagement (AI/generative AI? - reading not part of the process?)
  • How could ML/machines/generative AI facilitate a deeper understanding (instead of distancing reviewers from the literature)?

Wrap-up! Plans for submission (presentation?!?!)

Communicate the expectation that students attend at least 2-3 meetings

Feedback Ask participants to note down one item that was good (keep) and one that was not good (leave).

IDIS: announce next seminar next year/summer, ask students to recommend it