Elsevier

Annals of Emergency Medicine

Volume 40, Issue 3, September 2002, Pages 323-328
Annals of Emergency Medicine

How Scientific Journals Work
Effect of structured workshop training on subsequent performance of journal peer reviewers*

Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain, September 2001.
https://doi.org/10.1067/mem.2002.127121Get rights and content

Abstract

Study objective: We sought to determine whether peer reviewers who attend a formal interactive training session produce better reviews. Methods: Peer reviewers were invited to attend a formal, 4-hour, highly interactive workshop on peer review. Attendees received a sample manuscript to read and review in writing in advance. The workshop included presentations on analyzing a study and the journal's expectations for a quality review, discussion of the sample manuscript's flaws and how to address them in a review, discussion of the reviews written by the attendees, and discussion of real reviews of other manuscripts illustrating key points. The performance of attendees on the basis of standard editor quality ratings (1 to 5) was assessed for the 2 years after workshop attendance. Control reviewers matched for previous review quality and volume were selected from nonattendees of the workshop. In study 1, all average reviewers received a standard written invitation. In study 2, 75 randomly selected average reviewers were personally and actively recruited with intensive follow-up by means of e-mail and telephone calls in an effort to reduce self-selection bias. Results: In study 1, 25 reviewers volunteered for the course, were eligible for study, attended, and were compared with 25 matched control reviewers. Of attendees filling out evaluations, 19% thought it somewhat and 81% thought it very helpful. All thought it would improve their subsequent reviews, and 85% thought it would improve their review ratings. The mean change in rating after the workshop was 0.11 (95% confidence interval [CI] −0.25 to 0.48) for control reviewers and 0.10 (95% CI −0.20 to 0.39) for attendees. In study 2, of 75 reviewers intensively recruited, only 12 (41%) of those who said they would attend did. All of the participants thought the workshop would improve their performance and ratings. Test scores at the end of the workshop improved in 73% of participants compared with scores on pretests. The control reviewers' average rating changed by −0.10 (95% CI −0.49 to 0.29) versus 0.06 (95% CI −0.34 to 0.23) for attendees. Conclusion: Among invited peer reviewers, voluntary attendance at a highly structured and interactive workshop was low and did not improve the quality of subsequent reviews, contrary to the predictions of attendees. Efforts to aggressively recruit average reviewers to a second workshop were time consuming, had low success rates, and showed a similar lack of effect on ratings, despite improvement in scores on a test instrument. Workshop teaching formats, although traditional, are of unproven efficacy. [Ann Emerg Med. 2002;40:323-328.]

Author contributions: MLC conceived and designed the study, collected data, drafted the manuscript, and takes responsibility for the paper as a whole. DLS designed the study, analyzed the data, and revised the manuscript. Both authors designed and conducted the workshop.

Introduction

High-quality reviewers are a crucial component in selecting quality science for publication. Most journals do not have objective methods of screening for reviewer quality, and not all reviewers are excellent. Training methods that would improve reviewer quality might be of value, but little is known about such methods. Instructional workshops are popular but require considerable logistic effort, reach only a small fraction of the potential audience, and are of unproven efficacy. We conducted 2 randomized trials to examine the efficacy of a structured half-day workshop format in improving subsequent review quality scores.

Section snippets

Materials and methods

All active peer reviewers at Annals of Emergency Medicine were screened for possible invitation to attend a formal, 4-hour, highly interactive workshop on peer review held simultaneously with a major meeting in the specialty and run by 2 senior editors of the journal (including a formally trained methodology and statistics editor). Exclusion criteria included attendance at a prior workshop, guest reviewer status, or membership on the journal's editorial board. Details of the workshop format are

Results

Twenty-five reviewers volunteered for the first course, were eligible for study, attended, and were compared with 25 matched control reviewers. Of attendees filling out evaluations, 19% thought it somewhat and 81% thought it very helpful. All thought it would improve their subsequent reviews, and 85% thought it would improve their review ratings. Fifty participants had sufficient data for analysis, completing 217 rated reviews before the workshop and 289 after the workshop. The quality scores

Discussion

Workshops to educate editors and reviewers are a common educational method used by universities, journals, and professional societies, such as the Council of Science Editors (www.councilscienceeditors.org ). Our journal had been offering such a course for more than 15 years, and it has been very popular. In 1998, we first assessed its effect on subsequent reviewer performance and found no benefit.3 We thought that lack of effect might be caused by the fact that its format was not sufficiently

References (12)

There are more references available in the full text version of this article.

Cited by (40)

  • A systematic review highlights a knowledge gap regarding the effectiveness of health-related training programs in journalology

    2015, Journal of Clinical Epidemiology
    Citation Excerpt :

    Of the five included studies (see Appendix I at www.jclinepi.com), two RCTs [30,31] took place in the United States and the United Kingdom, whereas the remaining CBA [32,33] and BA [34] studies all took place in the United States. Participants included peer reviewers for major journals [30–33] and students in an undergraduate cell and molecular biology course for engineers [34]. Sample sizes ranged from 22 to 418 participants.

View all citing articles on Scopus
*

Reprints not available from the authors. Address for correspondence: Michael L. Callaham, MD, Department of Emergency Medicine, University of California, Box 0208, San Francisco, CA 94143-0208; 415-353-5885, fax 415-353-1799; E-mail [email protected].

View full text