How Scientific Journals WorkEffect of structured workshop training on subsequent performance of journal peer reviewers*
Introduction
High-quality reviewers are a crucial component in selecting quality science for publication. Most journals do not have objective methods of screening for reviewer quality, and not all reviewers are excellent. Training methods that would improve reviewer quality might be of value, but little is known about such methods. Instructional workshops are popular but require considerable logistic effort, reach only a small fraction of the potential audience, and are of unproven efficacy. We conducted 2 randomized trials to examine the efficacy of a structured half-day workshop format in improving subsequent review quality scores.
Section snippets
Materials and methods
All active peer reviewers at Annals of Emergency Medicine were screened for possible invitation to attend a formal, 4-hour, highly interactive workshop on peer review held simultaneously with a major meeting in the specialty and run by 2 senior editors of the journal (including a formally trained methodology and statistics editor). Exclusion criteria included attendance at a prior workshop, guest reviewer status, or membership on the journal's editorial board. Details of the workshop format are
Results
Twenty-five reviewers volunteered for the first course, were eligible for study, attended, and were compared with 25 matched control reviewers. Of attendees filling out evaluations, 19% thought it somewhat and 81% thought it very helpful. All thought it would improve their subsequent reviews, and 85% thought it would improve their review ratings. Fifty participants had sufficient data for analysis, completing 217 rated reviews before the workshop and 289 after the workshop. The quality scores
Discussion
Workshops to educate editors and reviewers are a common educational method used by universities, journals, and professional societies, such as the Council of Science Editors (www.councilscienceeditors.org ). Our journal had been offering such a course for more than 15 years, and it has been very popular. In 1998, we first assessed its effect on subsequent reviewer performance and found no benefit.3 We thought that lack of effect might be caused by the fact that its format was not sufficiently
References (12)
- et al.
Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts
J Clin Epidemiol
(1999) - et al.
Effect of attendance at a training session on peer reviewer quality and performance
Ann Emerg Med
(1998) - et al.
Teaching emergency medicine residents evidence-based critical appraisal skills: a controlled trial
Ann Emerg Med
(1999) - et al.
Reliability of editors' subjective quality ratings of peer reviews of manuscripts
JAMA
(1998) - et al.
Impact of a medical journal club on house-staff reading habits, knowledge, and critical appraisal skills
JAMA
(1988) - et al.
The effect of a medical journal club on residents' knowledge of clinical epidemiology and biostatistics
Fam Med
(1992)
Cited by (40)
Towards the enhancement of quality publication practices in clinical psychological science
2020, Behaviour Research and TherapyThe Effect of Peer Review on the Quality of Data Graphs in Annals of Emergency Medicine
2017, Annals of Emergency MedicineA systematic review highlights a knowledge gap regarding the effectiveness of health-related training programs in journalology
2015, Journal of Clinical EpidemiologyCitation Excerpt :Of the five included studies (see Appendix I at www.jclinepi.com), two RCTs [30,31] took place in the United States and the United Kingdom, whereas the remaining CBA [32,33] and BA [34] studies all took place in the United States. Participants included peer reviewers for major journals [30–33] and students in an undergraduate cell and molecular biology course for engineers [34]. Sample sizes ranged from 22 to 418 participants.
Longitudinal trends in the performance of scientific peer reviewers
2011, Annals of Emergency MedicinePeer Reviewer Training and Editor Support: Results From an International Survey of Nursing Peer Reviewers
2009, Journal of Professional Nursing
- *
Reprints not available from the authors. Address for correspondence: Michael L. Callaham, MD, Department of Emergency Medicine, University of California, Box 0208, San Francisco, CA 94143-0208; 415-353-5885, fax 415-353-1799; E-mail [email protected].