Assessment of competences in rheumatology
| Author, country, year | Berman JR, USA, 200913 | Kissin EY, USA, 201415 | Pascual-Ramos V, Mexico, 201512 | Pascual-Ramos V, Mexico, 201814 | Humphrey-Murto S, Canada, 200916 |
| Competence assessed | Clinical skills, communication skills, professionalism | Practical skills (MSUS) | Clinical skills | Clinical skills | CanMEDS roles |
| Tool* | OSCE | OSCE | OSCE | OSCE | DOPS† |
| Comparator tool | Rating by programme directors | MCQ (76-question) | Theory Test Board Exam (300 MCQ) | Theory Test Board Exam (200–222 MCQ) | OSCE (national 10-stations) |
| Sample size | 70 (number of participants to year 4 NA) | 35 | 68 | 80 | 73 |
| Workplace based or simulation based | Simulation based | Simulation based | Simulation based | Simulation based | Workplace based |
| Current practice (CP) or research purpose (RP) | CP | RP | CP | CP | RP |
| Control population | None | 3 Faculty members | 3 Certified rheumatologists | ≥6 Certified rheumatologists | None |
| Study duration | 5 years | NA | 2 years | 2 years | 18 months |
| Internal consistency (Cronbach’s α unless otherwise stated)‡ | – | r=0.15 | 0.83–0.92 | – | – |
| Inter-rater reliability | Correlation between raters not significant | ICC 0.3 between live assessors and assessors on videotapes ICC 0.7 between assessors on videotapes | – | – | – |
| Test–retest reliability/intrarater reliability | – | – | OSCE total and partial scores in Year 1 vs Year 2 r=0.80–0.95 | – | – |
| Intermethod (or parallel forms) reliability/concurrent validity | OSCE versus rating by programme directors r=0.48 | OSCE versus MCQ r=0.52 | OSCE total score versus MCQ Year 1: r=0.203; Year 2: r=0.436 | OSCE total score versus MCQ Year 1: r=0.277; Year 2: r=0.445 | DOPS versuss OSCE r=0.48 |
| Feasibility | – | – | – | – | 14 Forms per resident to achieve a g coefficient of 0.8 |
| Construct validity | – | OSCE scores significantly discriminated fellows and fellows versus faculty | Significantly higher scores in higher levels of training | – | – |
| Predictive validity | – | – | – | – | – |
| Risk of bias (MERSQI score) | Unclear (11) | High (9.5) | Low (14.5) | Unclear (10) | Low (12.5) |
*The number of OSCE stations was 6–8 of 12 min each13, 915, 12 or 15 of 8 min each.12 14.
†Rated with the ambulatory clinic evaluation form.
‡The range of the study12 indicates the values across the stations repeated in the 2 academic years.
DOPS, direct observation of practical skills; ICC, intraclass correlation coefficient; MCQ, multiple-choice questionnaire; MERSQI, Medical Education Research Study Quality Instrument; MSUS, musculoskeletal ultrasound; NA, not available; OSCE, objective structured clinical examination.