INTRODUCTION

A rheumatologist is defined as a physician who has received further training in the diagnosis (detection) and treatment of musculoskeletal disorders and systemic autoimmune conditions, commonly referred to as rheumatic and musculoskeletal diseases.^{1 2} Rheumatology is recognised as a specialty or sub-specialty in most of the European League Against Rheumatism (EULAR) countries.³ However, the scope of rheumatology practice varies across countries.^4,6 Indeed, in some countries, rheumatologists focus on inflammatory joint and connective tissue diseases whereas in others, rheumatology covers a broader scope, including soft tissue lesions, fibromyalgia and rehabilitation.⁵

In order to become a rheumatologist, trainees must successfully complete a rheumatology training programme.^{2 6} Both the content and the assessments within these programmes are regulated by national authorities. Some initiatives aiming at harmonising training across countries of the European Union (EU) exist. The European Union of Medical Specialists, a professional body of representatives from medical specialities from the EU member states, has developed a general European curriculum with the competences to be achieved at the completion of training, including theoretical and clinical knowledge, practical skills and non-clinical competences.^{2 7}

Over the past years, in addition to this European curriculum, efforts have been made to gain insights and provide an in-depth analysis of the differences and similarities in national curricula and assessment methods across EULAR countries.^{6 8} In a prior study, a questionnaire was answered by young rheumatologists and trainees to assess the acquisition of competences during the training and other information not clearly stated in the curricula (eg, assessment of competences).^{6 8 9} Interestingly, while this approach provided useful information on some of the differences and similarities on training across countries, data on the assessment of competences were incomplete and several limitations hampered the interpretation of the findings.^{6 8} A further attempt was made to gather information from a principal investigator (PI) per country with a short questionnaire with open questions. This further highlighted the difficulty in obtaining useful and reliable information from a single person. Inquiring into assessments during rheumatology training leads to answers reflecting a personal experience and perception; therefore, a more comprehensive evaluation, obtained from different sources, is needed before a full picture can be obtained. For this reason, we decided on a qualitative approach in a representative selection of European countries.

The present study aimed at gathering information and in-depth views on the assessment methods of competences in rheumatology and the experiences around them, as well as underlying priorities for competence assessment through focus groups (FGs).

This qualitative approach will ultimately inform EULAR Points-to-Consider on the assessment of competences in rheumatology across Europe.

RESULTS

DISCUSSION

This work is, to our knowledge, the first qualitative study gaining insights into competence assessments in rheumatology across Europe. Online FGs are an innovative methodology and have shown to be a feasible method to involve discussants across a wide geographical area. Through this study, we highlighted interesting differences between countries regarding the current assessment of competences strategies and methods. While a portfolio was mandatory in all five countries, in a mostly electronic format, other assessment types, such as assessment of professionalism through multisource feedback, occur in a structured manner in only some countries. Portfolios were supported by trainers but were also felt to be burdensome by trainees, sometimes reduced to an ineffective, time-consuming checklist of requirements. Participants particularly valued portfolios which provided a framework to integrate all the required aspects of performance, often derived from the CanMeds approach. Where components were closely linked to the curriculum and included non-clinical competences, such as professionalism and teamwork, portfolios were seen as particularly useful and valid. Trainees were commenting on their current experience while compiling their portfolios and understandably expressed their current challenges and frustrations, which would be expected to become more balanced in retrospect.

Structured feedback took place regularly in all five countries but with a variable frequency. Providing feedback was highly valued by both trainers and trainees. One main difficulty, described by many FG participants, was the lack of protected time for giving and receiving feedback, which limits its feasibility. Indeed, giving feedback is a skilled educational task, particularly when related to attitudes and professionalism.¹³ Many trainers and trainees described how they valued or would welcome opportunities to develop their skills in a constructive and consistent approach, where feedback received regularly and in a positive manner could help catalyse changes in performance and motivation.

A prolonged oral test in a clinical setting was performed in only one country, and knowledge tests were generally felt to be neither effective nor desirable as key assessment methods at this stage in training. Indeed, there is evidence that summative oral exams should not be strongly relied on to assess competences and are not supported in the literature across medical education as the main approach to assessment for postgraduate education.^{14 15} In addition, knowledge tests might also be inappropriate at this level as assessment in advanced stages of training focuses on the acquisition of complex competences integrating knowledge, skills and attitudes.¹⁶

Interestingly, this work provides a summary of current practices, and the initial document provided by the country’s PI evolved with the interventions from FG participants. This allowed us to obtain a more comprehensive view of the current assessment strategies within specific countries. Indeed, reflections on assessment practices are not fully objective and include a perception and judgement on them. Moreover, recent reviews highlight person al and environmental factors influencing the way trainees perceive the developmental value of assessment, self-motivation also being reported as an important driver of feedback seeking.^{17 18} This is a possible reason explaining why previous attempts for gathering such information⁸ were felt as incomplete and unreliable. By using a qualitative approach, we were able to combine and report in-depth views of several individuals from different positions and countries.

In the second part of this work, we used a novel technique, the 9 Diamond methodology, in order to identify underlying beliefs and attitudes likely to influence how assessment of competences take place. Through this work, it has been possible to identify general priorities in assessment of competences in rheumatology across Europe. This work has the potential to help some countries to develop their approach to assess competences in rheumatology training and avoid pitfalls. One of the main priorities of FG participants, especially for trainees, was the identification of trainees at risk of failure and their support for progression. Failure signs should be monitored and addressed in a timely fashion, so that a variety of appropriate solutions can be offered. While the importance of regular skill assessment was discussed, the particular case of technical procedural skills such as joint aspiration was not specifically addressed, which could be perceived as a lacking element.

Some limitations of the study must be considered. Qualitative research often relies on purposive sampling. By its nature, the sample was predisposed towards participants with an interest in education and current interaction with EULAR, and so may well be biased towards those with a fuller understanding of and commitment towards the assessment of competences and current best practice. However, the FG participants, through their engagement, are also likely to have critical views, as well as positive ones, and are therefore suitable for such a qualitative study. It is likely that there are other examples of good practice either nationally, institutionally or locally that have not been captured by this approach, but they were considered out of scope of this project. Online FGs present unique challenges. The group size must remain small, but on some occasions, one or two participants were unavailable at the last minute, making the number of participants potentially low. However, the overall sample size and methods of data collection, building a comprehensive account of practice with contributions by different groups (trainers and trainees) and in different stages, provide a significant sample for a qualitative study of this type. In addition, general agreement in priority ordering across groups suggests that variation in FG sizes has not significantly affected the results. One limitation of the 9 Diamond method lies in the framing of the statements. It is important to formulate statements in a way that provokes discussion and leads to disagreement. This strategy was used in particular for the professionalism statement, which was worded in a negative manner. Despite leading to subsequent discussion, this particular statement ended up appearing low in the priority order, precisely because it was felt to be very important. This illustrates the fact that the order of priorities might not fully reflect importance in a linear way. Traditional qualitative data (ie, quotations from transcriptions of the discussion) clearly demonstrated that assessing professionalism was felt to be extremely important and have a high priority in training. Of the 11 countries fulfilling our initial selection criteria, the inclusion of five countries allowed data saturation, while ensuring feasibility. Despite this limited number of countries, very few further aspects were mentioned by participants in response to prompts about what else needed to be included. Although we strove for representativity of different assessment systems and cultures, given the wide heterogeneity in training programmes across the 41 EULAR countries, selection bias could have hampered retrieving other useful insights or best practices.

In conclusion, we identified current practice in the assessment of competences across five countries, incorporating the views of expert trainers and current trainees. Additionally, priorities and underlying beliefs about the assessment of competences were identified. Together, these provide a rich and coherent picture on the assessment of competences in rheumatology training across European countries, which will inform the EULAR Points-to-Consider for the assessment of competences in rheumatology training.