Article Text
PDF
Abstract
Objective About half of the rheumatology trainees do not use a portfolio. This project was established to reach consensus about the content of a EULAR portfolio for Rheumatology training and subsequently develop portfolio assessment forms.
Methods After establishing a portfolio working group (WG), including nine rheumatologists and one educationalist, a systematic literature review (SLR) on the content and structure of portfolios for postgraduate learning was conducted (November 2018). This was followed by a survey among WG members and members of the EMerging EUlar NETwork, inquiring about the content and structure of existing national portfolios. The portfolio WG selected the key components of the portfolio, taking previous experience and feasibility into account. Assessment forms (eg, case-based discussion) were developed and pilot-tested.
Results 13/2034 articles were included in the SLR (12 high/1 moderate risk of bias). Information on procedural skills, personal reflections, learning goals and multisource feedback was most often included a portfolio. Twenty-five respondents completed the survey (response≈50%). Feedback from assessors, reflective writing and formulation of learning goals were considered important dimensions to be covered in a portfolio. Six key components of the portfolio were established: curriculum vitae, personal development plan, clinical work, professional behaviours, education and research activities. Suggested minimal content for each component was formulated. Four assessment forms were successfully pilot-tested by 11 rheumatologists and their trainees.
Conclusion A EULAR portfolio for Rheumatology training and assessment forms were developed. Portfolio implementation, particularly in countries without an existing portfolio, may promote a higher standard of rheumatology training across Europe.
- health care
- health services research
- education
Data availability statement
Data are available on reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Key messages
What is already known about this subject?
It is important that rheumatology trainees gain insight into their personal and professional development and get the opportunity to reflect and act on this. A portfolio may support this process.
What does this study add?
Six key components of the portfolio were established: curriculum vitae, personal development plan, clinical work, professional behaviours, education and research activities.
Four assessment forms were developed: Mini-Clinical Evaluation Exercise, Direct Observation of Procedural Skills, Case-Based Discussion and Multisource Feedback Form.
How might this impact on clinical practice or further developments?
Implementation of the portfolio may promote a higher standard of rheumatology training and ultimately patient care across Europe.
Introduction
A medical portfolio is an instrument to collect evidence of a trainees’ exposure to training opportunities and progression in knowledge, skills and behaviours.1 Information collected as part of a portfolio can be used for different purposes. In several European countries, the portfolio is solely used as a register of clinical and educational activities (‘a logbook’). However, a portfolio can play a decisive role to foster self-awareness by encouraging personal reflection using the recorded information. The collected evidence can also be used by third parties, like a mentor, as a formative or summative assessment tool to help the trainee in making decisions about individual progress and future plans.1 There is substantial evidence from undergraduate and postgraduate medical programmes that if well implemented, portfolios contribute to improved learning during training.2–4
As training goes beyond clinical work, it is important that rheumatology trainees gain more insight into their personal and professional development and get the opportunity to reflect and act on this. In some countries, the use of a portfolio during rheumatology training is already mandatory. However, many other countries do not currently use a portfolio.5 In a survey among 132 members from 34 countries disseminated by the Emerging EULAR Network (EMEUNET) in 2016, about half of participants reported not using a portfolio during their rheumatology training. Of the participants who did not use a portfolio, 86% (strongly) agreed that it might be a useful tool.5 This indicates that a EULAR portfolio for Rheumatology training including a clear guide and flexible content might improve overall training standards by raising educational content, fostering the setting of goals and direction for trainees in institutions or countries that do not use a portfolio. A EULAR approved portfolio can also provide a benchmark for existing national portfolios and help harmonise training across countries. In this way, the portfolio may facilitate movement of trainees between countries, thereby promoting greater utilisation of international exchange opportunities.
The aim of this project, undertaken under the auspices of the EULAR School of Rheumatology (ESOR), was to (1) reach consensus about the key components that should be included in a portfolio, (2) investigate the potential barriers when introducing a portfolio and (3) develop and subsequently pilot test the EULAR portfolio for Rheumatology training.
Methods
After approval by the ESOR, a working group (WG) comprising nine European rheumatologists with an interest in education and one educationalist was established.
Systemic literature review
First, a systematic literature review (SLR) was conducted by one WG member (MO), on the content and structure of portfolios for postgraduate learning in all medical fields. Eligible studies were identified according to the PIM structure: P (population): trainees, fellows; I (instrument of interest): portfolio; M (measurement of properties of interest): content portfolio. Medline, PubMed, Education Resources Information Center and Web of Science were searched for this purpose for the period 1946–November 2018. The risk of bias in included experimental and observational studies was assessed using the medical education research study quality instrument (MERSQI).6 Cut-off points of the MERSQI (score range 5–18) are not clearly defined, but based on previous studies, the following cut-off points were used:<10 high risk of bias; 10–11 moderate risk of bias, ≥12 low risk of bias.7 Studies were too heterogeneous to allow pooling; descriptive results are therefore presented.
Survey
Second, as it was expected that an SLR would not capture all available information related to the content and structure of a portfolio, a survey was disseminated among WG members of the portfolio project and WG members of EMEUNET. The survey consisted of the following parts: (1) questions related to the content, format and organisation of the portfolio (multiple choice), (2) opinion about the functionality of several key areas often included in medical portfolios (open questions) and (3) barriers for implementation (open question). The intended time length of the survey was approximately 10–15 min to ensure participation. In addition, all survey participants were requested to collect and send portfolio materials (ie, print screens from the portfolio used in their country) to one WG member (MO). These portfolio materials were categorised based on the specific content, for example, multisource feedback (MSF) forms, personal development plans (PDP) or Direct Observation of Procedural Skills (DOPS) forms. Portfolio materials were collected between January 2019 and March 2019.
Meeting WG and development portfolio
Third, during a 1-day meeting of the portfolio WG (April 2019), the evidence and quality assessment of retrieved studies from the SLR was presented in the form of tables and figures. The results of the survey and collected portfolio materials were also discussed among the WG members. Following this, through a process of discussion and consensus, based on the available evidence, collected information and expert opinion, the key components of the portfolio were established. After this, it was decided what minimal content (eg, assessment forms, personal reflections) should be included in each key component, also taking feasibility into account. Through a consensus process, the content of the portfolio was extensively discussed and agreed on. Consensus was sought through discussion, if not possible, formal voting would to take place. Subsequently, a guide, examples and if applicable, templates for the selected key components of the portfolio were developed. This portfolio was subsequently reviewed and pilot tested by members from the EMEUNET WG and ESOR Fellow’s classroom. The final portfolio was approved by the above-mentioned stakeholders.
Results
Results of the SLR and survey
The database search yielded 2034 studies (the search strategy is shown in online supplemental file S1). After screening the titles and abstracts, 99 articles remained for full paper review. After full-text reading, 13 articles were included (online supplemental files S2 and S3). Twelve articles described the content and evaluation of a national or local portfolio4 8–18 and one article was a personal opinion, based on previous experiences with portfolios.19 In 8 out of 13 articles, application of the MERSQI was possible: except for 1 article4 with a moderate risk of bias, all articles had a high risk of bias.11–15 17 18
Supplemental material
According to these publications, portfolios most often include information on DOPS (9/13), personal reflections (8/13), learning goals (5/13) and MSF (5/13).
In total, 25 respondents from the WG and also EMEUNET WG of 13 countries filled out the survey (response rate approximately 50%). Sixteen (64%) of these 25 respondents had used a portfolio either as trainee and/or supervisor. In about half of the cases, the portfolio was online-based. When asked about the functionality of several key areas often included in medical portfolios (open question), respondents found reflective writing (n=7), formulation of learning goals (n=4) and feedback from supervisors (n=4) especially useful. Thirteen respondents indicated that managing a portfolio is important because it is a single and complete source of information and a good resource to demonstrate progression over a specific time period.
Of the 25 respondents, about half indicated that a portfolio was often time-consuming and/or a bureaucratic form of administration. Four respondents mentioned that lack of feedback by supervisors made them hesitant to use the portfolio. Last, three respondents indicated that it was difficult to implement a portfolio at a local or national level.
Portfolio materials of eight countries were collected (UK, Denmark, Netherlands, Germany, Switzerland, Italy, Greece and Spain). Several portfolios (Germany, Switzerland, Italy, Greece and Spain) are merely logbooks, that is, a record of clinical and educational activities. Other portfolios (UK, Denmark and Netherlands) also included competency-based learning elements, which means that the way knowledge and attitudes are applied in clinical situations by trainees, is also documented and assessed, often in the form of MSF. These three portfolios also focused on workplace-based assessments, learning goals and personal reflection on professional behaviour and training progress.
Key components and associated content of the EULAR portfolio for Rheumatology training
Based on the available evidence, collected information and expert opinion, the key components of the portfolio and minimal content of each key area were established by WG members (table 1). In the next sections, all key components and associated content are introduced, together with the underlying rationale. Templates and four assessment forms that can be used in practice are provided in online supplemental material S4. The assessment forms were reviewed and pilot tested by 11 rheumatologists and their trainees. The wording and layout of the assessment forms were only slightly changed after the pilot test.
Key components of the EULAR portfolio for Rheumatology training and suggested minimal content
Curriculum vitae (CV)
A CV is an important tool to register and communicate all relevant clinical, educational and research experiences and achievements of the trainee. A CV can also help the trainee to identify areas in need of additional training and development.20 21 Ideally, the CV should be maximum of 2–3 pages to keep information concise and relevant. However, an academic CV can be longer, especially when research activities and publications are being included. Links to websites that provide examples of medical CVs, can be found in online supplemental material S4.
Personal development plan (PDP)
A PDP helps the trainee to (1) identify new learning goals and (2) develop an action plan to achieve them. The PDP needs to be updated at a predetermined regular interval, for instance every 6 months. Completed goals can serve as building blocks for future plans.22 Feedback or personal reflections noted down in other key components of the portfolio can also be used to formulate new learning goals. It is important to include both clinical and non-clinical learning goals in the PDP. Among others, non-clinical learning goals may include academic or research objectives, such as writing a research protocol under supervision. The learning goals should preferably be formulated according to the SMART framework: Specific, Measurable, Attainable, Realistic and Time-bound.23 24 The template for the PDP, including two examples of learning goals and subsequent actions plans, is included in online supplemental material S4. It is expected that in every country there is an official Rheumatology Curriculum framework informing the trainee and his mentor(s) of the competencies, skills and knowledge they are expected to master at the end of training, and the ones they can voluntarily elect.
Clinical work
In this key area, three parts are distinguished:
Information on managing patients (eg, rheumatoid arthritis, spondyloarthritis) and skills (eg, joint aspiration, deliver bad news); preferably recorded in a logbook.
Assessments (summative and formative).
Personal reflections.
A logbook is a registry of representative clinical cases and tasks concerning a specific training period.25 Logbooks give trainees and supervisors a quick overview of the requirements of training and what needs to be achieved in a specific training period. Logbooks are especially useful if different training sites are involved and can help trainees and supervisors to write a PDP and engage in personal reflections.25 For the patient categories that should be included in the logbook, one can use the national curriculum or alternatively available European curricula (including the rheumatology section of the European Union of Medical Specialists as a basis.26 To make the logbook an integral part of training, it is important that data can be easily entered and that time spent on the logbook is kept to a minimum. As examples, a list of patient categories that the Dutch trainees can use to group patients in their portfolio and part of the German logbook can be found in online supplemental material S4.
Apart from aggregation of a sufficiently representative array of patients and skills in this key area, it is also essential to include competency-based formative assessment tools in a portfolio. Assessments should happen throughout the training and frequently enough to provide the trainee with appropriate feedback to guide ongoing professional development and to identify areas for improvement before a final summative assessment.27 The feedback included in the assessment forms can result in new learning goals and be included in the PDP. WG suggests that the following assessment tools should be included in a portfolio:
Mini-Clinical Evaluation Exercise (mini-CEX).
DOPS.
Case-Based Discussion (CbD).
The assessment forms for these tools can be found in online supplemental material S4. The assessment forms can also be downloaded from the ESOR website.
The information from the logbook and formative assessments should be the object of personal reflection. Reflection is defined as ‘a metacognitive process that creates greater understanding of self and situations to inform future action’. Personal reflection is considered essential in a portfolio as it improves engagement and is often the first step in the making of new learning goals.28 Personal reflections can be kept private. With regard to patient confidentiality or misuse of information in legal cases, it is important to only use anonymised information in the portfolio. For more information on personal reflection, including a template and examples, consider online supplemental material S4.
Professional behaviour
Professional behaviour or ‘professionalism’ is defined as ‘the commitment to carry out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population’.29 30 Professional behaviour should be considered a core competence and was considered an essential aspect of a portfolio by all WG members. Feedback from both colleagues and patients as well as personal reflection are the backbone to develop and mature professional behaviour. Professional behaviour can typically be assessed by a MSF form (see online supplemental material S4; the assessment form can also be downloaded from the ESOR website). The template for personal reflection described in the section ‘clinical work’ can also be used in this area.
Education
Healthcare professionals need to regularly update or further develop their knowledge, skills and attitudes to serve the continuous improvement of their professional practice. In this key area, all educational activities a trainee has received or engages in can be listed and cross-checked against the minimum educational standards and, if necessary, be reflected on. In addition, teaching activities of the trainee (eg, giving a presentation) can also be documented here.
The template for personal reflection described in the section ‘clinical work’ can also be used in this area.
Research
Documentation that reflects the research activities of the trainee should be included here. Conducting research during the rheumatology training period is often encouraged and in some countries a research project is a mandatory component of training. In this section, one can include relevant publications, academic posts, bursaries, scholarships and a list of successful research grant projects.
Discussion
Portfolios are increasingly considered to support postgraduate training by identifying learning needs and critical evaluation of trainee performance by the trainee itself and other persons involved in the learning process, while making reference to the expectations of the regulatory curriculum as well as personal choices and values.
We describe here the development of the EULAR portfolio for Rheumatology training, which was derived through consensus, based on an SLR, a survey among target users with an interest in medical education, existing portfolios and expert opinion.
The EULAR portfolio for Rheumatology training might ultimately help to harmonise training in Europe without interfering with specific national regulations, as the portfolio is flexible and can be adapted to the national curriculum. Countries that already use a national portfolio can benchmark their existing portfolio, if desired. Our project is in alignment with an ongoing ESOR taskforce ‘Standards for the training of European rheumatologists’. The aim of this taskforce is to further foster harmonisation by developing a framework describing individual competences that a successful trainee should demonstrate. This information can be included in the portfolio. It is explicitly not our intent to supersede the national rheumatology authorities' competence in defining their approach to rheumatology training. These educational initiatives may however give direction with regard to long-term decisions about the medical education ‘climate’ and further standardise training across Europe. In addition, using the EULAR portfolio can facilitate movement of trainees across Europe without facing educational or bureaucratic barriers, as the skills of the trainee are documented together with MSF and a learning plan. This way, trainees can instantly share the details about their learning process and development over time with their new supervisors.
Portfolios have many advantages, but there are also disadvantages that deserve consideration. In our survey, several respondents considered a portfolio a bureaucratic form of administration. This ‘resistance’ against portfolios was also highlighted in a recent EULAR educational project by Najm et al with focus group interviews of rheumatology supervisors and trainees from five different European countries.31 In order to maximise its potential and meaningfulness, it is important that working on a portfolio is not just seen as a time-consuming requirement and that trainees are empowered and motivated to do this. When used correctly, a portfolio helps and stimulates to look back and reflect on the collected information. In the study of Najm et al, it was particularly found important by focus group participants that the portfolio is a practical working tool which guides trainees and their mentors through clinical discussions and observations of clinical practice towards a successful outcome.31 Feedback and support of a mentor is essential in this process.1 17 31 32 In a study by Snadden et al, using semistructured interviews, it was found that the quality of feedback and enthusiasm of mentors was extremely important in encouraging the use of a portfolio.32 As it is sometimes difficult for trainees and mentors to find time for face-to-face feedback, a portfolio might facilitate this process without the time and location constraints.33
We envisage facilitation implementation of the EULAR portfolio for Rheumatology training by contacting the national rheumatology associations via the EMEUMET country liaisons, and in particular those of countries who do not use a portfolio (yet) to make them familiar with the portfolio. In addition, the assessment forms can be downloaded from the ESOR website. We will prospectively collect data on the use of the portfolio and suggestions to improve its feasibility and efficiency as a tool for excellence in the training process.
In conclusion, this ESOR initiative resulted in the development of a EULAR portfolio for Rheumatology training and hope that its implementation may harmonise training and ultimately promote a high standard of patient care across Europe.
Data availability statement
Data are available on reasonable request.
Ethics statements
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @ElenaNikiUK
Contributors MvO, SR, AA, EN and FS participated in the design of the study. All authors participated in the development of the project, the interpretation of the data and the manuscript preparation and approved the current version of the manuscript.
Funding This project was supported by a EULAR grant (EDU043).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.