Article Text
PDF
Abstract
Objective Digital symptom-checkers (SCs) have potential to improve rheumatology triage and reduce diagnostic delays. In addition to being accurate, SCs should be user friendly and meet patient’s needs. Here, we examined usability and acceptance of Rheumatic?—a new and freely available online SC (currently with >44 000 users)—in a real-world setting.
Methods Study participants were recruited from an ongoing prospective study, and included people ≥18 years with musculoskeletal complaints completing Rheumatic? online. The user experience survey comprised five usability and acceptability questions (11-point rating scale), and an open-ended question regarding improvement of Rheumatic? Data were analysed in R using t-test or Wilcoxon rank test (group comparisons), or linear regression (continuous variables).
Results A total of 12 712 people completed the user experience survey. The study population had a normal age distribution, with a peak at 50–59 years, and 78% women. A majority found Rheumatic? useful (78%), thought the questionnaire gave them an opportunity to describe their complaints well (76%), and would recommend Rheumatic? to friends and other patients (74%). Main shortcoming was that 36% thought there were too many questions. Still, 39% suggested more detailed questions, and only 2% suggested a reduction of questions.
Conclusion Based on real-world data from the largest user evaluation study of a digital SC in rheumatology, we conclude that Rheumatic? is well accepted by women and men with rheumatic complaints, in all investigated age groups. Wide-scale adoption of Rheumatic?, therefore, seems feasible, with promising scientific and clinical implications on the horizon.
- Arthritis
- Autoimmune Diseases
- Health services research
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Digital symptom-checkers (SCs), which hold a promise to improve rheumatology triage and reduce diagnostic delays, need to be user acceptable, but there is a lack of large-scale user experience studies based on real-world data.
Together with patients, clinicians and eHealth experts, we have developed Rheumatic?—a widely used (>44 000 users in 16 months) and freely available online SC targeted to people with rheumatic complaints.
WHAT THIS STUDY ADDS
This is the largest (n=12 712) user experience study of a digital SC in rheumatology.
The study finds that real-world users’ perception of Rheumatic? is positive, and even though the current version of Rheumatic? does not yet suggest a diagnosis or give care advice, people find it useful in summarising their complaints, and would recommend it to friends and other patients.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
This study has contributed crucial end-user feedback towards optimisation of Rheumatic? that is currently being addressed in large prospective studies, including: (1) development of an algorithm for diagnosis and care advice, (2) inclusion of more targeted questions and (3) assessment of symptoms described as free text using artificial intelligence—with the ambition to integrate Rheumatic? in standard healthcare.
Introduction
Diagnostic delay is a big challenge in rheumatology, and there is a need to accelerate access to specialist care and therapy for people with inflammatory rheumatic diseases (IRDs) as early diagnosis and treatment are key for improving clinical outcome.1–4 At the same time, up to 60% of patients with rheumatic complaints visiting rheumatologists do not have IRDs.5 6 With an ageing population, this group will grow, together with the cost and burden on the healthcare system. Hence, there is a need to improve rheumatology triage. Here, digital preassessment tools could be helpful.7
Online symptom-checkers (SCs) are patient-facing diagnostic decision support systems with the potential to reduce diagnostic delays and errors.8 9 A handful of studies exploring digital SCs within rheumatology have been performed, yet they are not commonly used in routine care, partly due to limited diagnostic accuracy and a lack of large-scale validation studies based on real-world data.5 9–11
We have developed a digital SC called Rheumatic? together with patients.12 When evaluated in a retrospective multicentre validation study,13 Rheumatic? demonstrated high discriminative performance in identifying individuals who would develop rheumatoid arthritis in an at-risk population (area under the receiver operating curve (AUC-ROC): 75.3%) and in differentiating IRDs from other musculoskeletal problems in individuals with early joint swelling (AUC-ROC: 79%). However, when clinicians already suspected an autoimmune IRD, Rheumatic? had less discriminative power (AUC-ROC: 53.6%).
To optimise the scoring system and further evaluate self-reported symptoms of rheumatic and musculoskeletal diseases (RMDs), Rheumatic? is currently being investigated in a number of ongoing prospective studies, and a public version—providing a symptom overview without diagnostic scores or care advice—is available at: https://rheumatic.elsa.science/, in English and Dutch. To this date, 44 395 people completed this public version of Rheumatic?.
Fundamental for eHealth tools is that they are user acceptable, as also pointed out by the European Alliance of Associations for Rheumatology.7 9 14 Hence, in this study, we have assessed usability and acceptance of this increasingly used digital SC in a real-world setting.
Methods
Study design
Study participants were recruited from an ongoing Dutch longitudinal observational prospective study. Briefly, since July 2021, people with musculoskeletal complaints searching online for information were directed to Rheumatic? via the Dutch Arthritis Association website or through social media campaigns. People who completed Rheumatic? and in conjunction gave online consent using a tick box consent form were asked to fill out the user experience survey within 1 week (figure 1). The study population comprises adults (≥18 years) with musculoskeletal complaints, who are fluent in Dutch, and have an email address. Questions regarding diagnoses, interventions and type of care are sent out at 3, 6 and 12 months, and not reported on here. Study endpoints include: (1) referral to rheumatologist, (2) inflammatory versus non-inflammatory diagnosis and (3) specific diagnosis.
Schematic illustration of the study design, including the five closed-ended user experience survey questions. Number (N) of study participants who completed Rheumatic? and were sent the user experience survey (n=24 061) and who completed the user experience survey (n=12 712) are shown. At 3, 6 and 12 months, study participants are asked whether they have visited a general practitioner (GP), been referred to rheumatologist or visited a rheumatologist.
User experience survey
The user experience survey included five questions on usability and acceptability of Rheumatic? (figure 1), with responses recorded on an 11-point (0–10) rating scale. In addition, an open-ended question concerning participant’s own suggestions for improving Rheumatic? was included. See online supplemental methods for a more detailed description of the rationale behind the survey questions and response analysis.
Supplemental material
Statistics
Data analysis was performed in R, V.4.4.2; t-test or Wilcoxon rank test was used for normally and non-normally distributed values, respectively; linear regression was used for continuous dependent variables (scores and age groups), with p values calculated for the complete distribution of the tested variable. P values <0.05 were considered significant.
Results
Who do we capture with Rheumatic?: baseline characteristics of the study population
By September 2022, 24 271 individuals had completed Rheumatic?. Of these, 24 061 also received the user experience survey, which 53% completed (figure 1). The response rate was higher among people ≥50 years (63%) compared with people <50 years (38%), p<0.0001. Thus, participants in the user experience survey were somewhat older than the total approached group (71% ≥50 years vs 60%, p<0.0001).
The study participants were normally distributed over the different age categories with a peak at age 50–59, and a majority of women (78%) (figure 2A). The proportion of women and men differed between age groups, with 9%–18% men in younger age groups (<60 years), increasing to 32%–47% in older age groups (≥60 years) (figure 2B), p<0.0001.
The age distribution of women and men in Rheumatic? and in the user experience survey. The number of women and men per age group (A), and the proportion (%) of women and men per age group (B) are shown. Blue bars=women; green bars=men.
What do the study participants think of Rheumatic?: user experience survey results
Rheumatic? is composed of 17–76 questions (depending on previous answer given), with a median completion time of 10.4 min. When asked: How appropriate did you find the number of questions?, 61% answered that the number of questions was good (scored 4–6), with more women (62%) being positive than men (57%), p<0.0001 (figure 3A). Those who did not find the number of questions appropriate, mainly thought Rheumatic? had too many questions (36% scored 7–10). Younger people were more satisfied than older, p<0.0001 (online supplemental figure 1).
Supplemental material
Results from the user experience survey. Scores for questions 1–5 and mean scores for questions 2–5 are shown for women (blue bars) and men (green bars). For question 1 (A) score 0=far too few questions; score 5=good number of questions; score 10=far too many questions. For questions 2–5 (B) score 0=the most negative rating; score 10=the most positive rating. CIs (95%) for mean scores (C) were: 7.75 to 7.82 and 7.74 to 7.86 (Q2); 6.79 to 6.87 and 6.67 to 6.84 (Q3); 6.59 to 6.66 and 6.81 to 6.94 (Q4); 6.73 to 6.81 and 6.8 to 6.95 (Q5), for women and men, respectively. Note the different y-axes in (A, B). N, number of study participants.
A large majority (90%) found the questions in Rheumatic? to be clear (scored ≥6), with no difference between women and men (figure 3B) or between age groups (online supplemental figure 2). Less than 4% disagreed (scored ≤4). Mean score was 7.8 (figure 3C).
A majority also found the test useful (78% scored ≥6; mean score 6.8), while 9% did not (scored ≤4). Older people were more satisfied than younger (increasing score of 0.1 points per age category, p<0.0001) (online supplemental figure 3).
Seventy-six per cent (74% women; 80% men) thought the questionnaire gave them an opportunity to describe their complaints well (scored ≥6), while 11% of women and 9% of men did not agree (scored ≤4). Mean score was 6.6 for women and 6.9 for men, p<0.0001, with no difference between age groups (online supplemental figure 4).
Seventy-four per cent (74% women; 76% men) would recommend Rheumatic? to a friend or other patient (scored ≥6), while 10% would not (scored ≤4). Mean score was 6.9. Older people were more positive than younger (increasing score of 0.09 points per age category, p<0.0001), (online supplemental figure 5).
Study participants’ suggestions to improve Rheumatic?
Twenty-six per cent provided comments on how to improve Rheumatic?. The most common suggestions were to provide more detailed questions, particularly regarding their own complaints (39%), to provide more open-ended questions (28%), and to suggest a diagnosis (14%) or give care advice (8%). Notably, only 2% suggested a reduction of questions.
Discussion
Given that Rheumatic? is increasingly being used by patients and clinicians, we have performed a user experience study among real-world users to explore whether it could and should be improved.
To the best of our knowledge, this is the largest user-evaluation study of a digital SC within rheumatology. A great majority of study participants found Rheumatic? useful (78%) and thought the questionnaire gave them an opportunity to describe their complaints well (76%). Three in four would recommend Rheumatic? to friends and other patients. Contrary to what was found in other studies,15 16 older people were in some measures more positive than younger, but differences were small. Most of the participants’ suggestions of improvement are being addressed in ongoing research, including (1) development of an algorithm for diagnosis and care advice and (2) assessment of symptoms described as free text.
The overall survey response rate was 53%, which is higher than generally reported for web surveys.17 18 Still, with 47% not completing the survey, we acknowledge the risk of bias. The response rate was highest (63%) among people ≥50 years, and we speculate that this may to some extent be related to the fact that older people are more likely affected by RMDs,19 and thus possibly more motivated to contribute to the survey.
The age distribution reflects the population with RMDs well, and with a lifetime risk of developing IRDs of 8.4% in women and 5.1% in men,19 also the ratio of women to men was as expected. The major patient-perceived shortcoming of Rheumatic? was the number of questions; 36% thought there were too many questions. At the same time, many participants suggested adding more specific questions about their own symptoms. This balance between individual needs and generalisability remains a challenge. In a separate study, we will investigate whether particular questions and the total number of questions can be improved.
A weakness of the study is that we lack data on socioeconomic status and health literacy, thus, we cannot exclude that socioeconomically disadvantaged groups or people with low health literacy may be underrepresented. We have also not assessed digital literacy. Notably, a key concern in the eHealth era is that the electronic format may exclude people with digital illiteracy and people without access to smartphones or internet. Moreover, results from the Dutch population may not be applicable to users in other parts of the world. These are important aspects to address in future research.
In summary, the good response rate to the user experience survey allows us to conclude that Rheumatic? is well accepted by people with RMD symptoms. Ongoing prospective studies will clarify if the high diagnostic accuracy of Rheumatic?—identified in a retrospective study13—can be validated in a real-world setting. In its current form, Rheumatic? offers the increasing number of people googling RMD symptoms11 16 20 a comprehensive summary of complaints as a basis for clinical consultation, generated from a 10 min online questionnaire developed by patients, researchers and clinicians together with eHealth experts.
Ethics statements
Patient consent for publication
Ethics approval
Being non-interventional, the study was exempt from complete ethical assessment by the regional Medical Research and Ethics Committee, registration number: N21.22. Patient recruitment, data collection, storage and analysis follow the European General Data Protection Regulation, and was approved by the Leiden University Medical Center privacy officer.
Acknowledgments
We thank all study participants for their contributions to the study, and Elsa Science AB for designing, developing and hosting the current version of Rheumatic? We would especially like to thank Marta Lason and Barbara Axnäs at Elsa Science AB for coordination, and for providing data on the total number of Rheumatic? users and median completion time, and Lars Klareskog at Karolinska Institutet for scientific advice during the development of Rheumatic?
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors Project administration and conceptualisation: RK; formal analysis and investigation: RK, KL and LQ; methodology: RK, MPM, WEvS and CA; visualisation: KL; manuscript writing—original draft preparation, editing and final manuscript: KL; manuscript review and editing; RK, MPM, LQ, WEvS and CA. All authors have read and agreed to the final version of the article.
Funding This research project was funded by ReumaNederland. RK was supported by ZonMW Klinsche Fellow, grant number: 90719069. This research is also part of an activity (DigiPrevent) that has received funding from EIT Health. EIT Health is supported by the European Institute of Innovation and Technology (EIT), a body of the European Union receiving support from the European Union ́s Horizon 2020 Research and Innovation programme.
Competing interests RK is an editorial board member of RMD Open. She has not been involved in handling of the manuscript at RMD Open, or in communication with the journal regarding the study. LUMC and KI are research collaborators with Elsa Science AB. Elsa Science AB has contributed to the collaboration with in-kind contributions to the design, development and hosting of the digital symptom-checker Rheumatic? KL was part-time employed by Elsa Science, January 2022–February 2023. All other authors declare no commercial or financial conflict of interest in relation to the study.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.