Article Text
PDF
Abstract
Objectives To develop and validate a web-based ecological momentary assessment (EMA) tool to enhance symptoms monitoring among patients with Sjögren’s disease (SjD).
Methods Consecutive adults with SjD were enrolled in this pilot observational study. Participants used the WebApp over a 3-month period, for the daily collection of individual EULAR Sjögren’s Syndrome Patient Reported Index (ESSPRI) scales and separate assessment of eyes and mouth dryness, using 0–10 numerical scales. Primary outcome was the measure of the interdaily variability of symptoms. Data collected through the WebApp were compared with those obtained with paper-based questionnaires administered during a final visit, using distinct approaches (predicted error, maximum negative error and maximum positive error). User experience was assessed using the System Usability Scale (SUS) score.
Results Among the 45 participants, 41 (91.1%) were women. Median age was 57 years (IQR: 49–66). Daily variability of symptoms ranged between 0.5 and 0.8 points across the scales. Over the 3-month period, the predicted error ranged between −1.2 and −0.3 points of the numerical scales. The greatest differences were found for fatigue (−1.2 points (IQR: −2.3 to −0.2)) and ESSPRI score (−1.2 points (IQR: −1.7 to −0.3)). Over the last 2 weeks, the predicted error ranged between – 1.2 and 0.0 points. Maximum negative error ranged between −2.0 and −1.0 points, and maximum positive error between −0.3 and 0.0 points. Median SUS score was 90 (IQR: 85–95).
Conclusion Our results demonstrate the usability and the relevance of our web-based EMA tool for capturing data that closely reflects daily experiences of patients with SjD.
- Sjogren's syndrome
- patient reported outcome measures
- autoimmune diseases
Data availability statement
Data are available upon reasonable request. The data that support the findings of this study are available on request from the corresponding author. Some data are not publicly available due to the privacy protection of patients.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
In patients with Sjögren’s disease (SjD), symptoms of dryness, fatigue and pain are those which have the greatest daily impact on patients’ quality of life. The assessment of these essential components of the disease primarily involves the use of patient-reported outcomes (PROs), especially the EULAR Sjögren’s Syndrome Patient Reported Index (ESSPRI). Ecological momentary assessment methods, allowing real-time assessment of symptoms, may contribute to the optimisation of current PROs.
WHAT THIS STUDY ADDS
Through this study, it was possible to show that the daily assessment of symptoms related to SjD can be easily implemented via a dedicated WebApp and is very well accepted by patients. It was also possible to demonstrate, as expected, that the intensity of SjD-related symptoms is subject to significant day-to-day variability. Finally, it was possible to demonstrate that ‘summarising’ a patient’s condition over a prespecified period using these longitudinal data can produce results different to those obtained through conventional, cross-sectionnal, approaches.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
At present, it cannot be confirmed that this innovative approach is likely to supplant current methods for collecting PROs in patients with SjD. The evaluation of the properties of this tool in the context of therapeutic trials assessing potentially effective immunomodulatory strategies should address this research agenda. This will soon be the case in the context of the European NECESSITY trial.
Introduction
Sjögren’s disease (SjD) is a rare systemic autoimmune disease characterised by the infiltration of lymphocytes into exocrine glands, primarily the salivary and lachrymal ones. This results in symptoms of dry eyes and mouth, accompanied by debilitating fatigue and pain.1 2 Patients with SjD may exhibit extraglandular/systemic manifestations in 30%–40% of cases. All organs may be affected by the disease, predominantly the joints, skin,3 lung, kidneys, nerves,4 small vessels and muscle,5 leading to serious complications in a subset of the patients. Nevertheless, symptoms of dryness, as well as general symptoms such as fatigue and diffuse musculoskeletal pain, are those which have the greatest daily impact on patients’ quality of life.6 7 As a result, any appropriate evaluation of the disease, whether in routine care or clinical research, should include the collection of patient-reported outcomes (PROs), along with determination of the presence and severity of systemic manifestations. Two disease assessment scores have been developed and validated by a group of European experts, allowing to assess these two complementary aspects of the disease: ESSDAI (Eular Sjögren Syndrome Disease Activity Index),8 a composite score completed by the physician which evaluates the systemic activity of the disease and ESSPRI9 (EULAR Sjögren’s Syndrome Patient Reported Index), which is a disease-specific PRO that evaluates the severity of their symptoms (dryness, fatigue and pain on a 0–10 numerical scale). More precisely, it evaluates the maximum intensity of symptoms during the 2 weeks preceding its completion.
To date, no immunomodulatory treatment has definitively demonstrated its effectiveness in influencing the severity of the disease or improving its core symptoms in the long term.10 Currently approved symptomatic treatments transiently relieve the symptoms of dryness, whereas treatments for more serious organ involvement are empirically adapted from those used in other rheumatic diseases with comparable features. Nevertheless, some recent trials displayed promising results, particularly those evaluating agents targeting B cells or CD40/CD40 ligand pathway.11–14 For agents targeting B cells, these preliminary demonstrations of efficacy pertain to systemic disease activity (especially ESSDAI) only and no clear evidence of efficacy regarding PROs, including ESSPRI, has been reported so far. Conversely, agents targeting CD40/CD40 ligand pathway have yet shown promising data for both control of systemic disease activity and sicca symptoms reduction.
PROs are increasingly being used in clinical trials evaluating innovative treatments, either as secondary or occasionally as primary endpoints.15 In the context of autoimmune diseases, that is true in both the fields of common and rare diseases. Based on the analysis of patient data, the minimal clinically important difference of the ESSPRI score has been defined as an improvement of at least 1 point (on the 0–10 numerical scale) or 15% between two assessments.16 In clinical trials, the change in the ESSPRI score is typically assessed between baseline and a follow-up visit conducted a few months later. However, symptoms reported by patients are likely to be highly variable over time, which can introduce imprecision in assessing symptoms based on a few data points. This may potentially be one explanation as to why several clinical trials have not been able to demonstrate treatment efficacy on the ESSPRI score. Furthermore, the collection of PROs is typically conducted in a hospital setting, alongside multiple assessments and procedures. This situation increases the risk that the responses obtained may not accurately reflect participants' daily experiences.
The assessment of the patient’s dynamic relationships between events and disease course is enhanced by the development of momentary data collection strategies, such as experience sampling methods and ecological momentary assessment (EMA).17 These approaches, which rely on delivering informative contents and self-administered questionnaires, reduce the recall bias as they are done in real time.18 Overall, an extensive panel of physical and mental conditions can be remotely monitored through these approaches. Our hypothesis is that in the specific context of SjD, a more frequent and real-time assessment over a defined period of time would be useful to address the issues associated with usual cross-sectional assessments in a hospital setting. As a result, we developed and tested an EMA digital tool, allowing a daily assessment of SjD-related core symptoms, with the ultimate goal of validating its use as an endpoint in future randomised controlled trials. The primary objective of this study was to determine the variability of symptoms collected through the innovative tool. Secondary objective was to compare symptoms collected through this method to those collected during hospital visits. Additional objectives were to describe the utilisation and usability of the tool by patients with SjD. This study is part of the European NECESSITY (New Clinical End-points in Patients With Primary Sjögren’s Syndrome) project (NCT05113004).19
Materials and methods
Study design and participants
Consecutive adult patients with SjD were enrolled in this pilot prospective observational study conducted at three tertiary care reference centres in France (Brest, Paris-Saclay and Strasbourg). Patients were eligible if they had regular access to a mobile device and met SjD ACR/EULAR criteria. There were no specific exclusion criteria, in order to be representative of the entire population affected by the disease, with the exception of patients under legal guardianship, who cannot be included in this type of clinical trial in France. Participants underwent a baseline and a 3 months hospital visit to collect data relevant to the study. Enrolment started in November 2019 and last follow-up visit occurred in May 2020. Inclusions were abruptly halted due to the COVID-19 pandemic and the implementation of a nationwide lockdown. All participants provided written informed consent before inclusion.
WebApp development and use
A Web-based EMA tool (‘PEPSS’), accessible from any mobile device, was specifically developed for the purpose of the study, with the aim of collecting SjD-related symptoms on a daily basis and in a real-life setting. The WebApp allows the daily collection of individual ESSPRI scales (dryness, pain and fatigue) and separate assessment of eyes and mouth dryness, using 0–10 numerical scales. The tool was developed by Sys.Vision (https://sys.vision/%23accueil). During the baseline visit, participants were systematically asked to use the dedicated PEPSS WebApp to report the intensity of their symptoms, on a daily basis, at the time of the day that was more convenient for them, until the final visit. A training on the use of the WebApp was provided by a specifically trained clinical study technician. In cases where questionnaires were not completed within a specified number of days, reminder messages were sent via SMS and/or email. As part of the secondary objectives, participants were also randomly assigned in a 1:1 ratio to receive reminders either after 48 hours of non-completion or after 7 days.
Outcomes
Sociodemographic and SjD characteristics were collected during the baseline visit. Disease activity parameters and reference-specific PROs (through standard paper-based questionnaires) were gathered during both the baseline and final visits. The primary outcome of the study was the measure of the interdaily variability of symptoms collected through the WebApp. The System Usability Scale (SUS) was evaluated during the final visit. The SUS is a widely used questionnaire for assessing the usability of a system or product. It consists of 10 items rated on a scale from 1 to 5. It provides a standardised measure of user perceptions regarding ease of use, learnability, efficiency and satisfaction. A normalised score above 80 indicates excellent usability, suggesting that the system or product is highly user-friendly and intuitive.20 Compliance data, including the number of uses over the 3-month period and the thoroughness of data completion, were extracted from the WebApp.
Statistical analysis
For this pilot observational study, it was initially planned to include 50 patients to estimate the intrasubject variability of the measures (individual scales and ESSPRI score). This sample size is considered consensual for pilot studies.21 The variability of symptoms over time was evaluated on repeated measurements. Absolute error at each measure was computed per patient and per day as the absolute difference between values of the day and mean over the 3-month period. The mean of these differences was then calculated for each patient. Data collected through the WebApp were compared with those obtained with paper-based questionnaires administered during a hospital visit after 3 months, using distinct approaches (predicted error, maximum negative error and maximum positive error) and considering both entire period and the last 2 weeks of use. To evaluate the error predicted at a given time point, a linear regression model was built for each patient, based on all daily (3-month period or last 2 weeks) measures, in order to obtain a prediction of the final value. This predicted value was subsequently compared with the actual value collected during the final clinical visit. Maximum negative error was defined as the ‘minimum value collected through the WebApp over the period of interest (3-month period or last 2 weeks)’ minus the ‘value collected during the final visit’. Similarly, maximum positive error was defined as the ‘maximum value collected through the WebApp over the period of interest (3-month period or last 2 weeks)’ minus the ‘value collected during the final visit’. Post-hoc analyses were conducted. The influence of patient characteristics on symptom variability was assessed by categorising participants into ‘low variability’ or ‘high variability’ groups, determined by the median ESSPRI variability observed over the 3-month period using the WebApp. The characteristics associated with the unwillingness to continue using the WebApp at the end of the study were also assessed. The data are presented as the median and IQR for quantitative variables and as frequency and percentage for qualitative variables. Comparative analyses for quantitative data were conducted using Wilcoxon’s test. The statistical analyses were conducted using the SAS software V.9.4. P values<0.05 were considered statistically significant.
Results
Study population
The detailed participants’ characteristics are reported in table 1. Among the 45 patients, 41 (91.1%) were women. Median age was 57 years (IQR: 49–66), with a minimum of 25 and a maximum of 78 years. About 88.9% had a positive salivary gland biopsy and 80% had anti-SSA antibodies. Most of patients complained of both eye and mouth dryness. Systemic involvement, defined as ESSDAI>0, was present in 28 (62.2 %) patients. Systemic activity of SjD was generally low, with 84.4% of patients having ESSDAI<5.
Participants’ characteristics at baseline
Symptoms intensity and disease activity at inclusion and 3 months
The intensity of the symptoms collected at the inclusion and final visits remained stable for most individual symptoms as well as for the overall ESSPRI score (table 2). Numerical differences were observed for individual oral and ocular dryness only. ESSDAI score was stable over the study period.
Symptoms intensity and disease activity at inclusion and 3 months
Daily variability of symptoms
Median daily variability of symptoms ranged between 0.5 and 0.8 points across the numerical rating scales (table 3). The lowest variability was for ESSPRI score (0.5 points (IQR: 0.4–0.7)). To illustrate the daily variability of the symptoms collected through the WebApp, figure 1 shows representative patterns of symptom variations over time for fatigue, pain, dryness and ESSPRI score. The only characteristic associated with higher variability of symptoms was use of artificial tears (95.8% vs 61.9%, p=0.007) (post-hoc data).
Daily symptoms variability over the 3 months period
Representative patterns of symptom variations over time. Representative examples of patterns of day-to-day symptom variations for dryness, fatigue, pain and ESSPRI score. For each measure, we show a patient with low variability (lowest quartile of mean error at each measure, blue curve), moderate variability (second and third quartiles, green curve) or high variability (highest quartile of mean error at each measure, red curve). ESSPRI, EULAR Sjögren’s Syndrome Patient Reported Index.
Measurement errors over the 3-month period
The predicted error ranged between −1.2 and −0.3 points of the numerical scales (minus sign indicating that the predicted value is lower than the actual value collected during the final visit). The greatest differences were found for fatigue (−1.2 points (IQR: −2.3 to −0.2)) and ESSPRI score (−1.2 points (IQR: −1.7 to −0.3)). Maximum negative error ranged between −3.0 and −1.0 points, and maximum positive error between+0.7 and +2.0 points.
Measurement errors over the last 2 weeks
The mean predicted error ranged between –1.2 and 0.0 points. The greatest difference was found for ESSPRI score (−1.2 points (IQR: −1.8 to −0.1)). For fatigue, the difference was −1.0 points (IQR: −2.2 to 0.0). Maximum negative error ranged between −2.0 and −1.0 points, and maximum positive error between −0.3 and 0.0 points.
Utilisation and usability of the WebApp
The median number of days of WebApp utilisation throughout the study period was 75 (min–max: 9–89 days) (table 4). Data completion rate was 83.3% (IQR: 57.8–90.0). The differences observed between the 2-day and 7-day recall groups regarding these parameters did not reach statistical significance (p=0.08). In the group of participants who received reminder messages every 2 days, the percentage of data completion seemed high and stable throughout the study (from 90% during month 1 to 86.7% during month 3), whereas it was initially lower and subsequently decreased in the group of participants who received reminder messages every 7 days (from 81.7% during month 1 to 76.7% during month 3). Nevertheless, this numerical differences did not reach statistical significance at any point of comparison. The median SUS score was 90 (IQR: 85–95), indicating excellent usability. 77.8% of the participants reported being willing to continue using the application at the end of study. No characteristics associated with this status were identified.
Use of the WebApp over the 3 months period, in the whole population and according to the recall strategy
Discussion
In this study, we assessed the relevance of a novel EMA digital tool for capturing daily symptom intensity in patients with SjD. Our findings revealed that patients consistently used the WebApp over a 3-month period and rated its usability as excellent. Additionally, we observed substantial day-to-day variability in symptoms reporting and relevant differences with data obtained at the end of the study, through usual PROs. This suggests the relevance of this method for capturing data that closely reflect daily experiences of patients with pSS over a specific time period.
We found that the median daily variability of symptoms over the 3-month period ranged between 0.5 and 0.8 points across the different dimensions that were explored. This magnitude corresponds to more than half of the minimal clinically important difference for the ESSPRI score.16 Interestingly, the results suggest that ESSPRI is the PRO the less subject to daily variability. These results therefore confirm the ability of an approach involving daily assessment of symptom intensity to detect symptom fluctuations of a magnitude that may be clinically relevant. This constitutes only a first step towards the validation of PROs based on longitudinal data, whose properties will need to be demonstrated as superior to those of validated cross-sectional assessments.
The comparison between the data obtained through the WebApp and the conventional evaluation of symptoms at final visit also yields interesting results. Regarding the maximum negative errors, they were notably important, both throughout the period of utilisation of the WebApp (ranging from −3.0 to −1.0 points across scales) and during the last 14 days (ranging from −2.0 to −1.0), the latter corresponding to the timeframe of interest for the conventional assessment of these PROs. This was particularly evident for the ESSPRI score, with a maximum negative error of −2.3 points over the 3-month period and of −1.7 points over the last 14 days of the study. This suggests that the conventional questionnaire may be unable to capture certain relevant information related to patient’s daily perceptions of symptoms over a specific period of interest. As for the maximum positive errors, the differences were not as pronounced, and they were absent or negligible when focusing on the last 14 days of symptoms recording. These results are likely related to the formulation of the ESSPRI questionnaire, which asks participants to assess the maximum severity over the last 14 days.9 This suggests the absence of a significant recall bias when focusing on the maximum intensity of symptoms, even isolated, over the last 14 days. Thus, classical use of ESSPRI, as administrated in all SjD clinical trials, seems to reflect correctly the patient disease status over the past 2 weeks, in the strict perspective of maximum symptom intensity.
The modelling, through linear regression, of a predicted value at 3 months based on the WebApp data, demonstrates clinically relevant differences from the actual value obtained during the hospital visit. For all the evaluated subscales, the predicted value was generally smaller than that obtained through the usual questionnaire. For the ESSPRI score, the predicted error was −1.2 points for both periods of interest. These results suggest that modelling a final value based on longitudinally obtained data can lead to different outcomes compared with cross-sectional evaluation, favouring periods of lower symptom intensity in the final result. Having a tool that is more sensitive to periods of symptom improvement can be potentially valuable in the specific context of placebo-controlled therapeutic trials comparing PROs, by mitigating the impact of extreme values likely to be influenced by confounding factors.19
Our results show good patient adherence to the WebApp, with an overall data completion rate of 83.3%. Furthermore, they rated its usability as excellent, with a median SUS score of 90. Interestingly, 77.8% of the participants reported being willing to continue using the application after the end of the study. This is noteworthy, particularly in the context of potential extended use in future therapeutic trials. The results did not reveal any statistically significant impact of the frequency of SMS/email reminders on the number of application uses or data completion exhaustiveness. However, since the study was not powered for that specific purpose and considering the clear trends observed in favour of the group receiving reminders every 48 hours, this frequency will be selected for future trials.
Our pilot project has several limitations that need to be acknowledged. First, it is important to note that this study is purely observational, lacking a therapeutic intervention and a control group. As a result, we cannot currently generalise the differences observed between the two methods of recording PROs to a larger population of patients receiving interventions and experiencing potentially more significant changes in their symptoms. But based on our results and its interpretation, it is highly probable that some discrepancies will be observed with these two ways of recording symptoms. Second, the study’s duration was limited to 3 months, which does not provide insights into the long-term adherence to the WebApp, beyond the reassuring data obtained from the corresponding item of the SUS questionnaire. Indeed, only future studies will provide a comprehensive understanding of long-term adherence and help determine the optimal usage pattern, continuous or sequential, for the WebApp. Third, our study required participants to have personal access to a mobile device or the internet, which may theoretically limit the generalisability of the results. However, considering the widespread use of electronic devices, this limitation may have minimal impact. Fourth, a significant part of the study was conducted at the beginning of the COVID-19 pandemic, particularly during the first lockdown period in France. A specific influence of this context on the results obtained, particularly regarding PROs and adherence to the study procedures, cannot be excluded. Additionally, only a small proportion of the patients included in this study had an ESSDAI score of 5 or higher. Consequently, the results obtained here cannot be considered representative of populations with moderate to severe disease activity that are recruited in therapeutic trials. Finally, all these limitations will be addressed in the future, since this WebApp will be used and evaluated in the context of an international multicentric randomised controlled trial of the above-mentionned NECESSITY project, including two distinct populations (moderate to severe disease activity, unacceptable disease burden and limited organ involvement).22
In conclusion, the symptoms of SjD exhibit significant day-to-day variability in the majority of patients. This fluctuating nature of symptoms is believed to contribute to the challenges faced in conducting successful clinical trials for this disease. In order to address this issue, we have developed and preliminarily validated a user-friendly Web-Based EMA tool, called ‘PEPSS’. By capturing daily variations in symptom intensity, this tool has the potential to provide a more sensitive assessment of symptom changes and may contribute to the demonstration of effectiveness of therapeutic interventions, in the specific context of SjD.
Data availability statement
Data are available upon reasonable request. The data that support the findings of this study are available on request from the corresponding author. Some data are not publicly available due to the privacy protection of patients.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by CPP Sud-Ouest et Outre-Mer III (IRB number 2018-A021242-53) on 1 October 2018. Participants gave informed consent to participate in the study before taking part.
References
Footnotes
X @alain.saraux
Contributors DC, RS, SB and XM designed the study. DC is the guarantor. MC wrote the statistical analyses plan and conducted the prespecified and post hoc analyses. LG and DG wrote a first version of the manuscript, under supervision of DC. All coauthors contributed to the final version of the manuscript.
Funding This project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking (JU) under grant agreement number 806975. JU receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA. The present article reflects only the authors’ view and JU is not responsible for any use that may be made of the information it contains.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.