IJMSC Insights: MS and Apps With Sharon Stoll and Charisse Litchman

App-based monitoring and testing of individuals is a hot topic in the field of multiple sclerosis (MS). The potential of telemedicine became much clearer during the COVID-19 pandemic, especially for chronic conditions like MS which has varying symptoms that require consistent monitoring for best treatment. Apps could also help with long wait times, care shortages, and access issues, as well as facilitate data collection for clinical trials.

The March/April 2024 issue of the International Journal of MS Care (IJMSC) features an article “Validated, Quantitative, Machine Learning-Generated Neurologic Assessment of Multiple Sclerosis Using a Mobile Application.” The paper reports on a clinical trial with the aim of validating Expanded Disability Status Scale (EDSS) scores derived by the BeCare MS Link app by comparing them with scores derived by neurologists. By analyzing data from app-based assessments, artificial intelligence (AI) machine learning algorithms (MLAs) calculate users’ EDSS scores. The MLAs continuously learn as the data from users accumulates, improving score accuracy over time and minimizing score variability. Of the MLAs tested, the Huber regression model demonstrated high accuracy, with over 90% of scores predicted within 1 point.

Although the trial size was small and based in only 1 location, the results hold promise. The BeCare Link MS app demonstrated utility for monitoring MS stability, improvement, or progression and reduced interrater variability. The use of apps could be cost-effective and save time in clinical care and clinical trials.

Sharon Stoll, DO

Paper authors and neurologists Sharon Stoll, DO, and Charisse D. Litchman, MD, FAHS, met and worked together at the Yale School of Medicine and are both on the development team at BeCare Link. Founded in 2017 after Dr Litchman’s sister died from an unmonitored health condition, the company is developing and testing several neurology-based apps. They sat down with IJMSC to discuss their paper and their ideas about how AI and apps may influence MS care and monitoring.

Charisse D. Litchman, MD, FAHS: Basically, what we wanted to do is replicate what is done in the clinic and create a monitoring tool, a screening tool, but vary it based on the disease state. We started with MS because we wanted to validate our AI method. Our MS app is not just EDSS; it’s EDSS plus we have a number of other activities and questionnaires. To get the gold standard validation, we compared our AI EDSS with a clinician [derived] EDSS.

Charisse D. Litchman, MD, FAHS

Sharon Stoll, DO: One of the reasons that we were so excited…is that in clinical research, the gold standard of monitoring MS patients is the EDSS, the Expanded Disability Status Scale. And if you ask 9…of 10 MS clinicians (1) if they do this in the office themselves, they’ll say no, and (2) if they find that it adequately reflects the patient’s disability journey, they’ll say no, it doesn’t. However, for clinical trials, it is still done in the office, either by a fellow or a researcher or somebody that’s involved more with the clinical trial aspect….This is what we call a necessary evil. The test is extremely comprehensive; it takes 30 minutes or so to complete for every patient. It adds that much [time] to the visit. The 2 aspects that are most troublesome are that (1) when it’s done in a clinical trial, usually the patients come in every 3 months, and it’s done then, and (2) if the patient is there first thing in the morning versus 4:00 in the afternoon, the results will will change. It doesn’t mean that the medication is working or isn’t working, it just means that the patient has [only] so much energy to expend. By 4:00 in the afternoon, they’ve expended that energy, so it’s [definitely] going to alter those results; they’re not as accurate as they could be. More importantly, in the clinic itself, we usually only see patients once every 6 months and a lot can happen in that 6-month period. For secondary progressive MS or progressive MS, it can take upwards of 2 years before somebody is diagnosed, and this is something that we all know in the field. One of the reasons that I was so excited about this app was [because] it is a way that we can monitor patients in between clinical visits. And because we’re monitoring those patients in between visits, we can pick up on progression much earlier than we would if we were just seeing them in the clinic.

Another interesting thing that happened with this app was [that] it hit the market right before COVID. I don’t need to tell you how COVID has changed medicine. When you’re not able to see the patient in the office, you…can’t get an adequate measure of their disability. When we did the clinical trials in the office, it really was ideal timing because when the results came out during COVID, we were able to then offer our patients a way to monitor them [and] their disability, and do a complete physical exam, a comprehensive neurologic exam, more complete even than we do in the office. This is something that we can do online, and [we can] actually make adjustments to somebody’s medications based [on the results] without even seeing the patient [whether that is] because of COVID, other infections, ambulation issues, [or] people not being able to get to their visit because they’re wheelchair bound. These are all important [issues] that we knew [about] prepandemic, and the pandemic kind of sped up that time line.

Another main issue with the EDSS is that it doesn’t account for cognitive disability, and again, every neurologist is well aware of that. So 1 of the important features of the app is the cognitive [aspect]. We can really pick up on cognitive issues, maybe even before a patient notices [them or perhaps concurrently]. It [assigns] a number to that disability and is something that we can then document in our notes and [use to] justify a change in medication.

CDL: Putting on my CRO (contract research organization) hat, I can tell you that BeCare helps decrease interrater variability and it also helps with patients who [do not adhere to treatment regimens] because we’re able to monitor people from our decision portal. That’s what our IJMSC paper was about: How do we compare in terms of interrater variability when one of them (ie, the raters) is not a clinician, but [is instead] BeCare? The results, both in the paper that was published in your journal, and a subsequent paper that was conducted at Yale, show that there is a greater than 90% correlation.

SS: The paper showed that there was a 91.4% correlation between the AI feature and the clinical assessments, which was phenomenal. And our second trial [had] almost the same results, a 90[%] plus correlation between the clinician-derived EDSS and the AI-derived EDSS.…The studies were done a little while ago and since then, with AI, the more users and the more data that’s input into the system, the better the results get. How many patients do we have, or how many users [are] using the app right now?

CDL: We have about 8000 MS patients, and we found them organically. And it took us on to our next mission. The bottom line is [that] the MS app is fantastic for MS patients to monitor their disease and to empower them to tell their physicians when they think they’ve changed. And because as our therapies get better there are [fewer] relapses, BeCare may actually pick up on changes in EDSS or disability before the patient does. Our [second] app came out last fall, a general neuro app.

We’ve developed a subscription in which patients can get an absolutely complete report saying, [for example,] “You are within 1, within 2 standard deviations of the general population. And compared to your previous 2 scores, there’s been X amount of decline.” We do suggest that if you’re greater than 2 standard deviations, you contact your clinician.

The other great thing about [the BeCare MS app] is that the patient can print out the report and/or email it directly to their doctor and say, “Hey, I think I have optic neuritis.” And instead of waiting 6 weeks, the doctor or the nurse practitioner or the physician extender, [who is] looking at the results that day can say, “Yikes, let’s start the steroids tonight. Let’s change the trajectory of the way this flare is going to go.”

SS: On the physician side, when I’m following up with a patient, I tell them, “Download the app. Do it either weekly or monthly and then the next time I see you, we can go over your results.” If they notice something and they complete the app, the app correlates with what they’re noticing. There’s also a way that the patient can send me a message through the portal saying, “Dr. Stoll, I felt X and the app also confirmed X. Can I see you sooner? Can you give me a call back? I think I’m having a flare. I think I’m having a relapse.” Also, the patient can choose me [as their provider] when they download the app and I can pull up those measures. I can see all of their scales and how they’ve been doing over the past few months or so. I also get a readout. The coolest thing about the app is that I get an actual EDSS so I don’t have to sit there with my EDSS calculator calculating their scores.

CDL: … But the app is even more important for non-MS specialists. Over 75% of patients with neurologic disorders aren’t seen by a neurologist. Patients with chronic neurologic disorders are being treated by their primary care doctors, many of whom have never learned how to do a neuro exam and, if they are able to do a neuro exam, [they] may not have the confidence or the backup to make a treatment change. [With BeCare MS] they have that objective evidence. I can tell you that non-MS neuro specialists don’t do an EDSS. Most of them have never heard of the Timed Up & Go (a functional mobility test), so it’s an education for other physicians. More importantly, it’s a tool for the patients and the doctors, the non-neurologist, the non-MS specialist, to use together. There’s [also] a problem with accessibility [to neurologic care]—by 2025, there’s going to be an even further shortfall of neurologists.

SS: Yes, when I started practicing in Philadelphia, there was a waitlist [of more than 1 year] at the large academic centers to see a general neurologist.

A huge difference between this app and a lot of other MS apps—Charisse and I actually reported on a number of different MS apps a few years ago, so I’ve played around with my fair share of MS apps—[is that] this one is actually fun to use. It’s gamified…and people definitely want to sign back on and see their next score or if they’ve done better. And a lot of people might say, “Well, people are doing better because it’s the same app, it’s the same [games], and then they become more familiar with it.” [But] It’s not. Every time you open it, it scrambles the different games, so you’re not getting or seeing the same things pop up time after time after time.

CDL: There's a plateau in the learning curve. And we've we've taken that into account when we looked at the improvement.

IJMSC: How has the development of AI and the development of the machine learning algorithms helped with the development of the app? What advantage do these fairly new kind of tools give you as app developers?

CDL: The AI allows us to draw on so many more points of information than a clinician. We can bring in their scores, but also their treatment.… We test all 4 cognitive domains, but we can also assess cognition on other aspects of the app. The number of points of data is tremendous.

SS: Infinite, really.

IJMSC: Are there further iterations for BeCare MS? Are there further refinements that you are working on? Or do you consider this portion of the project complete at this point?

CDL: I don’t think anything is ever complete with AI. And, if it is, then you’re losing ground. As Sharon said, the more patients who use it, the better our data is. We are always changing our reports to make them more user friendly. We keep modifying it. And we learn when we go to another disease state and [then go back] and change the MS app.

SS: Also, every time [there’s a new phone], every time there’s a new update, the camera becomes better. The voice detection becomes better. The face recognition becomes better. Features on the app, they’re also going to become better.…We were able to add speech recognition and scanning speech as a feature.…So every time a new version comes out, it gives us better opportunity to also fine-tune some of the features without losing the data that we’ve already collected.

CDL: I would credit our ever-changing apps, or ever-improving apps, to the team. We have a small team, but it’s a very creative, very motivated team that works together. And somebody’s always coming up with a new way to do something better. So we don’t consider any project done. We move on to the next and the next one’s a little bit better. We add the improvements to the first.

IJMSC: So you started with MS and you mentioned the general neurology app...

CDL: The general neuro one was not tailored to the MS exam. We’ve borrowed some of it. We are heavily screening for Parkinson [disease]; we’re heavily screening for cognitive impairments, for speech. We’re hoping to do early detection for Alzheimer [disease], for Parkinson [disease], for other neurodegenerative disorders. And we are about to release our third app, which is one [that measures] neurotoxicity.…It’s also neurobehavioral, so this app is really leaning toward a mental health app for PTSD, depression, anxiety. We have also developed 3 external sensors, one of which is a restrictive lung device. It is not a spirometer. It measures restricted lung disease, for example, somebody with neuromuscular weakness, for [those with] ALS, myasthenia. Hopefully, it [would identify] somebody before respiratory failure [occurs] so intervention can be made to keep them from going to the hospital. We also do dorsiflexion and grip. And [all] those results [would be incorporated] into the BeCare app and be a part of the analysis. So we’re very excited about that.

[An]other thing I wanted to point out that I don’t think we mentioned is that when people do an activity they go right to the overview. [While] on the app, they can say, “Okay, how did I do? How did I do last month? How did I do 6 months ago?” They can get immediate feedback that’s subjective.

SS: Right, they get their own score, which is, I think, one of the reasons that patients like using it. It’s not just [that] I see their score, but they see their score [and] they understand it. Another thing that makes this app very, very useful, at least for clinicians, is that when patients first use it, they input all of their data: age, sex, which disease modifying therapies they they’ve been on in the past, how long they were on them, why they stopped, what they’re currently on. It really helps us with monitoring. When I see a new patient [who] has had the disease for 20 years, they don’t remember what they were on 20 years ago. [BeCare] is also a personal diary that the patient can keep….They can input any changes in, for example, diet, how often they exercise.…All of that is in the app. And it also helps us pull up different things that we can then study. Does it make a difference in MS patients [who] are on drug X? If they exercise 1 day a week versus 5 days a week? Does that make a difference? Common sense would say yes, obviously, but you don’t know until you do the clinical trial. Again, the more users, the more data we have, the more fine-tuned it becomes. I think that’s really where the future of not just MS, not just neurology, but the future of medicine is really headed.

CDL: One of the main benefits for BeCare is for all of those patients who are in neurologic deserts. BeCare really acts as a walking EMR (electronic medical record) for their neuro disease. They can send their information to somebody…up at Yale or up at the Brigham (Brigham and Women’s Hospital) and they know they have all that information.

SS: Especially in this country where we don’t have socialized medicine, where not just every state is its own separate IT system, but every institution is its own separate system [and] you can’t share records with patients from one health care system to another. When I switched states—[previously in] Connecticut, now practicing in Pennsylvania—if somebody wanted to come see me in Pennsylvania, in the past, I would have said, “Absolutely not. It’s too far. God forbid something happens; I can’t do anything.” Or via telehealth, I wouldn’t feel comfortable. If you’re having a relapse, I can’t assess you. Now I can. I would feel more comfortable treating somebody in a different location [when] utilizing an app like BeCare.…They will still need a local neurologist, but now I can help that local neurologist make a change.

IJMSC: Dr Stoll, could you comment on how you see AI and technology being used in your care of people with MS 5 years down the road, 10 years down the road. What does the future look like to you?

SS: In 5 years, I see something like BeCare being used more frequently. I think a lot of our patients push us as clinicians into uncomfortable territory. A lot of patients are very well educated. When they come in, it’s not just even with the new diagnosis, it’s, “I think I have this. These are my symptoms. And these are the different clinical trials that I found online on nih.gov. And these are the medications that I know are available.” And that’s intimidating for me. It’s also intimidating, I’m sure, for a general neurologist that isn’t as up-to-date on on specific neurologic diseases like MS or Parkinson [disease] or epilepsy. I know [that] if somebody with epilepsy came to me, I would not be as up-to-date on all of the new antiepileptic medications. So I think that care is only going to get better and more fine-tuned because tools like this are available to the general population. Patients [have] already honed in on their specific disabilities and their specific dysfunctions.

I also know there’s an AI component to a lot of the imaging modalities that we’re using in the field. Looking at total brain atrophy, thalamic atrophy, even lesion size change over time, [all that] uses AI components.…So I think 5 years from now, AI is really going to be a huge part of our lives, and a huge part of practicing MS care. And I think the more patients become involved with their own care, the more they’ll utilize tools like this, and the more we’ll be able to trust those measures.

CDL: At BeCare, we have specific goals in mind with where we want to go with AI. Number 1, we think that through analysis of how these patients have done, we can look at the algorithms to predict the best personalized treatment plan, which is the best group of medication[s] for any given patient. We think we’re going to be able to do a predictive analysis based on how they’ve been doing over time, what amount of disability they’re going to have down the road, and that plays into whether or not it’s time to receive riskier medications. We also want to look into virtual rehabilitation…and cognitive training.…And finally, because we’re going to have a better idea of people’s personal history, we’re looking into developing an alliance or partnership with a community app; think supportive care and patient education.