A Digital Health Poster Child for Reporting Invalid Savings

We’ve never seen a shorter abstract with more curious findings than the one done by Dignity Health and Dr. Rajan Merchan evaluating Propeller Health's gadget.

We’ve seen shorter abstracts, and we’ve seen abstracts with more curious findings, but we’ve never seen a shorter abstract with more curious findings than this one, done by Dignity Health and Dr. Rajan Merchant, and financed by the California Healthcare Foundation, evaluating a gadget made by Propeller Health.

The study group’s use of inpatient care for asthma declined by a whopping 62% vs. the control group. You might think this result violates Dr. John Ioannidis’ well-known conclusion that large treatment effects are usually wrong, but you’d be mistaken. You see, there was no treatment here.

There was only an effect. Dr. Ioannidis’ result applies only to actual comparisons of effects due to different treatments, not to random changes in effects using the same treatment. In this study, the actual treatment protocol was the same and the inhalers were the same.

The only thing different was the frequency of drug use. Whereas the conventional wisdom for disease management states that hospitalizations can be avoided by more adherence and hence more drug use, in this case, the study group used less medication than the control group, reaching for their rescue inhalers 25% less—once every 6.3 days vs. every 4.7 days for the control group.

The inevitable conclusion would be that the study group needed far fewer inpatient days and considerably fewer rescue puffs because their asthma was controlled much better. And that brings us to the most counterintuitive and impressive finding of all.

 

The most impressive part of the study

Yes, the most impressive part of the study was that this 62% reduction in inpatient days and 25% reduction in rescue drug usage was achieved even though overall asthma control improved similarly[1] in both groups. Specifically, actual patient scores on the Asthma Control Test improved only by a statistically insignificant 2% in the study groups vs. the control groups.

In other words, the Propeller Health intervention had no effect on relative asthma control scores.

Putting all this information in a table makes it clear that Dr. Ioannidis is going to have to re-examine his conclusions because this Propeller Health study shows that it is indeed possible to achieve a massive reduction in events and costs even if nothing else changes except frequency of drug use, which changes in the “wrong” direction:

 

The Wishful Thinking Multiplier

Nonetheless, let’s give the Propeller Health intervention credit for that 2% improvement in control scores. Avid readers of our wellness postings and Why Nobody Believes the Numbers may recall that the ratio of event reduction to quality score improvement is called the “Wishful Thinking Multiplier,” and Propeller Health’s Multiplier of 31x just shattered Viverae’s record of 17x.

(The real answer, of course, is that on a good day event changes mirror quality improvement changes. That itself could be wishful thinking, and there are many reasons why even a 1x Wishful Thinking Multiplier is difficult to achieve, as described in the book.)

We’re also not sure where exactly Dr. Merchant found these asthmatics but, wow, are they ever sick. During the baseline period, they were running 326 asthma inpatient days per 1000 asthmatics. A typical group of 1000 asthmatics will incur about a quarter that many inpatient days.

Perhaps that’s because when these folks show up in the ER they almost always get admitted: Whereas the typical asthmatic has a ratio of about 5-to-2 of ER visits to inpatient days, these folks were averaging less than 1-to-2. Neither of these statistical anomalies seemed to bother Dr. Merchant enough to explain why his population might be such an outlier.

All this leads up to the savings figures. Annualized gross savings were $688 in the study group. Assuming a typical commercial health plan’s typical 4% asthma rate, $688 in savings/asthmatic equates to about a $27 per-member-per-year (PMPY) savings. Nice work when you consider that we’ve never seen a commercial health plan that spends more than $20 PMPY on total asthma ER and IP events, and most spend about $15.

You can’t reduce a number by more than 100%

And if readers learn only one thing from our books and columns, it’s that you can’t reduce a number by more than 100%, no matter how hard you try.

Two postscripts are in order. First, no one has ever accused us of being nuanced, so for a more nuanced view (but nonetheless a view generally consistent with ours) of Propeller Health, we might suggest David Shaywitz’ column in Forbes. Shaywitz points out, among other things, that the results are still preliminary.

Shaywitz is right, but Propeller isn’t letting such subtleties as a study which only has 4 months’ data and impossible results spoil their party: Their home page announces that “patients on Propeller save more than $600 a year.” It further suggests that visitors “find out how” but no need—you now already know.

Second, we hope that the individuals involved with this study, who fortunately for readers on THCB turned down our offer to critique their study privately, decide after reviewing this posting to make their final results actually appear even slightly plausible, as opposed to epidemiologically and mathematically impossible, or at least explain how they achieved such massive savings and event reductions in the absence of increased drug use and asthma control scores.

The bad news, though, is that recent history shows reaction to our column on the part of perpetrators is exactly the reverse. Recently Al and I outed Aetna for pitching questionable obesity drugs to people who aren’t even sick and didn’t request them. Shortly thereafter, the Journal of the American Medical Association piled on with a “special communication” pointing out that these drugs are so unsafe they aren’t even approved in Europe.

 

Aetna doubled down

Faced with that information, instead of backing down Aetna doubled down, a Tom Sawyeresque $300 will afford you the privilege of watching Aetna sell this program via a webinar.

We wonder if Propeller Health will be “hovering” on this webinar to figure out how they too can charge $300 to sell their program to attendees who haven’t read this column.


[1] Of course, that “improvement” was simply regression to the mean, perhaps with some Hawthorne Effect thrown in, due to the selection criteria favoring previously high utilizers.  However, we are focusing on the differences between the groups, not the groups compared to the baseline. Nonetheless, we would wonder why a principal investigator can’t distinguish between regression to the mean and actual improvement.

First posted on The Health Care Blog 3/21/2014.

Books by Al Lewis & Vik Khanna:

5 COMMENTS

  1. Two related points:

    (1) I wish you hadn’t posted this. My math was all wrong and it turns out their results were correct, and they’re already suing me, and this doesn’t help.

    (2) April Fools.

    PS I’d encourage people to look at the comments to this one too — at http://www.thehealthcareblog.com. This was one of those rare instances where the perpeprator, Dr. Merchant, responded. He apologized for the errors and said he would fix them in the final version and defended Propeller Health and said they had nothing to do with it. April Fools again. Merchant basically threw them under the bus.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.