The Myth of Multiple Outcome Measures

There is nothing wrong with using an additional outcome measure. For example, using the PHQ-9 as a periodic measure in addition to the every session Outcome Rating Scale (ORS) makes total sense with folks presenting with depression symptoms as their reason for service. Or using the PHQ-9 with a depression group or the GAD-7 with an anxiety group while using the ORS with the rest of your caseload makes sense as well. But thinking that anyone can navigate the use of 250 different outcome measures with any degree of competence or consistency is complete mythology. But that is what is being sold these days.

Measurement Based Care

Measurement based care is all the rage. It is already a requirement of accrediting bodies, lauded as “an evidence based” practice, and soon to be a Professional Practice Guideline of the American Psychological Association. While many are claiming their approach or system is evidence based under the generic rubric of MBC, only two systems are truly evidence based, meaning they have randomized clinical trial support: Michael Lambert’s Outcome Questionnaire (OQ) system and our beloved Partners for Change Outcome Management System (PCOMS).  These two systems account for most of the clinical trial evidence supporting all the approaches, accounting for 74% of a recent meta-analysis. In other words, the now many available systems that tout their evidence-based status are standing on the accomplishments of others, namely Lambert and colleagues and Duncan, Reese, and colleagues.

Systems other than the OQ and PCOMS are about as evidence based as election fraud was in 2020.

Of course, marketing rules and trumps (pardon the word) real data every day of the week and twice on Sunday.  And MBC is super hot right now and is considered prime for investment and expansion based on the above requirements and the increased need for mental health and substance use services often reported by the media. I get at least two calls a month from private equity investors looking to invest in or buy Better Outcomes Now (BON). We resisted investors from the very beginning because they ultimately take control of your business as well as want your first born in terms of repayment. Given we didn’t take money from investors, it took us three years to build BON (2013-2016) but we retained control and kept it true to PCOMS in both spirit and intent—to privilege client perspectives and culture while improving outcomes and reducing dropouts.

MBC Companies Arriving Like Trump Indictments

The result of the market being flaming hot is that MBC companies have sprung up like Trump indictments, accompanied by slick websites and ubiquitous social media presence. Not deriding free enterprise here but the army of marketeers end up convincing leaders of behavioral health of some very stupid things. Case in point: The more the merrier; the more outcome measures available to choose from, the better the MBC system. Consequently, I read a proposal recently that required the inclusion of 250 outcome measures. Complete and utter nonsense that reflects a fundamental misunderstanding of measures and implementing MBC.

Let’s break this down. First the technical side of things. Systems that purport to include 250 measures or more are including measures that are in the public domain. In other words, they are free. There is a wide range of psychometric and normative integrity in such measures. Some are reliable, valid, and have normative metrics to understand their scores like clinical cutoffs and indices of reliable and clinically significant change, like for example, the PHQ-9. (The downside of the PHQ-9 is that it doesn’t cover other reasons for service resulting in a high percentage of clients scoring in the non-clinical range and not showing benefit over the course of therapy). But many, if not most of the included measures do not have a normative base that provides any true meaning to the scores. An eating disorder measure, for example, may have been validated in a study, but it has no real world data to base an understanding of what a change of score means. If it is reported that a given client attained a 20% reduction of score on the measure, what does that really mean? And, if you are using five different measures for five different programs within your agency, how can you compare the outcomes across programs. Given the range of integrity and the different psychometrics, it would be impossible. Finally, no measure available for free has a significant database, algorithms, or expected treatment response (ETR) trajectories that help you understand how clients typically respond in therapy, and importantly, alert you when clients are not benefiting as expected.

Then there is the practical side of having 250 measures to choose from. Anyone who has implemented MBC knows that it isn’t for the faint of heart. With almost 25 years of experience implementing just one outcome measure, I know that It takes a sustained effort over a significant period of time to successfully implement any MBC system. Therapists don’t take to it like a duck to water. Many have negative perspectives and fears that must be assuaged for success to occur. Training is essential and so is leadership. Any system that is seen to encumber the therapy will be quickly cast aside. In fact, any measure that takes more than five minutes to administer, score, and discuss will likely be unused. Feasibility, in other words, is key to implementation. But how feasible is choosing among 250 forms for a mixed caseload? Do you think therapists will track down a different measure for each client, and understand its parameters of change for discussions with clients?

Of course, if you only want to flick forms to superficially meet accreditation standards, and it doesn’t matter what measure you use or how you understand the data, or even if you use the data to inform practice, then pick any measure from a dropdown list.

Unfortunately, the appearance of doing MBC is all that is important to some. But if you want to privilege clients and implement a transparent, collaborative MBC process that has been shown in peer reviewed, published studies to improve outcomes and decrease dropouts, and you want data that mean something beyond flicking forms and supported by millions of administrations, algorithms, and ETRs, that’s a horse of a different color.

Organizations are implementing every day. With over 400 organizations on board, 2 million + administrations in our database, and over 250 thousand clients helped, we know what we are doing.