Tuesday, September 30, 2014

AHRQ Draft Report -- Fundamentally and Irredeemably Flawed

[This is a blog post by Erica Verrillo. She mentions below: "Anyone
who wishes to reprint this blog post may do so. (Remember to link back
to the original and provide attribution."

Aside: I posted Jennie Spotila's review of the draft yesterday but it
never came through on my e-mail account. It's here:
http://bit.ly/1rzvZWw   i.e.
http://www.occupycfs.com/2014/09/29/a-review-of-the-p2p-systematic-review/ .

http://bit.ly/1mNQ76Z   i.e.
Monday, September 29, 2014

The AHRQ Draft Report - Fundamentally and irredeemably flawed

The Jury, by John Morgan
The Agency for Healthcare Quality and Research (AHRQ) has released its
draft report on the Diagnosis and Treatment of Myalgic
Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS).

For those who find it hard to keep track of the plethora of reports,
committees, panels, and reviews currently underway, the AHRQ report
provides the basis for the Pathways to Prevention, or P2P Workshop.
The P2P Workshop has been convened by the NIH for the purpose of
making a report that will be used to evaluate research grants for
ME/CFS, and upon which pharmaceutical companies will base their
clinical trials.

The P2P panel is composed entirely of non-experts. The preference for
non-experts was intended, according to Susan Maier, Executive
Secretary for the NIH Advisory Committee on Research on Women's
Health, to act like a "jury," by which she meant the composition of
the panel was meant to be unbiased.

Leaving aside for a moment the absurdity of consigning a research case definition for a disease to people who aren't even physicians, the
jury system itself does not work. Trial by jury is one of the most
inefficient systems of justice in the world. Swayed by bombastic
arguments, prejudice, and crocodile tears, juries of twelve unbiased
peers routinely find innocent people guilty, and allow the guilty to
go free. As a method for determining who gets grants for medical
research, anything even remotely resembling the jury system is
entirely inappropriate.

I have touched upon some of the short-comings of the draft report
below. I've tried to keep it brief. (There is so much wrong with this
report that if I were to write a thorough critique, it would be longer
than the report itself.)

You can read the AHRQ draft report and appendices here

You can make comments (until October 20) here

You can read more about the P2P in these posts:

Advocates to NIH - "Pull the P2P!"
All the reasons why the P2P is dangerous

Protocol for Disaster?
Jennie Spotila explains why the AHRQ report is a recipe for disaster,
and how they pulled a "bait and switch" by changing their Key

P2P: The Question They Will Not Ask
Mary Dimmock and Jennie Spotila explain how refusing to consider how
CFS and ME differ will affect the results of the P2P Workshop.

Note: Anyone who wishes to reprint this blog post may do so. (Remember
to link back to the original and provide attribution.)


By Erica Verrillo

Point 1: The draft report shows limited understanding of the illness

The report begins with the erroneous statement that "The term ME was
first used in the 1930s after an outbreak of neuromyesthenia ..."

A quick google search would have revealed that "myalgic
encephalomyelitis" was first used in a letter to the editor entitled,
"A New Clinical Entity?" published in the Lancet in 1956. The authors
– Emile Nihoul, Lise Quersin-Thiry, S.Chalmers Parry and Robert A.
Good – were referring to what became known as Royal Free Disease, an
outbreak that occurred in London's Royal Free Hospital in the 1950s.
Obviously, none of the members of the panel felt inclined to check
their facts.

The report goes on to say that "Uncertainty persists regarding the
etiology and whether the condition reflects a single pathologically
discrete syndrome, subsets of the same illness, or a nonspecific
condition shared by other disease entities." This statement reflects a
thorough misunderstanding of how the illness is perceived by people
who investigate and treat it. There are no experts in ME/CFS who would claim that it is a "nonspecific condition shared by other disease entities." There are, however, numerous researchers who author papers
on "chronic fatigue" and "fatiguing illnesses" such as cancer and MS.
The authors of the AHRQ draft report, unlike experts in the field, do not have enough background to be able to distinguish between "chronic
fatigue" and ME/CFS.

While the introduction to the report is not crucial, it does indicate
that the people writing it not only had no knowledge of the illness,
but that they did not want to spend any time acquiring it. The willful ignorance that is demonstrated in the introduction permeates the entire report.

Point 2: Problematic Search Methods

In order to find abstracts and articles, the AHRQ searched three main
databases using the terms: fatigue; Fatigue Syndrome, Chronic; and
Encephalomyelitis. With the notable exception of PsycINFO, a database
of abstracts of literature in the field of psychology produced by the
American Psychological Association, these are the same databases used
by the Drug Class Review: Drugs for Fibromyalgia: Final Original
Report <http://healthandwelfare.idaho.gov/Portals/0/Medical/MedicaidCHIP/FibromyalgiaFinalExecutiveSummary.pdf>
published by the Oregon Health & Science University in 2011. Ovid and
EBM/Cochrane are large medical databases, though they don't
necessarily include every study conducted on a given illness or
condition. Only controlled trials are included in the Cochrane
<http://www.cochrane.org/cochrane-reviews> databases.

The most glaring problem with the search is that it included studies
on "fatigue." Indeed, a number of studies included in the review were
on "fatiguing illnesses" rather than ME/CFS. Like the introduction,
the search reflects a state of confusion on the part of the authors.
The confusion is not altogether surprising, given that researchers
also appear to be confused about the difference between CFS and
chronic fatigue. Nonetheless, experts in the field are not confused.
They are aware that while ME has been used abroad since the 1950s, it
has not been used as a diagnosis here in U.S. Specialists have been
limited to CFS as a diagnosis, like it or not.

A second problem is that with the perennial lack of NIH funding for
ME/CFS controlled trials, much of the information about treating the
disease is based on clinical observations. None of these were
included. nor were studies that were controlled, but which did not
meet the set of criteria for inclusion in the review – such as
addressing the Key Questions.

Point 3: Studies used for the report are inadequate to address the Key Questions

The studies that the reviewers included were not only too few, they
were completely inadequate to properly address the Key Questions.

The Key Questions to be addressed by the report are as follows:

1. What methods are available to clinicians to diagnose ME/CFS and how
do the use of these methods vary by patient subgroups?

a) What are widely accepted diagnostic methods and what conditions are
required to be ruled out or excluded before assigning a diagnosis of

b) What is the accuracy and concordance of diagnostic methods?

c) What harms are associated with diagnosing ME/CFS?

2. What are the (a) benefits and (b) harms of therapeutic
interventions for patients with ME/CFS and how do they vary by patient

a) What are the characteristics of responders and non-responders to

There are problems with the wording of some of these questions. For
example, in a country in which 80% of the physicians don't believe
that CFS is a real disease, what could "widely accepted" be referring
to? And, "What harms are associated with diagnosing ME/CFS?" seems to
have an a priori assumption that diagnosing the disease may in itself
cause harm. But aside from the oddness of the wording, the studies
they chose do not adequately address the questions.

The criteria for exclusion from the review included, among others,
that the study did not last not long enough (therapeutic trial of less
than 12 weeks), was published before 1988, had wrong study design, or
did not address a Key Question. (There were 8 more exclusions.)

From among the thousands of studies that have been conducted, the
criteria limited the review to a scant 64 studies. Some of the
landmark studies that were excluded were all of the studies
demonstrating immune dysfunction
(e.g. NK cell deficiency studies by
Brenu et al.), studies of viral reactivation and antiviral treatments
(e.g. all Lerner and Jessop studies, Kerr parvovirus B19 study),
studies documenting brain abnormalities (e.g. Lange's MRI study), and
all of the papers published by Tom Kindlon on harms associated with
GET and CBT. Not even appearing on the excluded list were the
ground-breaking 2-day CPET studies conducted by Keller, Stevens and
Snell, Peckerman's cardiac insufficiency studies, and the recent
Watanabe study on CNS inflammation.

The fact that some of the most significant studies in the ME/CFS
literature did not even appear on the excluded list was mind-boggling.
Of the studies that appeared on the exclusion list, the reasons given
were various, but among the most frequently cited were that the
studies did not address the Key Questions. Yet, several studies that
directly addressed the Key Questions were omitted (for example, 2-Day
CPET studies were not even considered), while studies that did not
directly address the Key Questions were included. This arbitrariness
permeated the entire study selection process.

(Going though the studies that were accepted I found three that did
not meet the criteria for inclusion without reading further than the
first page. I also found studies in the excluded section that met the
criteria. Having no experience with ME/CFS, the panel lacked the
ability to distinguish relevant from irrelevant studies.)

Point #4: Contradictory and unsupported conclusions

In terms of treatment, the report was heavily weighted toward
psychological studies. Out of the 36 studies used to address Key
Question 2, 14 concerned CBT. Considering that the PACE trial was
included, but not any of its critiques, it is not surprising that the
report favored CBT:

"Based on 13 trials, cognitive and behavioral therapy (CBT), either
group or individual; self-instruction booklets; pragmatic
rehabilitation: peer-to-peer counseling: and symptom consultation
provide improvement in fatigue, function, quality of life, and
employment in adult patients with ME/CFS."

Yet, after a detailed examination of the actual results of the trials,
the report went on to conclude that:

"In summary, head-to-head trials had mixed results with two trials
finding improvement with GET, two trials finding improvement with CBT,
and one trial finding no differences between CBT, GET, and usual care.
In considering non-head-to-head trial data, there is low strength
evidence that CBT and GET provide similar improvement in measures of
fatigue and/or functioning."

Those reading this report will base their recommendations on the first
statement, as the second statement requires wading through a lot of
statistics. They won't even realize that the conclusion that CBT
provides "improvement in fatigue, function, quality, and employment"
is ultimately derived from the results of a single, deeply
manipulated, study. (The PACE trial.)

In terms of Key Question 1, the report suffered from similar
inconsistencies, For example, the report concludes with the statement
that "the negative effects of being given a diagnosis of ME/CFS appear
to be ... universal."

I could not find a single study from among the list of included
studies that would support the conclusion that the diagnosis of ME/CFS
had negative effects universally.

In the Asbring and Narvanen study
<http://www.ncbi.nlm.nih.gov/pubmed/11837367> entitled "Women's
experiences of stigma in relation to chronic fatigue syndrome and
fibromyalgia," the authors concluded that " The women experienced
stigmatization primarily before receiving a diagnosis." [Emphasis
mine] In addition, they stated that "Stigma consisted of questioning
the veracity, morality, and accuracy of patient symptom descriptions
and of psychologizing symptoms."

The Dickson et al. study
"Stigma and the delegitimation experience: An interpretative
phenomenological analysis of people living with chronic fatigue
syndrome," reported that "participants reported delay, negotiation and
debate over diagnosis: further, they perceived their GPs to be
sceptical, disrespectful and to be lacking in knowledge and
interpersonal skills." [Emphasis mine]

In the Green et al. study
"Stigma and Chronic Fatigue Syndrome," the authors state that "Most
subjects (77%) were labeled as 'psychological cases' by one or more of
the physicians (mean = 8) consulted, but of the 4 stigma measures,
only disclosure was related to physician labeling." This means that
patients only felt stigmatized by their physicians after they
attempted to educate them about ME/CFS.

There was nothing in these studies to support the claim that the
diagnosis itself had negative effects. Rather, it was the delay in
diagnosis, and subsequent debate on the part of family and physicians
– as well as the delegitimation resulting from a trivializing name –
that led to negative effects.

In sum

The conclusions reached in this review were the result of a poorly
framed set of key questions, a literature search that managed to
exclude the most fundamental research studies, and a misinterpretation
of the studies that were eventually deemed acceptable for inclusion.
As a whole, this report is fundamentally, and irredeemably, flawed -
even given its narrow search results.

These major short-comings are the inevitable result of appointing a group of people who have no expertise in ME/CFS to evaluate 26 years of research. ME/CFS is a disease that demands expertise. It cannot be  evaluated be a panel of non-experts.
- See more at: http://cfstreatment.blogspot.ie/2014/09/the-ahrq-draft-report-fundamentally-and.html#sthash.Gh65Hra9.dpuf

No comments: