Submission 166 — Ms Cathie Davies — Select Committee on Autism | External submission

Submission to Select Committee on Autism

Some Comments on the CRC Umbrella Review of

Research Addressing Interventions for Autistic Children

Cathie Davies

April, 2021

No machine-readable text was extracted from this page.

On 3 March, 2021, Professor Andrew Whitehouse addressed the Senate Select1. “Enforced” Choice and Control? Committee on Autism in Brisbane. His topic was a recently published synthesis of

intervention research for children on the autism spectrum. The “umbrella review” had been

commissioned from the Autism Cooperative Research Centre (CRC) by the National

Disability Insurance Agency (NDIA), for the purpose of informing “decision-making and the

future development of a Guideline” (Whitehouse, Varcin, Waddington, Sulek, Bent,

Ashburner, Eapen, Goodall, Hudry, Roberts, Silove, Trembath, 2020, p.21). Professor

Whitehouse was the lead author of the review.

During the hearing, the Chair (Senator Hollie Hughes) is reported to have asked the

following question:

How do we ensure that service providers and some of the larger autism organizations,

which seem to … almost have state based responsibilities, are actually utilizing those

programs [for which there is research evidence]? I’m talking about service providers

like speech pathologists, but, more to the point, I’m more concerned about

occupational therapists who are still using a lot of sensory-based therapy options

swinging in a swing, or climbing through one of those socks. They are still being

utilized when we know that they’re not evidence-based therapies. How do we stop the

chelation, the riding of horses, the music therapy, the pure dietary interventions that

we all know aren’t evidence based? How do we, in some way, enforce that?

(Commonwealth of Australia, 2021, pp.11-12).

Professor Whitehouse’s response was as follows:

Ultimately, it’s a hand-to-mouth existence in the disability service provider world. It’s

a really tough sector to make ends meet. I believe that the government plays a major

role as the key funder to enforce the standards. … My view, and I believe it’s the

view of the Autism CRC, is that we need to demand the same standards for autism

that we demand in the standards for every other area of health and medicine. That

means defining what is evidence based practice and sticking to it. That is definitely

going to be an adjustment period for the sector. Nevertheless, if we are funded

through public mechanisms, we need to demand that, not just for the taxpayers but

also for the kids, adolescents and adults we seek to help. We have a finite bucket of

money and we need it for the service, so they can live happy, healthy and dignified

lives, as we would want (Commonwealth of Australia, 2021, p.12).

The Chair’s question and Professor Whitehouse’s answer appear to be entirely at odds

with the declaration in Whitehouse et al. (2020) that “the current report is primarily

concerned with the selection and use of interventions within an evidence-based practice

framework. … The framework does not impose a prescriptive approach on service provision

… .” (p.19). To make sense of the apparent tension, it is necessary to be aware of two quite

separate and potentially incompatible concepts: “Evidence-Based Practice” (EBP) and

“Evidence-Based Practices” (EBPs).

Evidence-Based Practice (EBP) is a model derived from Evidence-Based Medicine

(EBM), in which clinical and policy recommendations are informed by research evidence;

practitioners’ clinical experience; and consumer factors including values and preferences. It

has been endorsed by a wide range of Australian health and allied health professional bodies.

Section 2.1 elaborates on EBP.

Evidence-Based Practices (EBPs), on the other hand, refers to interventions identified

solely on the basis of research evidence. The term was coined in the “political milieu” of

twenty-first century American human service policy and its applicability to Australian

contexts is, thus far, unexamined. Section 2.2 elaborates on EBPs.

The almost identical terms belie a vast conceptual difference, and confusion is

common. Although this paper uses the acronyms “EBP” and “EBPs” to disambiguate the

two concepts, it is important to recognise that “Evidence-Based Practice” may be used to

denote either, depending on the speaker or author. Further, it may be used by less informed

individuals without a clear understanding of which meaning is intended. Indeed, throughout

Whitehouse et al. (2020), the meaning appears to fluctuate from page to page – and even, at

times, on the same page1 - perhaps reflecting alternative positions held by different co

authors. Such ambiguity threatens to make transparent development of clinical and policy

recommendations extremely challenging. Yet transparency is a central goal of EBP in both

clinical and policy contexts (Satterfield, Spring, Brownson, Mullen, Newhouse, Walker, &

Whitlock, 2009, p.371; Schünemann, Brożek, Guyatt, Oxman, 2013, p. 22).

While Whitehouse et al. (2020), appears to present conflicting meanings, it seems

clear that Senator Hughes and Professor Whitehouse were referring to Evidence-Based

Practices (EBPs) at the Senate Select Committee hearing. This is deeply disturbing. There

has been no attempt to examine the applicability of the EBPs approach to Australian contexts,

and certainly no publicly announced decision to adopt that approach. Any such decision

would have a deleterious effect on the quality of research, as argued in sections 2.3 to 2.5. It

would also remove practitioners and consumers from the process, or at the very least relegate

their input to an afterthought. Such a decisions should not be made in error, based on

confused language. It should certainly not be made by stealth.

The architects of EBP (Haynes, Sackett, Gray, Cook, and Guyatt, 1996, p.197;

Sackett, Rosenberg, Gray, Haynes and Richardson, 1996, p.72), together with the editors of

1 For example, on page 19 a description of EBP consistent with EBM is given in a section of the report that bears the title “Evidence-Based Practices” (Whitehouse et al., 2020, p.19)

GRADE2 (Schünemann et al., 2013, p.29), have made it very clear that there is no place in

their models for “enforcing” the use of specific interventions (see section 2.1). Even authors

who are clearly writing about EBPs in the American context have said: “As for concerns that

EBPs in special education will force instruction to conform to an approved menu of

interventions, we believe that EBPs will not and should not ever take the place of

professional judgment …” (Cook, Tankersley, & Landrum, 2009, p.380).

The remainder of this paper discusses how the choice between EBP and EBPs may

impact research, practice and policy (sections 2.3 to 2.5); the suitability of Whitehouse et al.

(2020) for its intended use as a foundation for clinical and policy recommendations (section

3); some of the umbrella review’s practical limitations with respect to appraisal of

intervention research (section 4) and, as an example, the review’s appraisal of research

addressing one specific intervention, Facilitated Communication (FC) (section 5).

2.    Evidence-Based Practice, or Evidence-Based Practices?       In a description that accords well with models of Evidence-Based Practice (EBP)2.1.   Evidence-Based Practice

based on Evidence-Based Medicine (EBM), Whitehouse et al. (2020) acknowledges

limitations in the research presented by the review, and notes the importance of evidence

from other sources:

The variability in the effects of interventions on child and family outcomes and the

quality of the evidence reinforces the need for clinical decision making to take place

within an evidence-based practice framework, which also draws on evidence from

clinical experience and accounts for each child’s and family’s preferences and

priorities (Whitehouse et al., 2020, p.15).

2 Grading of Recommendations, Assessment, Development and Evaluation (GRADE). Current National Health

and Medical Research Council (NHMRC) advice to reviewers and guideline developers is now based on GRADE (NHMRC, n.d.)

EBM was conceptualised and hailed as a “new paradigm” in the early 1990s. By

1996, the iconic “three circles” model had emerged, figuratively represented by three

overlapping circles labelled “clinical expertise”, “research evidence” and “patient

preferences” (Haynes et al., 1996, p.196). The model has continued to evolve, with versions

of it now adapted for use in a wide range of health-related disciplines in many parts of the

world. The core features remain, i.e., decisions are made on the basis of an integration of

knowledge from research; clinical experience; and consumer factors (e.g., values,

preferences, circumstances, etc.). In Australia, the NHMRC have added “availability of

resources” to the mix (NHMRC, 1999, p1), while the definitions of EBP used by Speech

Pathology Australia (SPA) and Occupational Therapy Australia (OTA) add information from

“the practice context” (SPA, 2020a, p.19; OTA, 2019, p.7).

As explained in one of the seminal 1996 papers that outlines the three-circles model

of EBM, “it is important to note that clinical expertise and patient preferences may override

the other components of the model for a given decision” (Haynes et al., 1996). Similarly,

“without clinical expertise, practice risks becoming tyrannized by evidence, for even

excellent external evidence may be inapplicable to or inappropriate for an individual patient”

(Sackett et al., 1996, p. 72). More specifically,

External clinical evidence can inform, but can never replace, individual clinical

expertise, and it is this expertise that decides whether the external evidence applies to

the individual patient at all and, if so, how it should be integrated into a clinical

decision. Similarly, any external guideline must be integrated with individual clinical

expertise in deciding whether and how it matches the patient’s clinical state,

predicament, and preferences, and thus whether it should be applied (Sackett et al.,

1996, p.72).

Despite these explicit statements, some commentators continue to misappropriate the

terminology of EBP in ways that emphasise research evidence above other elements of the

model, as discussed in section 2.2.

While classical descriptions of EBP offer a model for clinical decision-making, the

role of consumers in Australian policy development has been made more explicit as a result

of the NHMRC’s adoption of GRADE as the basis of their advice to reviewers and guideline

developers. None-the-less, earlier NHMRC advice also acknowledged the role of consumers,

specifically in terms of defining the relevance of research evidence (e.g., NHMRC, 2000,

p.23-27; 2009, p.14). In short, consumer factors cannot merely be seen as “a means of

moving [the EBP] integration process to decision-making” (Schlosser, 2004, para. 6).

Consumers must be consulted from the beginning when planning and synthesising research.

This is elaborated in section 3.3.

Whitehouse et al. (2020) very clearly states that the EBP framework “does not impose

a prescriptive approach on service provision, instead it focuses on the integration of the best

available research evidence with appropriate clinical expertise and the values of the

individual(s) receiving the services” (Whitehouse et al., 2020, p.19). This description is

consistent with the seminal papers that initially described the “three circles” model (Haynes

et al., 1996; Sackett et al. 1996), and is also reflected in GRADE’s advice, (quoted more fully

in section 3.2), that recommendations should never be viewed as “dictates” (Schünemann et

al., 2013, p.29).

Given these assurances, how did the Chair of the Senate Select Committee on Autism

and the lead author of the CRC’s umbrella review come to make their statements about

enforcing the use of certain practices, as discussed in section 1 of this paper? The

background to this kind of prescriptive, top-down approach is discussed in section 2.2.

As with EBP, the ideas behind Evidence-Based Practices (EBPs) date from the early2.2. Evidence-Based Practices 1990s. At that time, the American Psychological Association’s (APA’s) Division 12

(Clinical Psychology) formed a task force on the promotion and dissemination of

psychological procedures. Criteria for identifying empirically validated treatments (later

relabelled empirically supported treatments - ESTs) were published in 1995, sparking “a

decade of both enthusiasm and controversy” (APA Presidential Task Force on Evidence

Based Practice, 2006, p.272). Unlike EBM and EBP, the EST approach focussed exclusively

on research evidence, with the intention of providing a basis for comparison between

psychological and pharmacological treatments.

From the outset, some psychologists expressed concern that the criteria for ESTs were

too narrow to serve as a primary guide for practice, and that the approach may

inappropriately restrict access to care and choice of treatment. There were also questions as

to whether interventions tested under controlled laboratory conditions would generalise well

in “real world” situations, and whether a “treatment manual” approach would exclude many

forms of therapy, tailored approaches, or types of patient from consideration (APA

Presidential Task Force on Evidence-Based Practice, 2006; Levant and Hasan, 2008).

Many of these issues were resolved in 2005, when the APA adopted a model of EBP

based on EBM. Evidence-Based Practice in Psychology (EBPP) took the familiar “three

circles” form, and was defined as “the integration of the best available research with clinical

expertise in the context of patient characteristics, culture, and preferences” (APA Presidential

Task Force on Evidence-Based Practice, 2006, p.273).

The difference between ESTs and EBPP has been described as follows:

EBPP is the more comprehensive concept. ESTs start with the treatment and ask

whether this treatment works for a certain disorder or problem under specified

circumstances. EBPP starts with the patient and asks what research evidence

(including relevant results from randomized clinical trials) will assist the psychologist

to achieve the best outcome. In addition, ESTs are specific psychological treatments

that have been shown to be efficacious in controlled clinical trials, whereas EBPP

encompasses a broader range of clinical activities (e.g., psychological assessment,

case formulation, therapy relationships). As such, EBPP articulates a decision

making process for integrating multiple streams of research evidence, including but

not limited to RCTs, into the intervention process. (APA Presidential Task Force on

Evidence-Based Practice, 2006, p. 273).

Although there is some evidence of continued tension between psychologists who

favour the EST approach and those more committed to EBPP, it should be noted that the list

of ESTs for psychology is now published with a caveat stating that “research evidence for

psychological treatments … will necessarily be combined with clinician expertise and patient

values and characteristics in determining optimum approaches to treatment” (Society of

Clinical Psychology, n.d.).

Past tension between ESTs and EBPP in the field of psychology appears to be echoed

in continuing tension between EBPs and EBP in American special education. The term

“Evidence-Based Practices” has been attributed to Odom, Brantlinger, Gersten, Horner,

Thompson and Harris, 2005 (Test, Kemp-Inman, Deigelmann, Hitt and Bethune, 2015). It is

unclear whether there is any significance in the emergence of this new terminology during the

same year that ESTs in psychology were demoted to “merely one potential operationalization

of the research component of EBP” (Lilienfeld, Ritschel, Lynn, Cautin, & Latzman, 2013, p.

885).

The concepts of EBP and EBPs are commonly conflated by commentators, resulting

in strange hybrids of the two. For example, in a peer reviewed article by the authors of the

largest systematic review included by Whitehouse et al. (2020), it is suggested that “an

important contribution of the evidence-based medicine movement … was that …

identification and verification of evidence-based practice is just the first step” (Hume,

Steinbrenner, Odom, Morin, Nowell, Tomaszewski, Szendrey, McIntyre, Yücesoy‑Özkan &

Savage, 2021, Introductory para. 3). Similarly, Whitehouse et al. (2020) asserts:

It is an essential responsibility of clinical practitioners, educators and other service

providers … to ensure the interventions they offer are underpinned by an evidence

base to help ensure a safe and positive effect on the wellbeing of the individual on the

autism spectrum and their family (Whitehouse et al., 2020, p.18).

These quotes imply, as stated more directly in (for example) Speech Pathology

Australia’s Clinical Guideline on Evidence Based Speech Pathology Practice for Individuals

with Autism Spectrum Disorder, that practitioners “have a responsibility to use only those

interventions that are supported by evidence” (SPA, 2016, p.17). (Note that this sentiment is

not repeated in more recent SPA policy papers. The recently reviewed “Professional

Standards” - SPA, 2020a - presents a definition of EBP that is fully compatible with EBM

and not suggestive of an EBPs approach).

This paper does not attempt to examine assertions that US legislation requires

practitioners to choose interventions from a prescribed list. In Australia, however, there is

certainly no such requirement. The model endorsed by most health and allied health

professional organisations is, quite clearly, based on EBM, and is not prescriptive.

Nonetheless, considerable confusion continues in academic and professional circles, as

evidenced by the mixed messages that appear throughout Whitehouse et al. (2020).

The largest systematic review included by Whitehouse et al., 2020 is Steinbrenner et2.3. Policymaking in the political milieu al. (2020), which is designed to identify EBPs in the U.S. context. A journal article by the

authors of Steinbrenner et al. (2020) acknowledges that

Lack of evidence does not mean that interventions are ineffective, as studies may not

yet have been conducted. However, current human services policies in the U.S.

requires (sic) that intervention practices have research evidence of their effectiveness

(e.g., Individuals with Disabilities Education Act, Medicaid waiver provisions,

insurance coverage regulations) (Hume et al., 2021, Introductory para.2).

The point has already been made that there are no similar requirements under

Australian legislation, and that such requirements do not fit well with the models of EBP

endorsed by many Australian professional bodies. Some Australian academics and

policymakers, however, appear intent on promoting an EBPs approach (e.g., see Paynter,

Luskin-Saxby, Keen, Fordyce, Frost, Imms, Miller, Sutherland, Trembath, Tucker & Ecker,

2020). For this reason, it is important to recognise that the approach has been treated with

much caution by some American academics. For example, in 2002, a report by the American

National Research Council (NRC) contrasted attitudes from “the research community” with

those “crafted in the political milieu” (National Research Council, 2002, p. 21):

Unprecedented federal legislation exalts scientific evidence as the key driver of

education policy and practice, but - here’s the rub - it also inches dangerously toward

a prescription of methods and a rigid definition of research quality. … Educational

researchers, like other researchers, worry that the good intentions underlying the SBR

[Scientifically Based Research] movement will go awry, that narrow definitions of

research or science might trivialize rather than enrich our understanding of education

policy and practice, and that the splendors of unfettered scholarship will be eroded by

creeping tides of conformity and methodological zealotry. (Feuer, Towne, and

Shavelson, 2002, p. 4).

In the same year a journal article written by the authors of the NRC report expressed

concern that “the law appears to instruct [academic scientists] on methodology and to tie

public funding of research to specific modes of inquiry”, and noted that: “for many, the key

question is whether legislators or scientists should ultimately decide issues of research

method” (Feuer et al., 2002, p.5). Revisiting the issue two years later, the authors confirmed

that, in some cases at least, their fears had been well founded:

In the best case scenario, states have integrated the SBR provisions into their

accountability systems by using evidence as a key component of their technical

assistance to schools that need the most help. In the worst case scenario, states are

simply developing lists of ‘evidence-based’ programs that school districts can adopt

with federal funds (Feuer, Towne, and Shavelson, 2004, p.39-40).

Despite such concerns, legislation was enacted and efforts commenced to develop

quality indicators for research and guidelines for identifying EBPs. The Council for

Exceptional Children established a task force for the purpose in January 2003, publishing the

findings in a special issue of the Council’s journal, Exceptional Children, in 2005. The

introductory article sought to provide “a context and rationale for this endeavour” (Odom et

al, 2005, p.138), and was followed by four papers outlining quality indicators for different

research designs: group experimental studies (Gersten, Fuchs, Compton, Coyne, Greenwood,

& Innocenti, 2005); single-subject research (Horner, Carr, Halle, McGee, Odom, & Wolery,

2005); correlational research (Thompson, Diamond, McWilliam, Snyder, & Snyder, 2005);

and qualitative research (Brantlinger, Jimenez, Klingner, Pugach, & Richardson, 2005).

The papers dealing with group designs and single-subject research (Gersten et al.,

2005 & Horner et al., 2005) continue to be very influential, as evidenced by their citation as

appraisal tools used in a number of the systematic reviews included in Whitehouse et al.

(2020, pp. 364-5, 367). More recent appraisal tools draw heavily on these earlier efforts,

although they tend to take the form of checklists despite the observation by Gerten et al.

(2005) that “we intentionally chose to look at quality indicators, rather than a set of “yes/no”

standards, because evaluating a research design always entails weighing the relative strengths

and weaknesses in a set of domains” (p.150).

Importantly, Gersten et al. (2005) and Horner et al. (2005) were the only papers in the

2005 series that went beyond quality indicators to suggest standards for determining EBPs.

This led later commentators to “assume that … these research designs are the only ones to

consider in determining whether a practice in special education is evidence-based … [due to]

the unique ability of these designs to exhibit experimental control” (Cook et al, 2009, p371).

Cook et al. (2009) also observed:

… if the main intent of the reviews is to yield a straightforward decision about

whether a practice is evidence-based, the benefit of additional information gained by

using a more detailed rating system may not be worth the cost of extra time involved

in assessing and reporting the information … (Cook et al, 2009, p. 376).

Perhaps this apparent acceptance of the loss of detail entailed in the EBPs approach

reflected the authors’ expectation that other forms of evidence would also be consulted:

As for concerns that EBPs in special education will force instruction to conform to an

approved menu of interventions, we believe that EBPs will not and should not ever

take the place of professional judgment but can be used to inform and enhance the

decision making of special education teachers. … We in no way imagine evidence

based special educators being directed as to when and in what situations they can or

cannot use particular teaching practices. Instead, EBPs should interface with the

professional wisdom of teachers to maximize the outcomes of students with

disabilities … . We concur, then, with Sackett et al.’s (1996) declaration that,

“clinicians who fear top down cookbooks will find the advocates of evidence based

medicine [or special education] joining them at the barricades” (p. 72) (Cook et al.,

2009, pp. 380-381)

As clearly as this position has been articulated, it is equally clear that many who

support an EBPs approach take it as license to disregard any research that does not “exhibit

experimental control” (Cook et al, 2009, p371); to ignore questions of quality not listed in

appraisal checklists; and to castigate practitioners who - on the basis of their own clinical

experience and their professional understanding of their clients – are not willing to limit their

options to lists of “approved” interventions.

Odom et al. (2005) maintain that “research and development on effective practices in2.4. Checklists, Textbooks and Threats to Validity special education exists on a continuum, with each methodology matched to questions arising

from different points of the continuum” (p.138). The paper identifies four stages in a

program of education research, each requiring different research designs, however it

expresses concern for consumers of research: “Textbooks on educational research describe

the methodology that investigators should follow, but they usually do not provide a succinct

or understandable set of indicators that are useful for individuals who lack graduate training

on research methodology” (p.137).

Gersten et al. (2005) appears to extend this concern from consumers of the research to

the researchers themselves, stating that “few of the numerous educational research textbooks

provide a clear list of criteria that fit the realities of contemporary field research, although

they present excellent ideas” (Gersten et al., 2005, p. 150). The paper suggests that the Study

Design and Implementation Assessment Device (DIAD; Valentine & Cooper, 2003),

published by the “What Works Clearinghouse” (WWC) as an aid to identifying valid and

reliable research for inclusion in systematic reviews, is more helpful in setting standards for

EBPs than textbooks on research methods. The authors of Gersten et al (2005) are more

ambitions than the WWC:

Our goal is a bit broader. We intend these quality indicators to be used not only to

evaluate the merits of a completed research report or article, but also to evaluate

research proposals, dissertation proposals, and grant applications submitted to funding

agencies. We also intend this to be useful to researchers as they think through the

design of a study, to serve as a checklist or organizer of critical issues for

consideration (Gersten et al, 2005, p.150).

Experience has confirmed that such an approach, adopted in response to American

legislation and public funding decisions that appear to favor particular methodologies, has

significant potential to shape the direction of research and professional training. Yet, as

recognised by Odom et al. (2005), good experiments rely very strongly on information gained

from other research designs. In the rush to produce a list of EBPs, important information

bearing on the validity and relevance of the research may be overlooked.

Gersten et al. (2005) supports its argument for the preeminence of experimental

design by citing an influential mid-last century textbook, Campbell and Stanley (1963/66),

which described experiments as

… the only means for settling disputes regarding educational practice, … the only

way of verifying educational improvements, and … the only way of establishing a

cumulative tradition in which improvement can be introduced without the danger of a

faddish discard of old wisdom in favor of inferior novelties (Campbell & Stanley,

1963/66, p.2, quoted by Gersten et al., 2005, p.150).

However, it should be noted that Campbell and Stanley had been updated twice

between 1966 and the publication of Gersten (2005): first in 1979 (Cook & Campbell) and

then in 2002 (Shadish, Cook & Campbell). The 2002 update quoted the same passage from

Campbell and Stanley (1963/66), prefacing it by saying: “As recently as 30 years ago, the

central role of the experiment in science was probably taken more for granted than is the case

today” (Shadish et al., 2002, pp. 29-30). The 1966 quote is given and the text continues:

Not so today. We now understand better that the experiment is a profoundly human

endeavor, affected by all the same human foibles as any other human endeavor,

though with well-developed procedures for partial control of some of the limitations

that have been identified to date (Shadish et al., 2002, p.30.)

The assertion by Gersten et al. (2005, p.150), that textbooks do not address “the

realities of contemporary field research”, does not appear to apply to Shadish et al. (2002)

the only textbook Gersten et al. identifies as “advanced”. Shadish et al. (2002) describes its

theory of generalized causal inference as being “grounded in the actual practice of scientists”

(p. 24). Like its predecessors, the text presents a taxonomy of validity types (internal,

statistical conclusion, construct, and external), together with lists of “threats to validity”, i.e.,

“specific reasons why we can be partly or completely wrong when we make an inference

about covariance, about causation, about constructs, or about whether the causal relationship

holds over variations in persons, settings, treatments, and outcomes” (Shadish et al., 2002,

p.39).

Shadish et al. (2002) notes that no single study, experimental or otherwise, can be

expected to simultaneously deal with all threats to validity, due to the logistical and practical

tradeoffs among the different types of validity. Over the course of a program of research,

however, researchers may “iterate back and forth among them” until, “by the end of a

program of research, each validity type should have had its turn in the spotlight” (pp. 101

102).

The lists of threats to validity have been described as “a widely used framework for

designing and evaluating experimental research in the behavioral and social sciences”

(Petursdottir & Carr, 2018, p. 228). In their scope and detail, they clearly offer a superior

heuristic aid when compared with the hierarchies and quality checklists favoured by

supporters of an EBPs approach.

This section has touched on two descriptions of research programs. The first,

described by Odom et al. (2005), suggests an essentially linear process with successive

research questions answered using appropriately varied study designs. The other, described

by Shadish et al. (2002), responds dynamically as threats to validity are suspected or

revealed. Again, research design varies with the nature of the research question, because

“validity is not a property of methods but of inferences and knowledge claims” (Shadish et

al., 2002, p. 480). These two programmatic approaches differ significantly from “policy

making in the political milieu”, as described in the previous section. Unlike the dynamic

process of examining threats to validity in the course of a program of research, the hierarchies

and quality checklists designed to identify EBPs appear static and less likely to identify

fruitful new directions for research. Most bear only on internal or statistical conclusion

validity, although those that include a requirement for descriptions of the population and

other study particulars may be useful in considering external validity.

Importantly, checklists are silent on the topic of “construct validity” (Shadish et al.,2.5 Construct validity, relevance of evidence, directness of evidence. 2002), a concept similar to the NHMRC’s “relevance of evidence” (NHMRC, 2009) and

GRADE’s “directness of evidence” (Schünemann et al., 2013). Construct validity has been

defined as “the degree to which inferences are warranted from the observed persons, settings,

and cause and effect operations included in a study to the constructs that these instances

might represent” (Shadish et al, 2002, p.38). Without construct validity, it is not possible to

argue that an experiment bears in any way on the real-world questions it has been designed to

answer. This is why one respected commentator has described experimental research as

“trivial, past-tense, and local” (Cronbach, 1982, p.137, quoted in Shadish et al., 2002, p.97).

The Cochrane handbook warns reviewers against reaching “conclusions that go

beyond the evidence” (Schünemann, Vist, Higgins, Santesso, Deeks, Glasziou, Akl, Guyatt,

2021, section 15.6.4), reflecting a concern understood since the middle of last century that “as

soon as experimental results are considered to have a significance that goes beyond the

specific conditions in which they were produced, theoretical considerations become crucial”

(Chalmers, 2013, p192). Put another way, “the principle of the underdetermination of theory

by evidence (Quine, 1951; Stanford, 2009) reminds us that the same set of data is typically

consistent with a broad array of competing theoretical explanations” (Lilienfeld, 2011). This

is why the Cochrane handbook urges caution:

Often this [going beyond the evidence] is done implicitly, without referring to the

additional information or judgements that are used in reaching conclusions about the

implications of a review for practice. Even when additional information and explicit

judgements support conclusions about the implications of a review for practice,

review authors rarely conduct systematic reviews of the additional information.

Furthermore, implications for practice are often dependent on specific circumstances

and values that must be taken into consideration. As we have noted, review authors

should always be cautious when drawing conclusions about implications for practice

and they should not make recommendations (Schünemann et al., 2021, section

15.6.4).

Similarly, Shadish et al. (2002) respects the value of randomly controlled trials and

other rigorous methods, but recognizes that such methods demand a “high level of resources

and a degree of logistical control that is rarely feasible” (p. 23). It suggests that “many

researchers prefer to rely on an implicit set of nonstatistical heuristics for generalization”

(Shadish et al., 2002, p 23) and, like GRADE, it stresses the importance of making these

implicit processes more explicit and systematic. The textbook observes that, for some threats

to construct validity such as inadequate construct explication, “the logic of design control

does not apply”, and that “in these cases, the appropriate method is to identify and explore the

role and influence of the threat in the study” (Shadish et al., 2002, p.40). Despite the

significant shift in attitude to experimental study since the 1960s, this statement appears to

echo one made by Campbell and Stanley: “The more mature focus … avoids crucial

experiments and instead studies dimensional relationships and interactions along many

degrees of the experimental variables (Campbell & Stanley, 1963/66, p. 3).

Shadish et al. (2002) acknowledges that the process of testing for construct validity is

“as much or more social than logical” (Shadish et al., 2002, p.460). Similarly, NHMRC

(2009) refers to “relevance” as “potentially the most subjective of the evidence assessments”

(NHMRC, 2009, p. 14), while GRADE emphasizes the role of judgment in all aspects of

rating the quality of evidence, observing that “the great merit of GRADE is not that it ensures

reproducible judgments but that it requires explicit judgment that is made transparent to

users” (p.12). In short, rigorous application of “scientific method” is inadequate to ensure

that experimental findings have any relevance outside the experiment.

While this acknowledgement of subjectivity is far from Professor Whitehouse’s vision

of research as “a solid foundation that is indisputable” (Commonwealth of Australia, 2021,

p.12), the focus on transparency advocated by Shadish et al. (2002), the Cochrane handbook

(Schünemann et al., 2021) and GRADE (Schünemann et al., 2013) ensures that subjective

judgments are explicit rather than implicit, and thus open to scrutiny by all stakeholders. For

this reason, the role of practitioners and consumers in generating and critiquing research - not

simply in implementing its findings - must be recognized and itself made an explicit part of

the policymaking process.

Shadish et al., does not comment on the suitability of an EBPs approach to education

research, as the book was published before the term was coined. It does, however, refer

negatively to the parallel concept of ESTs in psychology:

Construct mislabelings often have serious implications for either theory or practice.

For example, … the move on the part of some psychotherapy researchers to call a

narrow subset of treatments “empirically supported psychological therapies”

(Chambless & Hollon, 1998; Kendall, 1998) implies to both researchers and funders

that other psychological therapies are not empirically supported, despite several

decades of psychotherapy experiments that confirm their effectiveness. When these

mislabelings occur in a description of an experiment, they may lead the reader to err

in how they apply experimental results to their theory or practice (Shadish, Cook and

Campbell, 2002, p71).

3.COMMENTSWhatONroleTHEcanCRCanUMBRELLA“umbrellaREVIEWOFreview”INTERVENTIONlegitimatelyRESEARCHplay in FORdevelopingASD policyand clinical recommendations? As noted in section 1, the purpose of the umbrella review was to “inform decision-3.1 The “umbrella review” and the future development of a Guideline

making and the future development of a Guideline, not to recommend particular

interventions or develop a Guideline” (Whitehouse et al., 2020, p.20- 21. Emphasis added).

The authors explain:

The development of a Guideline requires the completion of additional steps, including

engagement with a broad group of stakeholders and the application of a system for

rating the quality of evidence and developing recommendations, such as via the

Grading of Recommendations, Assessment, Development and Evaluation (GRADE)

approach (Whitehouse et al., 2020, p.21).

This statement correctly reflects GRADE’s position that systematic reviews “should

typically not include health care recommendations” (Schünemann et al., 2013, p.3) – see

section 3.2. However, it appears to misrepresent GRADE by implying that only guideline

developers (and not reviewers) are required to engage with stakeholders and to “apply a

system for rating the quality of evidence”. These points will be discussed in sections 3.3 and

3.4, below.

Some Australian policymakers clearly envisage a very direct pathway from3.2. From evidence to recommendations identification of EBPs to clinical or public policy recommendations. While input from

practitioners and consumers is acknowledged as important throughout Whitehouse et al.

(2020, e.g., pp. 15, 18, 19, 101, 103), their role appears to commence only after EBPs have

been identified, and to be limited to tailoring EBPs to the individual child, “to optimise the

intervention effect” (Whitehouse et al., 2020, p.102). This division between researchers and

stakeholders is further illustrated by Professor Whitehouse’s comments to the Senate Select

Committee: “the hard bit, so much of the hard bit, which is getting the evidence base, has

been done. The next step is to consult with the clinical and consumer communities to

operationalise the science into recommendations” (Commonwealth of Australia, 2021, p.12).

The process of moving from evidence to recommendations is quite different under

GRADE, which clearly separates the role of systematic reviews from that of clinical

guidelines. Reviews provide estimates of treatment effect and rate the quality of evidence

across all studies for each outcome, with particular emphasis on “patient-important”

outcomes. As described in the Cochrane handbook,

Review authors should not make recommendations about healthcare decisions, but

they can – after describing the certainty of evidence and the balance of benefits and

harms – highlight different actions that might be consistent with particular patterns of

values and preferences and other factors that determine a decision such as cost

(Schünemann et al., 2021, Chapter 15, introductory paragraph, dot-point 5).

It is the role of guideline developers to combine the information from reviews into

recommendations aimed at balancing desirable and undesirable consequences. Again, as

elaborated in the Cochrane handbook:

Drawing conclusions about the practical usefulness of an intervention entails making

trade-offs, either implicitly or explicitly, between the estimated benefits, harms and

the values and preferences. Making such trade-offs, and thus making specific

recommendations for an action in a specific context, goes beyond a Cochrane Review

and requires additional evidence and informed judgements that most Cochrane

reviews do not provide … . Such judgements are typically the domain of clinical

practice guideline developers for which Cochrane Reviews will provide crucial

information (Schünemann et al., 2021, section 15.6.2).

In seeking to achieve this balance, guideline developers are advised to separate

judgments about the quality of evidence from judgments about the strength of

recommendations, “as high confidence in effect estimates does not necessarily imply strong

recommendations, and strong recommendations can result from low or even very low

confidence in effect estimates” (Schünemann et al., 2013, p.4). Both the direction and the

strength of recommendations may be modified on the basis of the values and preferences of

those most likely to be affected by the recommendations. Importantly,

There will be cases in which those making judgments will have legitimate

disagreement about the interpretation of evidence. GRADE provides a framework

guiding through the critical components of the assessment in a structured way. By

allowing to make the judgments explicit rather than implicit it ensures transparency

and a clear basis for discussion (Schünemann et al., 2013, p.4).

This transparent and responsive model of guideline development appears to be in

stark contrast with the image presented by Professor Whitehouse to the Senate Select

Committee on Autism. Having asserted that the umbrella review “provides a solid

foundation that is indisputable”, he continued: “What we … need to do is to operationalise it

through the intervention guidelines and then we need to rigorously enforce it. … Only

evidence based practices would be funded” (Commonwealth of Australia, 2021, p.12).

A similar sentiment is expressed in the NDIS “Consultation Paper: Interventions for

children on the autism spectrum”, although that document links the top-down approach with

an insurance business model: “It is important for the NDIS insurance approach that the

Scheme funds evidence-based interventions …” (NDIS, 2021, p.4.) This appears to relate to

commentary suggesting that judgement based on probabilities derived from controlled

experiments, or even on probabilities derived from actuarial formulas, is superior to clinical

judgement “because at least some grounds for statistical generalization are almost always

superior to none (Lilienfeld et al., 2013, p. 891). However, as discussed in section 2.4 and

2.5, the tools that have been developed to test the quality of experimental studies (and,

perhaps, actuarial formulas) are inadequate to ensure “construct validity” (Shadish et al.,

2002), the “relevance of evidence” (NHMRC, 2009) or GRADE’s “directness of evidence”

(Schünemann et al., 2013). Please see section 2.5 regarding the importance of these

overlapping concepts.

As argued in section 2, a prescriptive, top-down approach based purely on research

evidence is not consistent with either EBP nor GRADE. Even if the quality of evidence (and

the appraisal thereof) had been exemplary – which it has not - GRADE’s position on any

such prescriptive approach is clear:

Clinicians, patients, third-party payers, institutional review committees, other

stakeholders, or the courts should never view recommendations as dictates. Even

strong recommendations based on high-quality evidence will not apply to all

circumstances and all patients. Users of guidelines may reasonably conclude that

following some strong recommendations based on the high quality evidence will be a

mistake for some patients. No clinical practice guideline or recommendation can take

into account all of the often compelling unique features of individual patients and

clinical circumstances. Thus, nobody charged with evaluating clinician’s actions,

should attempt to apply recommendations by rote or in a blanket fashion

(Schünemann et al., 2013, p.29).

Such warnings against top-down, prescriptive approaches ring through the EBP

literature. (Some examples are quoted in section 2.1). Notably, the APA’s decision to adopt

the EBPP model was motivated by reports that funding agencies, courts and policy-makers

were using the EST list to take decision-making out of the hands of practitioners (Levant &

Hasan, 2008, p.658).

Lists of empirically supported treatments have been referenced by a number of local,

state, and federal funding agencies, who are beginning to restrict reimbursement to

these treatments, as are some managed care and insurance companies (Levant and

Hasan, 2008, p.658).

Levant and Hasan (2008) argued that psychologists need to define their decision

making process or

… someone else will dictate what treatments are acceptable and what types of

evidence are privileged. Psychologists will have more leverage with insurers, courts,

and policy makers when APA has a clear statement asserting that psychology is a

science-based profession and preserving the right for psychologists to make the final,

evidence-informed decisions in clinical practice (Levant and Hasan, 2008, p.658).

Organisations that represent Australia’s health and allied health professionals have

similar aspirations for their members. For example, the recently reviewed “Professional

Standards” for Speech Pathology Australia state: “We practise as autonomous professionals,

using independent judgement in accordance with the profession’s knowledge base, Code of

Ethics and our individual scope of practice” (SPA, 2020a, p.11).

Any move by the NDIS to insist on what can and cannot be funded must be made

very explicit and open to public scrutiny and debate. Similarly, the requirement (outlined in

“Consultation Paper: Interventions for Children on the Autism Spectrum”) that people

delivering intervention should “follow established guidance, especially when that guidance

has been published by relevant regulating bodies or is based on research evidence” (NDIS,

2021, pp.14-15) must be questioned. Although the NHMRC recommends GRADE, it cannot

require “regulating bodies” to follow that recommendation. As a result, the quality of any

“established guidance” published by regulating bodies is likely to vary and must be

appraised. (See, for example, for example, see Appendix A). The GRADE handbook notes:

Experts may have opinion about evidence that may be based on interpretation of

studies ranging from uncontrolled case series (e.g. observations in expert’s own

practice) to randomized trials and systematic reviews known to the expert. It is

important to describe what type of evidence (whether published or unpublished) is

being used as the basis for interpretation (Schünemann et al., 2013, p. 13).

Further, even when preparation of a Clinical Guideline has been exemplary, it must be

remembered (as discussed earlier in this section) that recommendations should never be

viewed as dictates, and that “nobody charged with evaluating clinician’s actions, should

attempt to apply recommendations by rote or in a blanket fashion” (Schünemann et al., 2013,

p.29).

With regard to the demarcation between reviews and guidelines, it may be noted that

item 10 of the quality appraisal checklist used by Whitehouse et al. (2020) appears

incompatible with GRADE. As noted, under GRADE systematic reviews should not make

recommendations. Despite this, item 10 asks: “were recommendations for policy and/or

practice supported by the reported data?” (Whitehouse et al., 2020, p.208). This question

may reflect the umbrella review’s reliance on superseded NHMRC advice rather than

GRADE – see section 3.4.

Under GRADE, the research on which clinical and public policy recommendations3.3. Engagement with stakeholders are based begins by considering all “patient-important” outcomes. That is, consumer factors

set the parameters of the research. The comment quoted in section 3.1, that “the development

of a Guideline requires the completion of additional steps, including engagement with a broad

group of stakeholders …” (Whitehouse et al., 2020, p.21), appears to understate the

importance of stakeholder factors in reviews, not just in clinical guidelines.

Although the task of considering the relative importance of patient-important

outcomes falls to guideline developers, their judgment in such matters is “ideally informed by

a systematic review of the literature focusing on what the target population considers as

critical or important outcomes for decision making” (Schünemann et al., 2013, p.8). Thus,

“guideline developers must, and authors of systematic reviews are strongly encouraged to,

specify all potential patient-important outcomes as the first step in their endeavour”

(Schünemann et al., 2013, p.2).

Whitehouse et al. (2020) acknowledges the limitations of the umbrella review and the

included systematic reviews in this regard, finding that “adverse effects were rarely

considered in the systematic reviews” (p.50) and that “the near complete absence of evidence

regarding the effects of intervention practices on quality of life outcomes highlights an urgent

research priority” (p. 15. See also pp. 73-78). The report itself appears to emphasise the role

of stakeholders in research:

Understanding how intervention outcomes may generalise to improvements in child

and family wellbeing and participation, and whether any improvements are sustained

over time, are critical areas of focus for future intervention research. Central to this

recommendation is the involvement of autistic individuals and their families in each

phase of the research process. Grounding research in the lived experience of the

autistic community provides the greatest opportunity for research to generate

outcomes that are meaningful to, and have a positive impact on, the lives of children

on the autism spectrum and their families (Whitehouse et al., 2020, p.103.)

It is very difficult to reconcile the primacy given by GRADE to “patient-important

outcomes”, together with the acknowledgement by Whitehouse et al. (2020) of the central

role “of autistic individuals and their families in each phase of the research process” (p. 103),

with following example from the NDIS consultation paper addressing the umbrella review:

A recent study of weighted blankets reported that a weighted blanket “did not help

children on the autism spectrum sleep for a longer period of time, fall asleep

significantly faster, or wake less often. However, the weighted blanket was favoured

by children and parents, and blankets were well tolerated.

In other words, the researchers did not think that the weighted blankets provided any

significant benefits but the parents did and wanted to keep them. This may be

because the parents saw some changes in the child that the study was not measuring,

such as making the child calmer. Or it could be that the parents thought they saw

some benefits that were not actually there. The choice of the weighted blanket in this

instance becomes an everyday expense not something considered as a reasonable and

necessary NDIS funded support (NDIS, 2021, p.15).

This is clearly not compatible with current NHMRC advice, based on GRADE, which

states:

Guideline developers must base the choice of outcomes on what is important, not on

what outcomes are measured and for which evidence is available. If evidence is

lacking for an important outcome, this should be acknowledged, rather than ignoring

the outcome (Schünemann et al., 2013, p.6).

Given this imperative, it seems the correct approach to the example given would be to

clarify the nature of the outcomes parents and children associate with the intervention, and to

synthesis that - not to summarily rule that an intervention valued by families and consumers

cannot be funded.

As noted in section 3.1, Whitehouse et al. (2020) appears to imply that only clinical3.4. Rating the quality of evidence guidelines, and not systematic reviews, require “the application of a system for rating the

quality of evidence … such as via the GRADE approach” (p.21).

It is important, therefore, to note that GRADE does provide procedures for rating the

quality of evidence in both systematic reviews and clinical guidelines, although the processes

are different for each. Systematic reviews are expected to provide separate quality ratings for

each patient-important outcome, while clinical guidelines provide an overall rating across all

of the critical outcomes that are essential to each of the recommendations they make. “The

lowest quality of evidence for any of the critical outcomes determines the overall quality of

evidence” (Schünemann et al., 2013, p.27).

Reviewers following GRADE guidelines begin by considering study design, in a

manner similar to the way traditional hierarchies may be applied. They then consider five

factors that may reduce the quality rating (risk of bias; indirectness of evidence;

inconsistency; imprecision; and risk of publication bias) and three that may increase the

rating (large magnitude of effect; dose-response gradient; and the influence of residual

plausible confounding). Some of these factors are similar to those found in quality checklists,

however, note that the list also includes “indirectness of evidence”, which is conceptually

similar to the NHMRC dimension three, “relevance of evidence” (NHMRC, 2009, p.14) and

to “construct validity” (Shadish et al., 2002). Please see section 2.5.

Quality rating under GRADE thus encapsulate most aspects of the “dimensions of

evidence” outlined by NHMRC (2009) in one figure, although it is important to note that

GRADE should not be seen as a quantitative system:

While factors influencing the quality of evidence are additive – such that the

reduction or increase in each individual factor is added together with the other factors

to reduce or increase the quality of evidence for an outcome – grading the quality of

evidence involves judgments which are not exclusive. … When the body of evidence

is intermediate with respect to a particular factor, the decision about whether a study

falls above or below the threshold for up- or downgrading the quality (by one or more

factors) depends on judgment (Schünemann et al., 2013, p.12).

As noted in section 3.2, the emphasis under GRADE is on making such judgement

explicit rather than implicit, as a clear basis for discussion (Schünemann et al., 2013, p.4).

As has been mentioned, the NHMRC currently basis its advice to reviewers and

guideline developers on GRADE. It is surprising, therefore, that Whitehouse et al. (2020) has

chosen to use selective, superseded advice from NHMRC (2009) for quality appraisal. The

report defines “high quality empirical evidence” by reference only to the 2009 “evidence

hierarchy” (Whitehouse et al., 2020, p.97), and this is the basis on which it is decided

whether a systematic review meets the quality criteria for inclusion. Some of the issues

addressed in other “dimensions of evidence” (NHMRC, 2009) are dealt with by use of an

appraisal checklist (Whitehouse et al., 2020, pp.60-61). However, systematic reviews “were

not excluded based on methodological quality” (Whitehouse et al. 2020, p.61). By the

review’s own account, treatment effects were predominantly derived from systematic reviews

that provided low to moderate quality evidence according to the quality checklist. Only four

of the included 58 systematic reviews met all of the quality criteria outlined in the appraisal

instrument adapted for the study (Whitehouse et al., 2020, p.12). Despite this, Professor

Whitehouse has described the umbrella review as “the most comprehensive review of

interventions that has ever been conducted” (Commonwealth of Australia, 2021, p.10) and as

providing “a solid foundation that is indisputable” (Commonwealth of Australia, 2021, p.12).

Prejudice held by those in power has enormous potential to influence research and4. Implications for policy and future research practice, particularly if it is presented as being “rigorously scientific”. For example, as noted

in section 1, the Chair of the Senate Select Committee adamantly asserted that “we all know”

sensory and music therapy are not evidence-based. She was apparently unaware that the

review undertaken by the American National Clearinghouse on Autism (Steinbrenner et al.,

had elevated two sensory-based interventions to the status of EBPs in 2020: Music- Mediated Intervention and Ayres Sensory Integration.

The findings on sensory-based interventions reported by Whitehouse et al. (2020)

appear to be more mixed, for example: “Positive intervention effects for sensory-based

interventions were reported for certain intervention practices only, and in those cases,

positive effects were limited to select child and family outcomes” (Whitehouse et al., 2020,

p.12). A detailed comparison of the methodology used in each study would be required to

understand why such variation may exist, and that is beyond the scope of this paper.

However, the fact that the findings are different should remind the reader that science is

fallible. As discussed in sections 2.4 and 2.5, changes in the philosophy of science since the

mid-twentieth century have resulted in “a greater appreciation for the equivocality of all

scientific knowledge” (Shadish et al., 2002, p.29).

Different results reported at different levels of analysis – for example, when

interventions are defined at an individual as opposed to a category level - demonstrate the

limitations of broad-brush approaches to research synthesis. Explanations for inconsistencies

are likely to be found in the details that are left out of a review, as appears to have been

acknowledged by Whitehouse et al. (2020): “The purpose of an umbrella review is to

summarize findings, but that does not negate the importance of reading the systematic

reviews and original studies from which the findings have been derived” (p.100).

It should be noted that the identification of Music-Mediated Intervention and Ayres

Sensory Integration as EBPs lagged considerably behind recognition of their importance by

families and practitioners. Surely there is value in utilising the experience and knowledge

that has been accumulated by practitioners, families and service users to inform research,

rather than using an absence of quality research as an excuse to stop people gaining access to

interventions they have found work for them. No two autistic people have the same

presentation of autism, nor the same constellation of characteristics and circumstances that

may impact on their access to support. How can it make sense to try to standardize responses

to autism?

Whitehouse et al. (2020) acknowledges that “an understanding as to why a particular

intervention may have a positive effect on child development is critical to the goal of

individualised service provision, and will further contextualise the findings presented in the

current review” (p.98-99). Investigation of such issues must, surely, demand involvement

from practitioners and consumers from the outset of a research program. It may be note that

GRADE makes provision for interventions “with thus far insufficient evidence of benefit”

(Schünemann et al., 2013, p.12) to be used in research settings. However, cooperation from

practitioners and consumers will be difficult to achieve if their experience has been

completely disregarded by researchers, who seem intent on blocking their access to supports

from which they have experienced positive results. Insistence by academic researchers that

the experience of practitioners and direct stakeholders is “a separate and ostensibly lesser

category of knowledge“ (Smith, Schmidt, Edelen-Smith, Cook, 2013, p. 149-150, quoted in

Mirenda, 2014/15, p5) brings about a situation in which “practitioners and parents reject

those who reject them“ (Mirenda, 2014/15, p5).

5.     Facilitated Communication     FC was introduced to Queensland by the State Government as a year-long study in5.1.   Reviews of Facilitated Communication research are unsystematic, empty, or both

1992. Informed by the outcome of that study, (i.e., clearly evidence-based), the strategy was retained and continued for twenty years, together with professional supervision, a quality

training program, and careful attention to “best practice”. A generation of individuals with

complex communication needs grew up with access to this well-supported strategy, resulting

in a community with significant practitioner expertise and rich consumer experience.

However, since 2011, the community has faced barriers to its use and, in the case of FC users

themselves, barriers to their own inclusion and personal development.

The change came about due to a series of reviews and clinical guidelines3 based

exclusively on research chosen because its design is “higher” than other available research on

an “evidence hierarchy”. As appears to have been the case in Whitehouse et al., (2020), other

indicators of quality (or lack thereof) were largely ignored, despite the fact that the results of

included studies are consistently contradicted by the results of research based on other

3 The Queensland review was released in 2012, although it has never been clear how it was used in policy making. All State-based public-sector policy related to FC appears to have been made and administered in strict secrecy since that time. Speech Pathology Australia released a 2012 Clinical Guideline (SPA, 2012) critical of FC, updated in 2020 with few substantive changes (SPA, 2020b). The project officer for the SPA Clinical Guideline went on to co-author two reviews of FC as background to “position statements” for the International Society for Augmentative and Alternative Communication (ISAAC, 2014 – see section 5.3) and for the American Speech-Language and Hearing Association (ASHA, 2018a – see section 5.4). Please see Davies (2019) for a critique of the ISAAC and ASHA reviews https://www.academia.edu/38797833/Recent_Reviews_of_Facilitated_Communication_Mislead_Policymaker s and Davies (2020) for a critique of SPA (2012, 2020) https://www.academia.edu/42756096/Speech_Pathology_Australias_flawed_position_on_Facilitated_Commu nication .

designs (the bulk of twenty-first century research on FC), together with the clinical

experience of practitioners and the lived experience of consumers. Attempts by practitioners

and consumers to draw attention to significant flaws in the design and execution of the quasi

experimental studies fell on deaf ears. All feedback had been passed for adjudication to the

original reviewers – the majority of whom were published critics of FC before their

appointment. Unsurprisingly, feedback was ignored on the same grounds that submissions

using similar arguments had been excluded in the first place.

After this decade of devastating loss of support, Whitehouse et al. (2020) has now

demonstrated that recent reviews of FC research, including some that have been very

influential in the formation of public policy, were unsystematic, empty, or both. However, as

discussed in section 5.2, this is misrepresented in the report.

5.2.   The umbrella review misrepresents the findings of the only included systematic      Whitehouse et al. (2020) reported “a null effect on general outcomes” for Facilitatedreview cited in relation to facilitated communication

Communication (p.71). However, that rating is based on only one systematic review, and the

term “null effect” misrepresents its findings.

The included systematic review was published by the National Autism Center (NAC,

2015a). Examination of the list of studies included by NAC (2015b) reveals that none

address FC. Consistent with this, the NAC lists Facilitated Communication among

interventions that “have been identified as falling into the Unestablished level of evidence …

for which no research has been conducted or, if studies have been published, the accepted

process for publishing scientific work was not followed” (NAC, 2015a, p.72).

This finding transmutes to “null effect” in Whitehouse et al. (2020) through the use of

an inappropriate definition. In the umbrella review, “null” is coded

… when all available evidence for an intervention effect for a particular

practice/category on a particular outcome of interest was null, or when rated as

‘unestablished’ (National Autism Center, 2015) or ‘insufficient evidence’

(Steinbrenner et al., 2020) (Whitehouse et al., 2020, p.61).

The definition appears to combine two incompatible concepts. The first deals with

intervention effects, which are rated as either positive, negative, null or inconsistent on the

basis of data extracted and summarised by the included systematic reviews (Whitehouse et al.

2020, p.59). In contrast, the label “unestablished” (NAC, 2015a) or “insufficient evidence”

(Steinbrenner, Hume, Odom, Morin, Nowell, Tomaszewski, Szendrey, McIntyre, Yücesoy-

Özkan, & Savage, 2020) are not based on data from included studies, but refer instead to an

absence of data deemed appropriate for inclusion. With no data, no intervention effect can be

calculated. Note that this is one of the errors the Cochrane Handbook warns against: the

“common mistake” of confusing “‘no evidence of an effect’ with ‘evidence of no effect’”

(Schünemann et al., 2021, section 15.6.4).

Whitehouse et al. (2020, pp. 221-244) provides lists of systematic reviews that were5.3. Two influential reviews of FC are deemed “not systematic” excluded during the full-text screening or data extraction stages of the umbrella review,

together with brief descriptions of the reason for exclusion. These lists reveal that two

reviews of Facilitated Communication, Mostert (2012) and Schlosser, Balandin, Hemsley,

Iacono, Probst, and Von Tetzchner (2014), were excluded on the grounds that they were “not

systematic” (Whitehouse et al., 2020, pp. 219, 221). No more specific details of their

shortcomings are provided.

Exclusion of Mostert (2012) is unsurprising. It had earlier been excluded by

Schlosser et al. (2014), which recognised that it did not closely follow systematic review

guidelines. It was thus treated it as commentary rather than a review.

Exclusion of Schlosser et al. (2014) is of more significance. The review was

undertaken at the request of the International Society for Augmentative and Alternative

Communication (ISAAC), and the methodology was later applied to research undertaken for

the American Speech-Language and Hearing Association (ASHA) (Hemsley, Bryant,

Schlosser, Shane, Lang, Paul, Banajee, and Ireland, 2018). FC practitioners, stakeholders,

and some researchers had submitted peer reviewed studies that used alternative research

designs; feedback on the methodological shortcomings of each review; and information based

on their own experience and circumstances. This was all summarily dismissed by the review

committees for both ISAAC and ASHA4. More robust and systematic reviews of the

literature addressing FC may have resulted if reviewers had been open to a genuine

consultation process.

Although it has now being acknowledged that the methodology applied in these

reviews was “not systematic”, both the ISAAC and ASHA reviews have been widely and

unquestioningly cited in academic and popular media, popularising the perception that FC

has been “debunked”. They have been used to justify anti-FC policy statements from a range

of bodies, including ISAAC (2014); ASHA (2018a); Speech-Language and Audiology

Canada (2018); the American Association on Intellectual and Developmental Disabilities

(2019); and Speech Pathology Australia (2020). The entrenched bias perpetuated by

4 Examples of feedback to ISAAC may be seen at https://isaacaustralia.files.wordpress.com/2014/08/australianchapterfeedeback.pdf and https://sites.google.com/a/annemcdonaldcentre.org.au/isaac-fc—frustrating-communication/ . Examples of feedback provided to ASHA may be seen at https://unitedforcommunicationchoice.org/letters-to-asha/ .

unquestioning acceptance of these reviews has had major repercussions for individuals with

complex communication needs around the world.

The umbrella review’s confirmation that the ISAAC and ASHA reviews are “not

systematic” must surely provide a catalyst for reconsidering any policy that has relied on

these two reviews.

On 8 August, 2018 ASHA released the following statement:5.4. The search process failed to identify ASHA’s 2018 “empty review” of FC Underscoring the importance of scientifically defensible communication interventions

and citing shared concern about harm that could stem from baseless practices, the

Board of Directors of the American Speech-Language-Hearing Association (ASHA)

has unanimously approved position statements that discourage the use of Facilitated

Communication (FC), the Rapid Prompting Method (RPM), and similar practices

such as Spelling to Communicate - techniques where “facilitators” ostensibly elicit

communication from individuals with disabilities (ASHA, 2018b).

At the time of their release, the ASHA position statements cited systematic reviews of

FC and RPM that had been “submitted for publication”. These were both subsequently

published as “empty” reviews: that is, reviews in which “no studies can be located that meet

pre-stated inclusion criteria” (Schlosser, Hemsley, Shane, Todd, Lang, Lilienfeld, Trembath,

Mostert, Fong12 & Odom, 2019. Discission, para.1). Inexplicably, the search process

applied by Whitehouse et al. (2020) captures only one of the ASHA reviews, that addressing

RPM (Schlosser et al., 2019). The ASHA review of FC (Hemsley et al., 2018) is neither

included nor does it appear in the logs of excluded studies.

As discussed in section 5.3, Hemsley et al. (2018) had adopted the 2014 ISAAC

review methodology. On this basis it seems certain that, had the review been identified by

the Whitehouse et al. (2020) search process, it would have been rejected as “not systematic”,

just as Schlosser et al. (2014) has been. Alternatively, as an “empty” review (Hemsley et al.,

2018, p.1), it may have been excluded on the basis that it contained “no relevant (summary

of) outcomes” (Whitehouse et al., 2020, p.63) – that is, the same reason given for rejection of

the ASHA review of RPM (Schlosser, Hemsley, Shane, Todd, Lang, Lilienfeld, Trembath,

Mostert, Fong & Odom, 2019).

Regardless of the grounds, it is apparent that Hemsley et al. (2018) is unlikely to have

met the requirements for inclusion by Whitehouse et al., 2020. As such, it must be concluded

that all recent reviews of FC research have been unsystematic, empty, or both. As

Whitehouse et al. (2020) is essentially an empty review in relation to FC, no findings

regarding FC should have been recorded. It is important to consider the potential impact the

review’s unsupported findings may have on continued bias against Facilitated

Communication, and on future policy decisions.

Whitehouse et al., (2020) states that “the introduction of the NDIS has created sector-6. Conclusions wide change in the choice and control exerted by individuals on the autism spectrum and

their families over their own clinical management, and presented increased opportunities for

the development of policies that are consistently applied across Australia” (p. 103). It does

not take an expert to see the contradiction in this statement. Any imposition of “policies that

are consistently applied across Australia” must imply a reduction of the “choice and control”

available to families and a concentration of power into the hands of academics and

bureaucrats. This was not the vision held for the NDIS, and it is not the intention of

Evidence-Based Practice.

APA7. PresidentialReferencesTask Force on Evidence-Based Practice. (2006). Evidence-Based Practice

in Psychology. American Psychologist 61(4): 271–85.

American Speech-Language-Hearing Association. (2018a). Facilitated communication

[Position Statement]. Retrieved from https://www.asha.org/policy/ps2018-00352/

American Speech-Language-Hearing Association. (2018b). ASHA Discourages Use of

Facilitated Communication, Rapid Prompting Method (web page).

https://www.asha.org/News/2018/ASHA-Discourages-Use-of-Facilitated

Communication-Rapid-Prompting-

Method/?fbclid=IwAR2VkgwO18baKFSw8i9d3bk5CrTIoufifJ298-

u3yOUqKpFhu03JVnLG91Y

American Association on Intellectual and Developmental Disabilities Board of Directors.

(2019). Facilitated Communication and Rapid Prompting Method: Position Statement

of the AAIDD Board of Directors. Retrieved from http://aaidd.org/news

policy/policy/position-statements/facilitated-communication-and-rapid-prompting

method .

Brandinger, E., Jimenez, R., Klingner, J., Pugach, M., & Richardson, V. (2005). Qualitative

studies in special education. Exceptional Children, 71, 195-207.

https://doi.org/10.1177/001440290507100205

Campbell, D. T., & Stanley, J. C. (1963/66). Experimental and quasiexperimental designs for research. Boston, MA: Houghton Mifflin. (Reprinted from Handbook of Research on Teaching, by N. L. Gage (Ed.). 1963. Chicago: Rand McNally.) Retrieved from

https://www.sfu.ca/~palys/Campbell&Stanley-1959

Exptl& QuasiExptlDesignsForResearch.pdf

Chalmers, A. (2013). What is this thing called science? (4th Ed.). Brisbane, Australia:

University of Queensland Press.

Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal

of Consulting and Clinical Psychology, 66, 7-18.

Commonwealth of Australia. (2021). Senate Select Committee on Autism: Proof Committee

Hansard. Wednesday, 3 March, 2021, pp. 1-14. Downloaded from

https://www.aph.gov.au/Parliamentary Business/Committees/Senate/Autism/autism/P

ublic Hearings

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues

for field settings. Skokie, IL: Rand McNally.

Cook, B.G., Tankersley, M., & Landrum, T.J. (2009). Determining evidence-based practices in special education. Exceptional Children, 75, 365-383.

https://doi.org/10.1177%2F001440290907500306

Cronbach, L.J. Designing Evaluations of Educational and Social Programs. San Francisco:

Jossey-Bass, 1982.

Davies, C. (2019). Recent Reviews of Facilitated Communication Mislead Policymakers.

https://www.academia.edu/38797833/Recent_Reviews_of_Facilitated_Communicatio

n Mislead Policymakers

Davies, C. (2020). Speech Pathology Australia’s flawed position on Facilitated

Communication.

https://www.academia.edu/42756096/Speech Pathology Australias flawed position

_on_Facilitated_Communication

Feuer, M.J., Towne, L., & Shavelson, R.J. (2002). Scientific culture and educational research.

Educational Researcher, 31(8), 4-14. https://doi.org/10.3102/0013189X031008004

Feuer, M.J., Towne, L. & Shavelson, R.J. (2004). Scientific Culture and Educational

Research. Zeitschrift für Erziehungswissenschaft, 7. Jahrg., Beiheft4/2004, S. xxx.

Downloaded from

https://web.stanford.edu/dept/SUSE/SEAL/Reports Papers/FeuerTowneShavelson.pdf

Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. (2005).

Quality indicators for group experimental and quasi-experimental research in special

education. Exceptional Children, 71, 149-164.

http://dx.doi.org/10.1177/001440290507100202

Haynes, R.B., Sackett, D.L., Gray, J.M.A., Cook, D.F., & Guyatt, G.H. (1996) Transferring

Evidence from Research into Practice: The Role of Clinical Care Research Evidence

in Clinical Decisions. ACP Journal Club, November/December, A14–A16.

Hemsley, B., Bryant, L., Schlosser, R.W., Shane, H.C., Lang, R., Paul, D., Banajee, M., &

Ireland, M. (2018). Systematic review of facilitated communication 2014–2018 finds

no new evidence that messages delivered using facilitated communication are

authored by the person with disability. Autism & Developmental Language

Impairments, 3, 1-8. https://doi.org/10.1177%2F2396941518821570

Horner, R., Carr, E., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of

single-subject design research to identify evidence-based practice in special

education. Exceptional Children, 71, 165-179.

https://doi.org/10.1177/001440290507100203

Hume, K., Steinbrenner, J.R., Odom, S.L., Morin, K.L., Nowell, S.W., Tomaszewski, B.,

Szendrey, S., McIntyre, N.S., Yücesoy‑Özkan, S. & Savage, M.N. (2021).

Evidence‑Based Practices for Children, Youth, and Young Adults with Autism: Third

Generation Review. Journal of Autism and Developmental Disorders.

https://doi.org/10.1007/s10803-020-04844-2 . Epub ahead of print.

International Society for Augmentative and Alternative Communication [ISAAC]. (2014).

ISAAC Position Statement on Facilitated Communication. Augmentative and

Alternative Communication, 30, 357-358. DOI: 10.3109/07434618.2014.971492

Kendall, P. C. (1998). Empirically supported psychological therapies. Journal of Consulting

and Clinical Psychology, 66, 3-6.

Levant, R.F. & Hasan, N.T. (2008). Evidence Based Practice in Psychology. Professional

Psychology, Research and Practice, 39(6), 658-662.

https://psycnet.apa.org/doi/10.1037/0735-7028.39.6.658

Lilienfeld, S. O. (2011). Distinguishing scientific from pseudoscientific psychotherapies:

Evaluating the role of theoretical plausibility, with a little help from Reverend Bayes.

Clinical Psychology: Science and Practice, 18, 105–111.

http://dx.doi.org/10.1111/j.1468-2850.2011.01241.x.

Lilienfeld, S.O., Ritschel, L.A., Lynn, S.J., Cautin, R.L., and Latzman, R.D. (2013). Why

many clinical psychologists are resistant to evidence-based practice: Root causes and

constructive remedies. Clinical Psychology Review, 33, 883-900.

https://doi.org/10.1016/j.cpr.2012.09.008

Lilienfeld, S.O., Marshall, J., Todd, J.T. & Shane, H.C. (2014) The persistence of fad

interventions in the face of negative scientific evidence: Facilitated communication

for autism as a case example, Evidence-Based Communication Assessment and

Intervention, 8:2, 62-101. DOI: 10.1080/17489539.2014.976332

Mirenda, P. (2014/2015). [Published online 13.1.15]. Comments and a personal reflection on

the persistence of facilitated communication. Evidence Based Communication

Assessment and Intervention, 8(2), 102–110. doi:10.1080/17489539.2014.997427

Mostert, M. P. (2012). Facilitated communication: The empirical imperative to prevent

further professional malpractice. Evidence-Based Communication Assessment and

Intervention, 6(1), 18-27. doi: 10.1080/17489539.2012.693840

National Autism Center. (2015a). Findings and conclusions: National standards project,

phase 2. (Web page). https://www.nationalautismcenter.org/national-standards

project/phase-2/

National Autism Center. (2015b). Articles Reviewed in Phase 2. (Web page)

https://www.nationalautismcenter.org/national-standards-project/phase-2/articles

reviewed-in-phase-2/

National Disability Insurance Scheme. (2021). Consultation paper: Interventions for children

on the autism spectrum. March 2021, Version 1.0. Downloaded from

https://www.ndis.gov.au/community/have-your-say/interventions-children-autism

spectrum

National Health and Medical Research Council. (n.d.). Resources for guideline developers

(web page). https://www.nhmrc.gov.au/research-policy/guideline

development/resources-guideline-developers

National Health and Medical Research Council. (1999). A guide to the development,

implementation and evaluation of clinical practice guidelines.

http://www.nhmrc.gov.au/ files nhmrc/publications/attachments/cp30.pdf

National Health and Medical Research Council. (2000). How to use the evidence: assessment

and application of scientific evidence.

http://www.nhmrc.gov.au/ files nhmrc/publications/attachments/cp69.pdf

National Health and Medical Research Council. (2009). NHMRC levels of evidence and

grades for recommendations for developers of guidelines.

https://www.nhmrc.gov.au/sites/default/files/images/NHMRC%20Levels%20and%20

Grades%20(2009).pdf

National Research Council. 2002. Scientific Research in Education. Washington, DC: The

National Academies Press. https://doi.org/10.17226/10236.

Occupational Therapy Australia. (2018). Evidence-Based Practice Position Statement.

Retrieved from https://otaus.com.au/publicassets/90977488-f433-e911-a2c2

b75c2fd918c5/ebppositionstatement.pdf

Odom, S.L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. (2005).

Research in special education: Scientific methods and evidence-based practices.

Exceptional Children 71(2), 137–148.

https://doi.org/10.1177%2F001440290507100201

Paynter, J., Luskin-Saxby, S., Keen, D., Fordyce, K., Frost, G., Imms, C., Miller, S.,

Sutherland, R., Trembath, D., Tucker, M. & Ecker, U. (2020). Brief report: Perceived

evidence and use of autism intervention strategies in early intervention providers.

Journal of Autism and Developmental Disorders, 50(10), 1088–1094. https

://doi.org/10.1007/s1080 3-019-04332 -2 .

Petursdottir, A.I., & Carr, J.E. (2018) Applying the Taxonomy of Validity Threats from

Mainstream Research Design to Single-Case Experiments in Applied Behaviour

Analysis. Behavior Analysis in Practice, 11(3), 228–240.

https://doi.org/10.1007/s40617-018-00294-6

Quine, W. V. O. (1951). Main trends in recent philosophy: Two dogmas of empiricism. The

Philosophical Review, 60, 20–43.

Sackett, D., Rosenberg, W., Gray, J., Haynes, R., & Richardson, W. (1996). Evidence-Based

Medicine: What It Is and What It Isn’t. British Medical Journal, 312:71–72.

https://doi.org/10.1136/bmj.312.7023.71

Satterfield, J.M., Spring, B., Brownson, R.C., Mullen, E.J., Newhouse, R.P., Walker, B.B., &

Whitlock, E.P. (2009). Towards a Transdisciplinary Model of Evidence-Based

Practice. The Milbank Quarterly, 87(2), 368-390. https://doi.org/10.1111/j.1468

0009.2009.00561.x

Schlosser, R.W. (2004). Evidence-Based Practice in AAC. The ASHA Leader, 9(12). (In print

publication, pp.6-10. Page numbers not available in on-line publication).

Downloaded from https://doi.org/10.1044/leader.FTR3.09122004.6

Schlosser, R.W., Balandin, S., Hemsley, B., Iacono, T., Probst, P., & von Tetzchner, S.

(2014a). Facilitated communication and authorship: a systematic review.

Augmentative and Alternative Communication, 30(4), 359-368. doi:

10.3109/07434618.2014.971490

Schlosser, R.W., Hemsley, B., Shane, H., Todd, J., Lang, R., Lilienfeld, S.O, Trembath, D.,

Mostert, M., Fong, S., & Odom, S. (2019). Rapid prompting method and autism

spectrum disorder: Systematic review exposes lack of evidence. Review Journal of

Autism and Developmental Disorders, 6, 403-412. https://doi.org/10.1007/s40489

019-00175-w

Schünemann, H., Brożek, J., Guyatt, G., Oxman, A. (Eds). (2013). Handbook for grading the

quality of evidence and the strength of recommendations using the GRADE approach.

Retrieved from

https://gdt.gradepro.org/app/handbook/handbook.html#h.svwngs6pm0f2

Schünemann H.J., Vist G.E., Higgins J.P.T., Santesso N., Deeks J.J., Glasziou P., Akl E.A.,

Guyatt G.H. (2021) Chapter 15: Interpreting results and drawing conclusions. In:

Higgins J.P.T., Thomas J., Chandler J., Cumpston M., Li T., Page M.J., Welch V.A.

(editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2

(updated February 2021). Cochrane, 2021. Available from

https://training.cochrane.org/handbook/current/chapter-15 .

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi

experimental designs for generalized causal inference. Belmont, CA: Wadsworth

Cengage Learning.

Smith GJ, Schmidt MM, Edelen-Smith PJ, Cook BG. (2013). Pasteur’s Quadrant as the

Bridge Linking Rigor with Relevance. Exceptional Children. 2013;79(3):147-161.

doi:10.1177/001440291307900202

Society of Clinical Psychology, Division 12, American Psychological Association. (n.d.).

Psychological Treatments: About This Resource (web page).

https://div12.org/psychological-treatments/

Speech-Language and Audiology Canada. (2018, 16 January). Official statement from

Speech-Language & Audiology Canada. Use of Facilitated Communication and

Rapid Prompting Method. Retrieved from www.sac-

oac.ca/sites/default/files/resources/sac_official_statement_on_facilitated_communicat

ion_and_rapid_prompting_method_jan2018_en.pdf

Speech Pathology Australia. (2016). Clinical Guideline: Evidence Based Speech Pathology

Practice for Individuals with Autism Spectrum Disorder. Melbourne: Speech

Pathology Australia.

Speech Pathology Australia. (2020a). Professional Standards for Speech Pathologists in

Australia. Retrieved from

https://www.speechpathologyaustralia.org.au/SPAweb/Resources For Speech Pathol

ogists/Professional_Standards/Professional_Standards/SPAweb/Resources_for_Speec

h_Pathologists/CBOS/Professional_Standards.aspx?hkey=583a56af-74e0-4111-95fa

656502269967

Speech Pathology Australia. (2020b). Augmentative and Alternative Communication Clinical

Guideline. Melbourne: Speech Pathology Australia.

Stanford, K. (2009). Underdetermination of theory. Stanford Encyclopedia of Philosophy.

http://plato.stanford.edu/entries/scientific-underdetermination/

Steinbrenner, J.R., Hume, K., Odom, S.L., Morin, K.L., Nowell, S.W., Tomaszewski, B.,

Szendrey, S., McIntyre, N.S., Yücesoy-Özkan, S., & Savage, M.N. (2020). Evidence

based practices for children, youth, and young adults with Autism. The University of

North Carolina at Chapel Hill, Frank Porter Graham Child Development Institute,

National Clearinghouse on Autism Evidence and Practice Review Team.

Test, D. W., Kemp-Inman, A., Diegelmann, K., Hitt, S. B., & Bethune, L. (2015). Are online

sources for identifying evidence-based practices trustworthy? An evaluation.

Exceptional Children, 82, 58-80. doi:10.1177/0014402915585477

Thompson, B., Diamond, K., McWilliam, R., Snyder, P., & Snyder, S. (2005). Evaluating the

quality of evidence from correlational research for evidence-based practice.

Exceptional Children, 71, 181-194. http://dx.doi.org/10.1177/001440290507100204

Valentine, J. C., & Cooper, H. (2003). What Works Clearinghouse Study Design and

Implementation Assessment Device (Version 1.0). Washington, DC: U.S. Department

of Education.

Whitehouse, A., Varcin, K., Waddington, H., Sulek, R., Bent, C., Ashburner, J., Eapen, V.,

Goodall, E., Hudry, K., Roberts, J., Silove, N., Trembath, D. (2020). Interventions for

children on the autism spectrum: A synthesis of research evidence. Autism CRC,

Brisbane, 2020X. Accessed via https://www.autismcrc.com.au/interventions

evidence/category-overview/developmental

Appendix A: Letter from Professor Gordon Guyatt, co-editor of GRADE, re recommendation on Facilitated Communication in

Speech Pathology Australia’s Clinical Guideline on Augmentative

and Alternative Communication (SPA, 2020)

McMASTER UNIVERSITY

Faculty of Health Sciences

Gordon H. Guyatt, MD, FRCP(C), OC

Professor, Departments of

Health Research Methods, Evidence, and Impact (HED, and Medicine

1280 Main Street West, HSC-2C12, Hamilton, ON L85 4K1

RE: Submission reference mamber SUB .100.00133 August 18, 2020

Royal Commission

into Violence, Abuse, Neglect and Exploitation of People with Disability

GPO Box 1422, Brisbane Old 4001

Dear Royal Commission:

Ihave been requested by Cathie Davies to evaluate the Speech Pathology Australia guidelines addressing the augmentative and altemative communication and particularly facilitated communication (hereafter the SPA guideline) and to comment on her critique of these guidelines. This note represents my findings.

Regarding my appropnateness to make this assessment, I am a Distinguished Professor of Medicine at McMaster University in Hamilton, Ontario, Canada I have over 35 years of experience as a clinical investigator and] have published over 1,000 research papers that have been cited over 145,000 times. I helped found the GRADE working group and am a current co Char.

There are two widely used instruments for assessing clinical practice guidelines. One is the Apprasal of Guidelines, Research and Evaluation (AGREE II) instrument and the other is the

National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS)

instrument. The SPA guideline fails key criteria of both instruments.

Specifically, the SPA guideline: ¢ The guideline didnot include a methodological expert ¢ The guideline panel did not deal effectively with financial and particularly non-financial Gntellectual) conflict of interests, including panelists with strong prior expressed opinions

¢ The guideline panel did not include clinician or expert members representing the diversity of the interested stakeholders

¢ The guideline panel did not include caregivers representing the diversity ofinterested individuals

¢ The guideline was dismissive inits consideration of the perspectives of parents and caretakers and patent advocates

¢ The guideline panel did not have available, and did not undertake, a methodologically rigorous systematic review of the relevant literature (including relevant eligibility criteria; a comprehensive search, which in this case wouldinclude both quantitative and qualitative studies; and ratings of nsk of bias in individual studies

¢ The guideline did notidentify all the outcomes important to patients and caregivers and include these in their systematic review

¢ The guideline did not provide an evidence summary that included ratings of the quality/certanty of the evidence for each relevant outcome using acredible system such as

GRADE

¢ The guideline did not make clear the values and preferences underlying their recommendations

Moreover, a number of Ms. Davies’ concems, including selective use of the literature with omission of key evidence, confusing commentanes with primary studies and with systematic reviews, and apparent deliberate exclusion of interested parties from both the professional and patient communities are all valid. Moreover, the guideline at times reads like anideologically driven polemic rather than arational scientific document.

I would be happy to discuss my review of the guideline and Ms. Davies commentary with any interested parties