Independent Assessments: Assessor Qualifications and Quality Assurance

JOINT STANDING COMMITTEE ON THE NATIONAL DISABILITY INSURANCE SCHEME

Independent Assessments

Submission from:

Scope (Aust) Ltd

INTRODUCTION

About Scope

Scope (Aust) Ltd (Scope) is a leading provider of services to people with disability in Victoria, and one of the largest not-for-profit organisations in Australia. Our origins stretch back to 1948, when a group of parents who wanted better lives and opportunities for their children with disability established the Spastic Children’s Society of Victoria.

Scope’s mission is to enable each person we support to live as an empowered and equal citizen.

Scope provides services including Supported Independent Living, Short Term Accommodation, Therapy and Lifestyle options to more than 7,000 people and their families across metropolitan and regional Victoria. Scope also works with corporate and community organisations to improve inclusiveness for people with disability.

Scope welcomes the opportunity to respond to the consultation and has responded to six of the Terms of Reference.

The independence, qualifications, training, expertise and

quality assurance of assessors

The assessors are from organisations that have been contracted by the National Disability Insurance Agency to conduct the independent assessments. There is a need for transparency and systems to manage the conflict of interest that could potentially arise because of the tension between conducting an objective and unbiased assessment and the Agency’s budgetary considerations. It is imperative that processes are implemented to prevent situations where decisions are largely driven by the Agency’s financial bottom line.

Of all the assessment tools in the Toolkit, use of The Vineland Adaptive Behavior Scales and the Pediatric Evaluation of Disability – Computer Adaptive Testing (PEDI-CAT) is restricted to people with certain qualifications (allied health or special education professional). Use of the other tools requires either no training or very little training. Hence, assessors will need to be allied health professionals in order to administer some of the tools, which is in accordance with the Agency’s position.

The Agency has stated that all assessors will be trained experts, for example, occupational therapists, physiotherapists, psychologists and other health and disability professionals. While there are many such professionals who might be considered experts and good practitioners, they may have never worked with people with disability. It is strongly recommended that the assessors have substantial and demonstrated experience working directly with people with disability and their families/carers, as well as formal qualifications. They will need to understand disability (and the sector), including specific disabilities, their impact and trajectory, have a positive attitude towards disability, and respect and know how to work with carers/ others who may attend the assessment.

Understanding and having experience with specific types of disability is especially critical in ensuring that the assessment and outcomes are accurate. For example, failing to understand that many people with intellectual disability acquiesce will result in a flawed assessment. Acquiescence is often observed in people with intellectual disability because of a desirability to please others, the perceived importance of the person asking the questions, or because of the complexity of the questions/s. Similarly, being unaware that some people with Acquired Brain Injury may have poor insight into their disability will also have a negative impact on the outcomes of their assessment; one’s insight into their disability will have an impact on how they respond to the questions.

For applicants with complex communication needs, skilled communication support partners may be required for participants to engage in independent assessments. The assessors will need to be confident in working alongside these communication partners and appreciate that they do not speak for the applicant; rather, the role of the communication partner is to facilitate the involvement of the applicant in the process. The independent assessors will also need some expertise in communication strategies and an understanding of how to use various communication aids.

Minimum standards for assessors should be developed, published and monitored.

The appropriateness of the assessment tools selected for

use in independent assessments to determine plan funding

The selection of the various independent assessment tools requires further interrogation prior to full implementation. Contrary to the conclusions drawn by the Agency, some of the tools do not “demonstrate strong evidence of reliability and validity” 1 to support their use with people with disability. Although some of the tools have some research available about the psychometrics, it is not for the cohorts with which they will be used. The psychometric properties of tools are context dependent and, to draw the conclusion that they are psychometrically sound, requires testing with the population with which they are used. This is not the case for at least three of the tools selected by the Agency (see below).

The Craig Hospital Inventory of Environmental Factors (CHIEF) is a tool that has been selected to measure the environmental barriers encountered by people with disability. There are two versions, however, it is unclear which of the two the Agency intends to use. There is a 25-item version that is administered as an interview to the person with disability and a shorter 12-item survey that is completed by the person. There is little evidence that it has been used or tested with people with autism, intellectual disability, or psychosocial disability, which are the three largest groups accessing the NDIS.2

During the early development of the CHIEF, an expert panel was convened that included consumers with hearing impairments, visual impairments, spinal cord injury, speech impairments, and cerebral palsy, as well as family members of people with intellectual disability and traumatic brain

1 National Disability Insurance Scheme (2020). Independent Assessment. Selection of Assessment Tools. https://www.ndis.gov.au/participants/independent-assessments/independent-assessment-pilot (page 4). 2 National Disability Insurance Scheme (n.d). Explore participant data. https://data.ndis.gov.au/explore-data.

injury.3 For the initial testing of the psychometric properties, 409 people with disability were recruited, which included 124 people with spinal cord injury, 120 participants with traumatic brain injury, 55 people with multiple sclerosis, 35 amputees, and 75 “others”. The “others” included people with auditory and visual impairments, developmental disabilities, cerebral, or multiple impairments resulting in disability. It is evident that intellectual disability, autism, and psychosocial disability were not represented sufficiently in the research and the evidence of reliability and validity presented by the developers are not for these groups.

Since its development, the CHIEF has predominately been used to determine environmental barriers experienced by other groups of people with disability, such as amputees4 and post-stroke patients. 5 There is some research that explores the psychometric properties, however, it is largely in the context of developing and testing the CHIEF in other languages. For example, it has been used with caregivers of children and adolescents with cerebral palsy to develop the Brazilian6 and the Persian translation, 7 and with elderly people with stroke for the Korean translation.8 For the most part, this research has not included people who have the types of disabilities that would be supported by the NDIS. It can therefore be concluded that the CHIEF does not have a strong evidence base for use with the people who are likely to access the NDIS (e.g., intellectual disability, autism, and psychosocial disability).

Similar conclusions can be drawn about the Lower Extremity Functional Scale (LEFS), the tool selected by the Agency to measure functional impairment across conditions that affect the lower extremity. The authors of a systematic review published in 2016 concluded that the LEFS was reliable and valid 9, however, it is evident that none of the 27 studies included in the review included participants that had disabilities beyond what can arise from a medical/ health condition. For example, most of the participants were patients recruited from rehabilitation wards, physical therapy or pain clinics, who had specific musculoskeletal conditions such as osteoarthritis, lower musculoskeletal injuries, ankle fractures, knee conditions, or were recovering from surgery. Although they had a disability, they are not representative of the people who would access the NDIS.

3 Craig Hospital (2001). Craig Hospital Inventory of Environmental Factors. https://craighospital.org/uploads/CraigHospital.ChiefManual.pdf 4 Ephraim, P.L., MacKenzie, E.J., Wegener, S.T., Dillingham, T.R., & Pezzin, L.E. (2006). Environmental barriers experienced by amputees: The Craig Hospital Inventory of Environmental Factors–Short Form. Archives of Physical Medicine and Rehabilitation, 87(3): 328-333. https://doi: 10.1016/j.apmr.2005.11.010. 5 Gunilla Carlsson, Björn Slaug & Eva Månsson Lexell (2020). Assessing environmental barriers by means of the Swedish Craig Hospital Inventory of Environmental Factors among people post-stroke. Scandinavian Journal of Occupational Therapy, June 16. https://doi: 10.1080/11038128.2020.1775885. 6 Furtado, S.R.C., Sampaio, R.F., Vaz, D.V., Pinho, B.A.S., Nascimento, I.O., & Mancini, M.C. (2014). Brazilian version of the instrument of environmental assessment Craig Hospital Inventory of Environmental Factors (CHIEF): Translation, cross-cultural adaptation and reliability. Brazilian Journal of Physical Therapy, 18(3), 259-267. https://doi: 10.1590/bjpt-rbf.2014.0036. 7 Nobakht, Z., Rassafiani, M., & Rezasoltani, P. (2011). Validity and reliability of Persian version of Craig Hospital Inventory of Environmental Factors (CHIEF) in children with cerebral palsy. Iranian Rehabilitation Journal, 9 (1):3-10. 8 Han, C-W., Yajima, Y., Lee, E-J., Nakajima, K., Meguro, M., & Kohzuki, M. (2005). Validity and utility of the Craig Hospital Inventory of Environmental Factors for Korean community-dwelling elderly with or without stroke. The Tohoku Journal of Experimental Medicine, 206 (1): 41-49. https://doi: 10.1620/tjem.206.41 9 Mehta, S.P., Fulton, A., Quach, C., Thistle, M., Toledo, C., & Evans, N.A. (2016). Measurement properties of the Lower Extremity Function Scale: A systematic review. Journal of Orthopaedic & Sports Physical Therapy, 46 (3): 200-216. https://doi: 10.2519/jospt.2016.6165.

Therefore, the conclusions drawn about the reliability and validity of the LEFS are for a different group of people. Hence, as with the CHIEF, there is a lack of evidence to support the use of the LEFS for applicants/ participants of the NDIS.

The Young Children’s Participation and Environment Measure (YC – PEM) has been selected to measure participation in home, day care/ preschool and community activities in children under 6 years of age. To our knowledge, there are only four published papers that report on the psychometric properties of this tool, two focussing on the original English version 10 & 11, a third on the Swedish adaptation12, and a fourth paper on the Singaporean translation. 13 Four published studies are insufficient evidence to support the use of the YC – PEM and make any authoritative conclusions about its validity and reliability. More research is required before such claims can be made.

There are other issues with the tools in addition to those relating to a lack of research and evidence supporting their use. Some of the tools will exclude many people with disability from the process because they rely on proxy response or are not accessible. For example, all the tools selected for adolescents have been designed to be completed by a proxy and do not allow for self-report. Whilst it is appropriate for carers/others to provide information on behalf of children and very young people with disability, the same cannot be said for adolescents. Of the four tools that are intended to be used with adults, three are self-report, however, none are available in Plain or Easy English. Not being available in Plain or Easy English means that the tools would be inaccessible to many people with intellectual disability, one of the largest groups of adult participants in the NDIS. Ignoring the voice of people with disability and not seeking their views contradicts the principles outlined in the United Nations Convention on the Rights of Persons with Disabilities14 and contemporary practice. The tools would also be inaccessible to other groups who could benefit from more simplified language, such as those from culturally and linguistically diverse backgrounds.

Even if the tools were available in more accessible formats, some of the concepts would remain inaccessible for many people with intellectual disability. For example, one of the key features of intellectual disability is having difficulty with abstract concepts such as time. Yet the WHODAS 2.0 asks for responses in relation to the last 30 days; for example, “In the past 30 days, not counting the days that you were totally unable, for how many days did you cut back or reduce your usual activities or work because of any health condition”. In addition to difficulties with the 30-day reference points, many people with intellectual disability would also have difficulty responding in

10 Khetani, M.A., Graham, J.E., Davies, P.L., Law, M.C., & Simeonsson, R.J. (2015). Psychometric properties of the Young Children’s Participation and Environment Measure (YC-PEM). Archives of Physical Medicine & Rehabilitation, 96(2): 307-316. https://doi: 10.1016/j.apmr.2014.09.031 11 Khetani, M.A. (2015). Validation of environmental content in the Young Children’s Participation and Environment Measure. Archives of Physical Medicine & Rehabilitation, 96(2), 317-322. https://doi.org/10.1016/j.apmr.2014.11.016. 12 Astrom, F.M., Khetani, M., & Axelsson, A.K. (2017). Young Children’s Participation and Environment Measure: Swedish cultural adaptation. Physical & Occupational Therapy in Pediatrics, 38(3): 329-342. https://doi:10.1080/01942638.2017.1318430. 13 Lim, CY., Law, M., Khetani, M., Rosenbaum, P., & Pollock, N. (2018). Psychometric evaluation of the Young Children’s Participation and Environment Measure (YC-PEM) for use in Singapore. Physical & Occupational Therapist in Pediatrics, 38 (3): 316-328. https://doi: 10.1080/01942638.2017.1347911. 14 United Nations. (2006). The Convention on the Rights of Persons with Disabilities. http://www.un.org/disabilities/convention/conventionfull.shtml.

relation to the number of days. Similarly, the CHIEF asks people to think about the past 12 months and then indicate the degree of difficulty from daily, weekly, monthly, less than monthly; and the LEFS asks people indicate the degree of difficulty they have with standing or sitting for one hour. These are all abstract concepts that many people with intellectual disability would find challenging and will likely result in them being excluded from self-reporting. It may be that the Agency intends for the self-report tools (i.e., the WHODAS 2.0, CHIEF, and LEFS) to be completed by or be administered to a proxy, however, it should be noted that both the CHIEF and LEFS have not been designed to be completed in this way. Indeed, the developers of the CHIEF caution against proxy completion because of the poor inter-rater reliability.15 To our knowledge, there is no research that has explored the psychometric properties of the LEFS when proxy response is used. More research is required if it is the Agency’s intention to have proxies complete tools that have been designed for self-report.

Despite the Agency’s assertions that the assessment tools consider different settings and times, they do not do this sufficiently well and there is a risk that an applicant’s needs will be inaccurately captured. For example, the LEFS asks applicants to respond in relation to today (i.e., “Today, do you or would you have any difficulty at all with…”). It is therefore possible that the outcomes of this assessment reflect only a good day rather than the applicant’s overall needs. Similarly, the 30-day reference point specified in the WHODAS 2.0 may not be sufficient for some disabilities that tend to be episodic or where functioning fluctuates (for example, psychosocial disability), or is dependent on what is happening in the environment (for example, autism).

The wording of some of the questions from the WHODAS 2.0 is inappropriate and may result in distress to applicants, their families and supporters. Two questions in particular stand out: D6.6 “How much has your health been a drain on the financial resources of you or your family?” and D 6.7 “How much of a problem did your family have because of your health problems?” These questions are supposed to encourage applicants to consider how other people and the world around them make it difficult for them to engage in life, however, may be misconstrued by the person with disability to mean that they are a burden to those around them.

The circumstances in which a person may not be required

to complete an independent assessment

A person should not be required to complete an independent assessment if they are unable to respond accurately to the questions on their own (e.g., someone with a severe or profound intellectual disability) and they do not have someone who knows them well enough to provide reliable information about them (e.g., a family member, health professional who has worked with the person, a service provider who knows the person well). Any person who provides information for the independent assessments must have an established relationship with the applicant and demonstrate that they have a good understanding of the applicant’s needs.

15 Craig Hospital (2001). Craig Hospital Inventory of Environmental Factors. https://craighospital.org/uploads/CraigHospital.ChiefManual.pdf

Opportunities to review or challenge the outcomes of

independent assessments

Independent assessments will be used to determine eligibility and to develop a NDIS plan. It is evident that there will not be an opportunity for those involved in the assessment (e.g., the applicant, family member/s, supporters, health professionals) to review the assessment findings before the plan is finalised. Further, according to the Agency, whilst the access decision remains reviewable, the results of the independent assessment are not (refer to page 23 of the Agency’s consultation paper). 16 This is in contrast with the Tune review, which recommended that “participants having the right to challenge the results of the functional capacity assessment, including the ability to undertake a second assessment or seek some form of arbitration if, for whatever reason, they are unsatisfied with the assessment”. 17

Not allowing the results of the assessment to be reviewed before being finalised is a missed opportunity to confirm its accuracy and could disadvantage applicants. It means that the work of one person, the assessor, over only a 20 min – 3 hour session/s, determines the applicant’s eligibility and needs, which in turn informs the participant’s plan. Given that the independent assessment will be used to make decisions about eligibility and the participants’ plans (i.e., a “high stakes” assessment), applicants (and their supports) must be given the opportunity to dispute the results if they believe them to be inaccurate.

According to the Agency, independent assessments cannot be reviewed because they will be “sound and robust”. 18 Because of the potential for human error and the very nature or assessment tools, it is not possible to guarantee that every assessment will be sound and robust, even with the quality assurance framework that the Agency will develop. Further, despite the Agency’s assertions that the assessment tools consider different settings and times and so will provide a picture of what good and bad days look like, they do not do this sufficiently well. The LEFS, for example, asks applicants to respond in relation to today (i.e., “Today, do you or would you have any difficulty at all with…”). It is therefore possible that the outcomes of this assessment reflect only a good day rather than the applicant’s overall needs. Similarly, the WHODAS 2.0 asks people to respond in relation to the last 30 days. This timeframe may not be sufficient for some disabilities that tend to be episodic or where functioning fluctuates (for example, psychosocial disability), or depends on what is happening in the environment (for example, autism).

In addition to the issues outlined above, it should be noted that no tool, including those chosen by the Agency, is perfect and there will always be the potential for the outcomes to vary depending on

16 National Disability Insurance Scheme (2020). Consultation paper: Access and eligibility policy with independent assessments. https://www.ndis.gov.au/community/have-your-say/access-and-eligibility-policy-independent-assessments. 17 Tune, D. (2019). Review of the National Disability Insurance Scheme Act 2013. Removing red tap and implementing the NDIS participant service guarantee. Department of Social Services: Canberra. (page 66). 18 National Disability Insurance Scheme (2020). Consultation paper: Access and eligibility policy with independent assessments. https://www.ndis.gov.au/community/have-your-say/access-and-eligibility-policy-independent-assessments. (page 23).

a range of factors (e.g., what is happening for the participant on the day, the experience of the assessor, human error). The psychometric properties of the tools provide evidence of how outcomes of assessment tools can vary. For example, some of the test-retest reliability statistics that have been reported for the WHOSDAS 2.0 do not meet the thresholds for what is considered adequate. 19 Low test-retest reliability means that different results can be obtained even when the same assessor repeats the assessment on the same person only days apart. There is, therefore, the potential for different interpretations to be drawn about a person, even though the same assessor conducted the assessment. Similarly, some of the inter-rater reliability statistics reported for the CHIEF are lower than what is considered acceptable 20, which means two assessors could rate the same applicant differently. For these reasons, challenging the results of the independent assessments should be allowed.

The appropriateness of independent assessments for

particular cohorts of people with disability, including Aboriginal and Torres Strait Islander peoples, people from regional, rural and remote areas, and people from culturally and linguistically diverse backgrounds

How the tools will be used with people from culturally and linguistically diverse backgrounds has not been addressed nor has whether the measures are culturally sensitive.

Although some of the tools are available in other languages, there is a lack of research that explores the psychometric properties of the translated versions or, where there is research, it is based on single studies utilising only small samples. For example, there is only one published study on the Italian translation of the LEFS utilising only 250 participants. 21 Whilst the results of this research (and research with cohorts using other languages) is promising, it is not sufficient to draw authoritative conclusions about the use of the translated tools. The research must be replicated on a much larger scale before such conclusions can be drawn.

It should also be noted that although translations may be available, they are not necessarily in the languages commonly used in Australia. For example, the PEDI-CAT is available in US Spanish, Brazilian Portuguese, Danish, Dutch, French-Canadian, German, Italian, Norwegian, and Swedish22,

19 Üstün, T.B., Kostanjsek, N., Chatterji, S., & Rehm, J. (2010). Measuring health and disability: Manual for WHO Disability Assessment Schedule (WHODAS 2.0). World Health Organisation: Geneva. 20 Craig Hospital (2001). Craig Hospital Inventory of Environmental Factors. https://craighospital.org/uploads/CraigHospital.ChiefManual.pdf 21 Cacchio, A., De Blasis, E., Necozione, S., Rosa, F., Riddle, D.L., di Orio, F., De Blasis, D., & Santilli, V. (2009). The Italian version of the Lower Extremity Functional Scale was reliable, valid, and responsive. Journal of Clinical Epidemiology, 63(5): 550-557. https://doi: 10.1016/j.jclinepi.2009.08.001. 22 Pearson. (n.d.). Pediatric Evaluation of Disability Inventory Computer Adaptive Test. https://www.pearsonassessments.com/store/usassessments/en/Store/Professional-Assessments/Behavior/Pediatric-Evaluation-of-Disability-Inventory-Computer-Adaptive-Test/p/100002037.html

however, the top five languages in Australia are Mandarin, Arabic, Cantonese, Vietnamese and Italian23.

A key issue with independent assessments is that they will be conducted within a short period of time (up to 3 hours on average depending on the person’s disability and age) by someone who does not know the applicant. This creates a significant risk that the applicant’s needs/ functional capacity will not be adequately captured. The outcomes for individuals with complex disability are likely to be even more seriously affected. Complex disability can include any combination of disability and physical health/ medical conditions, mental health problems, challenging behaviour, alcohol or drug issues24-25, contact with the criminal justice system, Aboriginal or Torres Strait Islander background, use of augmentative and alternative communication, a history of being placed in (or leaving) state care (i.e., child protection services)26, homelessness or being at risk of homelessness. Any of these factors may influence the way that an applicant presents during the independent assessment, including: trust issues owing to previous negative experiences with authority figures; inability to participate fully owing to use of augmentative and alternative communication devices that are unknown to the assessor or rely on the involvement of a familiar communication partner; or assessor uncertainty about how the interaction of multiple disabilities impacts on the applicant’s functional abilities. The assessor will need to allow more time, especially for people who use augmentative and alternative communication as communication rates 15-25 times slower are common when these forms of communication are used rather than speech.27 There needs to be careful consideration of how an applicant’s individual circumstances, notably any complex disability, will be addressed within the time being allotted for them to complete an independent assessment.

Conclusions

The NDIS (Becoming a Participant) Rules 2016 28 state that independent assessments can be used, however, the tools that have been selected are not fit for purpose and lack evidence to support their use with NDIS applicants and participants. The Productivity Commission29 had also recommended an

23 Australian Bureau of Statistics (2016). Cultural Diversity in Australia, 2016. https://www.abs.gov.au/ausstats/abs@.nsf/Lookup/by%20Subject/2071.0~~2016~~Main%20Features~~Cultural%20Diversity%20Article~~60 24 St Vincent’s Hospital. (2013). Young Adult Complex Disability Service [Brochure]. https://www.svhm.org.au/ArticleDocuments/2073/YACDS-Brochure.pdf.aspx?embed=y 25 Disability Justice Project. (2017). Support planning for people with complex needs. http://www.disabilityjustice.edu.au/supporting-people-with-complex-needs. 26 Department of Family and Community Services. (2014). Leading clinical practice and supporting individuals with complex support needs in an NDIS environment. New South Wales, Australia: Author. 27 Beukelman, D.R., & Mirenda, P. (2013). Augmentative and alternative communication: Supporting children and adults with complex communication needs. 4th ed. Baltimore, MD: Paul H. Brookes Publishing Co. 28 Australian Government Federal Register of Legislation (2018). National Disability Insurance Scheme (Becoming a Participant) Rules 2016. https://www.legislation.gov.au/Details/F2018C00165 29 Productivity Commission (2011). Disability Care and Support. Report no. 54. Canberra: Author.

assessment process, however, there was much stronger emphasis on it being person-centred and collaborative (keeping the person with disability front and centre), which is not the case with the Agency’s approach. There was also a strong focus on the tools being rigorous and valid, and to be the best available tools, which is also not the case with the tools selected by the Agency.

Although pilots of the independent assessments have been reported to result in higher quality and more consistent decisions and more equitable plan outcomes for participant, the reports have not been made public nor subjected to peer review. These pilots were conducted by the Agency but should have been conducted independent of the Agency by qualified researchers in order to manage bias. More extensive, independent pilots are required, as well as broader consultation with stakeholders.

The introduction of independent assessments has caused considerable stress and angst in participants and others. It is evident that further consideration and more extensive consultation is required before any changes are implemented.

Scope’s recommendations

Conduct more extensive consultations with participants and stakeholders about the use of independent assessments.
Conduct independent pilots on the use of independent assessments and the specific tools selected.
Establish the prerequisite for assessors to have substantial and demonstrated experience working directly with people with disability and their families/carers, as well as formal qualifications.
Further interrogate the proposed independent assessment tools prior to full implementation, particularly the CHIEF, LEFS and YC – PEM for which additional research about the psychometrics is required specifically with the groups that are likely to access the NDIS.
Amend the policy to specify which groups are exempt from an independent assessment. Exemptions should be given to those who are not able to self-report and do not have someone who can provide reliable information about them.
Amend the policy to institute an appeals mechanism for decisions not to grant exemptions from independent assessments.
Provide the opportunity for applicants, family members, and their supporting health and disability professionals to review the results of independent assessments prior to finalisation.
Amend the policy to institute an appeals mechanism in relation to the results of the independent assessments.
Further interrogate the cultural appropriateness and sensitivity of the tools prior to full implementation.

Allow sufficient time for assessors to understand applicants, their disabilities, and the impact their disabilities have on their lives.
Make provision during independent assessments for the deeper involvement of people who are known to applicants, in particular for people with complex disability.