JOINT DYNAMICS
Joint Standing Committee on the National Disability Insurance Scheme
PO Box 6100
Parliament House Canberra ACT 2600
Dear Committee Members,
Introduction
Joint Dynamics have looked at ways that Government operated Disability Support Systems could be improved to provide greater accuracy and fairness to the benefit of the community as a whole and most importantly to the individual claimant.
We recommend the use of accurate objective measurement which would in part include use of the unique Joint Function Assessment System (JFAS) – recently winning a Australian Design Award against international medical companies.
The purpose of this document is to highlight the present methods and look at ways to gain a higher quality result by replacing subjective interpretations with objective unbiased measured data to the evidence base when assessing people for possible benefits.
In crude terms, the following methods of assessment of physical disabilities could be used:
-
Ask the claimant what their capabilities are.
-
Observe the claimant carrying out a number of capability tasks.
-
Have an assessor provide an opinion based on subjective interpretations provided by the claimant.
-
Measure various common physical activities with a system that is independently verified for accuracy.
The above is also true for physical disability resulting from neurological issues.
From the above options, it is necessary to define what level of accuracy and quality the assessment needs to be. Does it matter much if the assessment is wrong by 25% over or under? If a multiple answer survey question has 4 possible answers and one is selected the error could be ±1 answer unit or in other terms ± 25%. The impact of that size of error (25%) could greatly impact the treatment; fairness and the compensation the claimant should rightfully receive.
Assessment decisions would be able to be made more confidently, if factual data was available to the medical professional. So-called “Evidence based” decisions can range from high quality objective data down to poor quality subjective interpretations. The quality of the foundation evidence is rarely and in most cases not defined in the assessment process.
If the aim is to achieve the highest quality assessment, then there is no approach other than approach 4 - provided it is cost effective.
While assessments are based on “grey” subjective information it is easy for quite large errors in treatment and processing to occur it is difficult for auditors to do their job in identifying factual information it is easy for the claimant to distort the factual status of their disability it is easy for a rehabilitation person to over-service a claimant because there are no measurements or progress curves. Under-servicing would normally not be an option. it is impossible to make any comparisons with population normative data to know where on the degree of ability scale the claimant sits. No one can confidently say the person’s objective performance is 5% better or worse than the last time they were tested.
It is only by accurately quantifying the status of a person’s capabilities that one can identify optimal treatment and planning strategies and effective management of the particular case. Planning built on a possibly erroneous and un-auditable foundation will definitely lead to poor quality outcomes.
Defining the Degree of Disability According to many definitions, a disability is a result of an impairment that may be cognitive, developmental, intellectual, mental, physical, sensory, or some combination of these.
The lack of ability is the nett result of when the person’s abilities are compared with person without the disability. It is impossible to directly measure an ability that the person does not possess. The only logical way to quantify their disability is to quantify their ability and compare that value with results expected from a person without the disability.
In mathematical terms “Disability” is the “Complement” of “Ability”. In general mathematicians use the word “complement” to mean “all except” or “the whole minus a given part”. The addition of “ability” plus “disability” equals one. An example of this readily shown in determining a hearing disability - a hearing ability needs to be carried out first.
In the case of physical disabilities, accurate and objective measurements of various parameters can be made and compared to population norms.
Put simply – you cannot accurately define or measure the degree of disability until (where possible) the degree of ability is accurately measured.
Most Common Presently Used Methods to Quantify Disability
Presently, the most common method of acquiring information about a person’s physical disability is with the use of multiple-choice questionnaires. These have the following limitations: Extremely difficult to manage and audit effectively. they are not quantitative assessments ie measured they are subjective and are an interpretation provided by the subject or the assessor.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 2 of 29
measurement accuracies cannot be calculated nor quoted eg Standard Deviations. they are easily subject to bias intentional or unintentional. they have a foundation of subjective self-reported information used directly or indirectly by the assessor. they highly rely on the subject’s recall abilities. they can have high inter- and intra- assessor and test-retest errors. most of the subject’s provided answers can never be proved with facts.
Please refer to the numerous presently used NDIS subjective methods and documents provided in the appendix at the end of this document.
Importance of Collected information The importance of the collected objective and survey data is critical to the cost effective, efficient and accurate planning of a disability case.
“When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts advanced to the stage of science.”
Extract Reference https://en.wikipedia.org/wiki/William_Thomson,_1st_Baron_Kelvin
The above quotation is from the 1880’s by William Thomson, 1st Barron Kelvin - a British mathematical physicist and engineer.
This quotation was abbreviated by Peter Drucker to: “If you can’t measure it – you can’t manage it.”
It must be understood that there will be cases where the subjective survey information will have reduced value and some will result in incorrect planning and management.
Subjective survey information can only be truly addressed when cross-referenced to measure, accurate and objective information.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 3 of 29
Importance of Accurate information The importance of the collecting accurate objective and survey data is very important to prove a solid foundation for important decisions to be based upon. If there is not the solid foundation based on maximum objective data many participants in the rehabilitation arena will tend to over-service than risk under-servicing the insurance claimant. This is non fraudulent over-servicing, as they have not been provided with true facts about the disability and do want to best service their claimant.
Over-servicing is difficult to identify and reduce if foundation information for opinions and decisions is based on subjective opinions. Lack of factual information only adds to inefficiencies and costs in the planning and rehabilitation process of a claimant.
High Quality Planning can only be possible if the collated evidence upon which the planning is based is of the highest possible quality. The old adage “garbage in equals garbage out” is very apt in producing a reliable factual outcome.
How is the Value of Collected Information Assessed? Terms like “validity”, “reliability”, “margin of error” and “confidence levels” are terms used to quantify the value of the collected information.
The Value Gradient of Collected Information (Evidence)
The importance of the collected is directly important to the factual content of the collected data. Very simply, at one end of the scale you have objective quantitative data providing the most valuable useable information. At the other end of the scale is subjective opinions provided by people who do not have the training experience or knowledge to gather the information. This would have very little value at all.
In the real world, the level of objective information should be maximised and the level of subjective should be minimised (where possible) to ensure the highest quality and value of the collected evidence.
To gather information regarding pain levels the assessor has to resort to survey questions. This very important parameter and presently there is no accurate objective method available to acquire this interpreted pain level data.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 4 of 29
NDIS Assessment Methods Presently used
Interpreted Resultant Physical
Severity Indicator Scales
Diagnosis Scaling Capability
Used
Or Measurement Impairment
Acquired brain Care and Need Scale Interpretation Yes
injury (CANS)
PEDI-CAT (16 and under) Interpretation
WHODAS 2.0 (17+) Interpretation
Autism Diagnostic and statistical Interpretation Yes
manual of Mental Disorders, Fifth edition (DSM-5)
Vineland Adaptive Interpretation
Behaviour Scale (vineland-
II)
PEDI-CAT (16 and under) Interpretation
World Health Organisation Interpretation
Disability Assessment
Schedule (WHODAS) 2.0 (17+)
Cerebral palsy Gross Motor Functional Interpretation Yes
Hearing Classification Scale
impairment (GMFCS)
Hearing Impairment Interpretation
Responses and Groupings
Guide
PEDI-CAT (16 and under) Interpretation
Other scales: Hearing loss (in decibels)
Intellectual Diagnostic and statistical Interpretation Yes
disability manual of Mental Disorders, Fifth edition (DSM-5)
Developmental Vineland Adaptive Interpretation Yes
delay Behaviour Scale (vineland II)
Global Interpretation Yes
developmental PEDI-CAT (16 and under) delay
Down WHODAS 2.0 (17+) Interpretation Yes
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 5 of 29
Interpreted Resultant Physical
Severity Indicator Scales
Diagnosis Scaling Capability
Used
Or Measurement Impairment
syndrome
Multiple Disease Steps Yes
sclerosis
Patient Determined Disease Interpretation
Steps (PDDS)
Other scales: Expanded
Disability Status Scale
(EDSS)
Psychosocial Health of the Nation Interpretation Yes
disability Outcome Survey (HoNOS)
Life Skills profile – 16 item Interpretation
(LSP-16)
PEDI-CAT (16 and under) Interpretation
WHODAS 2.0 (17+) Interpretation
Spinal cord Level of lesion Yes
injury
American Spinal Injury Interpretation
Association Impairment
Scale (ASIA)
PEDI-CAT (16 and under) Interpretation
WHODAS 2.0 (17+) Interpretation
Stroke Modified Rankin Scale Interpretation Yes
(mRS)
Vision Vision Impairment Interpretation Yes
impairment Questionnaire
PEDI-CAT (16 and under) Interpretation
Other scales: Visual acuity level and visual field loss
Other WHODAS 2.0 (17+) Interpretation Yes
In all of the above listed processes no measurement is made of the impact of the disability on any of the person’s living functions only subjective interpretation information is collected.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 6 of 29
Planning starts with understanding the factual abilities that a person possesses. During a comprehensive objective Functional Living Capabilities Assessment, the following key parameters could be expected to be accurately measured:
Range of movement & maximal strength of specific Joints (using unique Australian technology which has won an Australian Design Award) Lifting and carrying Pushing and pulling How long one can stand or walk Flexibility and reaching Grasping and holding capabilities Balance capabilities Dexterity Motor function & response times Walking and running Crawling bending and stooping Stair climbing Sit/Stand function Driving a Motor Vehicle
Measurement data collected by the above tests would include: Task times Power produced Work done Fatigue index Capabilities progress graphs over time Comparison with meaningful population normative performance data
Non-physical objective tests that may be carried out are:
Hearing and vision tests Blood tests X-Ray
ECG, EKG
CT
MRI
echocardiography
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 7 of 29
The true issue to be addressed by the government and disability agencies is simply – does the community deserve to be provided with (1) highly variable and arguable subjective assessment interpretations OR (2) should assessments be based on factual, unbiased, accurate, objective and measured data about a claimant’s true capabilities.
Taken from Government Committee Enquiry page
https://www.aph.gov.au/Parliamentary_Business/Committees/Joint/National_Disability_Ins urance_Scheme/NDISPlanning
NDIS Planning (Extract from Web page)
Aspartofthecommittee’sroletoinquireintotheimplementation,performanceand
governanceoftheNationalDisabilityInsuranceScheme(NDIS),thecommitteewillinquire intoandreportonNDISPlanning,withparticularreferenceto:
a. the experience, expertise and qualifications of planners; How can the planner plan without accurate and objective base data directly relating to the factual abilities of a claimant?
The more objective unbiased data the measurement of the capabilities of the claimant is provided to the highly qualified assessment personnel, the more they could focus on appropriate and effective planning.
b. the ability of planners to understand and address complex needs; Complex needs can only be accurately defined after the factual physical capabilities of the claimant are accurately quantified and known. The planner will be assisted in their decision as to whether support is reasonable and necessary by quantifying the impairment of the physical disability.
c. the ongoing training and professional development of planners; No comment.
d. the overall number of planners relative to the demand for plans; No comment.
e. participant involvement in planning processes and the efficacy of introducing draft plans;
Participant involvement in care plans involving exercise or other rehabilitation would be encouraged by the objective nature of measured progress and outcomes at the onset as well as feedback to the participant during the plan. The task of planning becomes extremely difficult when the foundation data and information cannot be proven to be correct and accurate.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 8 of 29
f. the incidence, severity and impact of plan gaps; No comment.
g. the reassessment process, including the incidence and impact of funding changes;
Reassessment of any individual participant’s plan should, where possible, be based on objective, accurate measurement of outcomes. In some instances, re-measurement may be required. After
Assessment or reassessment can only be meaningful if unbiased, objective measured data is maximised and subjective opinions are minimised. For instance in the case of a participant with cerebral palsy and restriction of joint extension ROM, progress, the effectiveness of physio could be measured accurately and compared
h. the review process and means to streamline it; This can only be achieved by consistently applying assessment tools to identify a participant’s progress and can only be effective if measured parameters are accurate and sensitive enough to detect changes in the relative ability of the claimant. The only way to streamline the process is to have accurate and objective data to define positive changes for the claimant.
i. the incidence of appeals to the AAT and possible measures to reduce the number;
Appeals can readily be reduced by using confirmed accurate objective ability measurements instead of subjective interpretations. Effectiveness of a plan could then be decided fairly, minimising dissention in regard to any possible bias in the reassessment.
j. the circumstances in which plans could be automatically rolled-over; Intermediate ability assessments and comparing that objective, unbiased and accurate data could guide this decision in a fair and efficient manner.
k. the circumstances in which longer plans could be introduced; Longer plans could be appropriately recommended where disability is objectively measured to be unchanged, together with expert opinion as to the prognosis associated with the underlying diagnosis impairment or diagnosis.
l. the adequacy of the planning process for rural and regional participants; and any other related matters.
There are many aspects relating to the benefits of the introduction of objective assessments in quantifying claimant abilities and in regard to many conditions or types of impairment. Accurate Planning can only result from accurate data and information.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 9 of 29
Conclusion
Efficient and accurate Planning can only occur if it is based on factually correct measured, assessment objective data and subjective information of the magnitude of the true impact of the disability, whether it be physical or motor skills related.
Joint Dynamics would like to discuss with key NDIS people ways to utilise more accountable and accurate physical function assessments to provide the basis of more accurate planning and servicing of NDIS claimants.
This approach can only result in fairer outcomes.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 10 of 29
Appendix
Presently Used Methods
Instruments Related to Physical Ability Quantification
ACQUIRED BRAIN INJURY CARE and Needs Scale (CANS)
CEREBRAL PALSY Gross Motor Functional Classification Scale (GMFCS)
SPINAL CORD INJURY American Spinal Injury Association Impairment Scale (ASIA)
DOWN SYNDROME PEDI-CAT (16 and under)
SPINAL CORD INJURY WHODAS 36 Item Assessment
STROKE Modified Rankin Scale
Gross Motor Function Classification System – used in Canada
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 11 of 29
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 12 of 29
Patient Name____________ Date/frne of Exam _________ INTERNATIONAL STANDARDS FOR NEUROLOGICAL
CLASSIFICATION OF SPINAL CORD INJURY ISC S ExamilerName ___________ Signature ____________ {ISNCSCI)
SENSORY SENSORY MOTOR KEY SENSORY POINTS KEY SENSORY POINTS MOTOR K£Y MUSCLES KEY MUSCLES RIGHT Liltd Tcu:11 (\.~ ~ P1ick (PPR) Lii,d: Toucb (\.1l) Pk! Prix (PPL) LEFT
C2 +—–+ C2 C3 C3 C4 C4
EltJowflexors C5 +-----+ CS Elbow nexors
Wrist extensors Wrist extensors CG C6 UEL UER § § (Upper Extremtty Left) Right) 8/Jow extensors extensors (Upper Extremity C7 C7 Ef/Jow
Finger flexors ea CS Rnger nexors
Finger abductors Q/1!1e ""JI!'! T1 T1 Rnger a/Jductors (Jttlt, lil!IJer/
T2 +-----+ T2 MOTOR T3 .,___-+ T3 (SCORINGON REVERSE SIDE)
T4 T4 0 • tvta/ pa,a,fSis
+-----+ TS 1 = pq,able or VISible wntrac1Jon TS
TG TG 23 == d>llacli>!l lllOWJffleJltmo,ement agailstg,arily eliminatedgmity
T7 T7 54 == a...d>e mow:ment,rntl""""'11, •(Pimlagainst some1111 resistanceresistance
poin/disuSI! TB +-----+ TB s· = oomJ/J/ comcll!d '°' NT=notlestlble T9 T9
t-----+ T10 SENSORY T10 T11 (SCORING ON REVERSE SIOE) T11
O T12 I T12 S2 +-----+ = absent 2NT=- normalnot testable I 1=a11en:d L1 · L1
Hip flexors L2 / , I
LEL LER Knee extensors L3 § ~ :;::sas (lower Extremity Right) Ankle <Jorsfftexors L4 +-----+ L4 Ankle <Jorsiflexors (Lower Extremity left)
Long toe extensors L5 +-----+ LS Long roe extensors Ankle plantar t/exors S1 +-----+ § S1 Ankle plantar flexors
S2 S2
S3 S3 (VAC) Voluntary Anal Co11tractton r--, S4-S S4-5 r---1 (OAP) Deep Alla/ Pressure
(Yes/No) L…..J L……J (Yes/No)
LEFT TOTALS RIGHT TOTALS D D D (50) (MAXIMUM) (50) (56) D(56) D(56) (56) (MAXIMUM)
MOTOR SUBSCORES SENSORY SUBSCORES
pp TOTAL ~ UER D +UEL D = UEMS TOTAL Cl LER CJ+LEL ~ = LEMS TOTAL Cl LTRLJ + LTL D =LT TOTAL Cl PPR ~ + PPL D =
MAX (25) (25) (50) MAX (25) (25) (50) MAX (56) (56) (112) MAX (56) (56) (112)
NEUROLOGICAL R L 3. NEUROLOGICAL r-1 4. COMPLETE OR INCOMPLETE? .-, ,., _,,,.,. ~ °"'11 R L LEVELS 1.SENSORYLJD LEVEL OF INJURY L--..J tlcomp/l!t,,= AIIT$l!rl$0Q'Ofrno/OTlundioninSf-5 L.......J ZONE OF PARTIAL SENSORYD D S/J!ps 1·5 for classification MOTORLJ DEnhancingAssessments&PlanningInTheNDIS190906.docas on n:-,oroe 2.MOTORc:Jc::J (NL/) 5. ASIA IMPAIRMENT SCALEPage(AIS) ~13 of 29-!!~'!!AI!E!-
This form may be copied freely but should not be altered without permissiofl from the America11 Spiflal Injury Association. 11811/15
ASIA Impairment Scale (AIS) Steps in Classification
TIil kltloWilg am Is rCIDTlll!nlOO fa de’.ermmg Ille c:lffisfftalion of hlvrulls vAih sa
A = Com lete. No S&r:S(J1 oc rrctor ·uncton is 8S!lrvoo .n
lhe sacr31 1 . Determine sensory levels for right and lett sides. 7119 sersay kM/ Is /11e rmst <aXti, /rltn lmmlrmB tu txxf1 ,.i µti< ;m ligti ll1ld1 SfmJ!lm
- Determine motor levels tor right and lett sides. Def1l8(J t,; the /oNest key mw:ie ’1mm /In/ tm a g;m ofat km! 3 (oo SVfJr1J tesllng), {m’tlnlJ 1119 liB’f ms:Je m:uoos r(}(XestmXJ t!f segnents .txMJ tlla/ level are j4}e<1 to /Je intact (~ as a 5) Note in regms 1me me is IV f11}{}tcxne to /list, /11e roolr:f kMt Is {Xes.JfTlfX1 to /Je the same as l1e sensay kMi, if /estalie mota ’1mm aoove /f1at kMi 1s aro rama.
Sensory Grading
0 A.':ffint 3. Determine the neurological level of Injury (NU)
T7rs refoo to /fie most ca/XJ/i segment ot /fie (X)((/ 1’,i/11 illact sensation m ;mp.efty (3 a mae) ll1U!lCfe m:tm str~. fXOWXXl /11at t!KJre Is oormal {iltact) smxxyam mota fUocbl rosJrat-; resped!vefy Too /’lJ is /fie most a,prolad at tne samy aoo mo&:Jr 1e1e1s detoonmJ ii When to Test Non-Key Muscles: steps 1 and 2.
In a patient with an apparent AIS B ciassiflcatloo, non-key muscle functions 4. Determine whether the Injury Is complete or Incomplete. more than 3 levels below the motor level on each side should be teste<1 to
most accurately classify the Injury (differentiate between AlS B and C). (Te. a/Jsence ()( pr/J!BUJ ofS8C,fij spaiing) If voor,ay anal coo/rad/on = No AND alf S4-5 SBrfHY !llJ(f!S = 0
Movement Root level AND deep analfX9SS.l(e = No, /lien in!ff1/ is Complete.
Shoulder: Rlldon, exterron, cbli:tm, ai:tt£1m. 1rma1 CS othefv,/se, Jnµy Is Incomplete.
m exurel rotalioo E = Normal. t srrsatcn a1d rrctor ·unctcn iE e;le::: l'ltl}
Elbow: s~m:m lhe ISl'CSG ar:: gwoo iE -crm r all ~ne1ts ;nt the 5. Determine ASIA Impairment Scale (AIS) Grade:
Elbow: Pnm1m C6 pater! md _prcr defclts. ltEl the AIS _g_rade Is E Sora:ne Is Injury complete? If YES, AIS=A .m can recad
a1 A1S_gra6: ZPP (b'l8St derrratcroo or myoboo Wrist: FleXDl wt1out 311 rntu ::Cl docs "OI rro:vo
NO oo eati side 1•rth oome pr8!BVatlon) Finger: FmJl al ~lrral j(tt, exti:nm C7 t
Thumb: Rexcn mroo, .m <D'.ll.Clb1 In ptm of thJntl Is Injury Motor Complete? If YES, AIS=B
Finger: AexXll at MY joli C8 NO .I (No='IOUlilfy .ml cootra:tm oo moor m:tioo Thumb: Op~ aJcwxm m .m.ctm perp9llicJ.J0r rrue 11m llree 18Y81s beloW trn mol!J leVel on a il palm T giVen side. nlhe patllll1 has oormy rroTl)l8te
Finger: MxJuclb'l of Ille rmx rrgr T1 cta.~mca:xm)
Hip: Aci1ml L2 Are at least half (hall or more) of the key muscles below the
Hip E.xanal ro1a:m L3 neurological level of Injury graded 3 or better?
Hip: Exmm. allc1Jc:mn, mraJ rola:b1 L4 NO YES 1 1 Knee: FlexicJJ A
Ankle: lrMlsm aid fMISIJl INTERNATIONAL STANDARDS FOR NEUROLOGICAL A1l=c All=o Toe: \f' <nl P exl!n;jm CLASSIACATION OF SPINAL CORD INJURY 11 sematloo and motor tunctlon ls nonnal in an segments, AIS=E
Halw and Toe: llP .m pp nexm ard ~ LS Note AJS EIs used in ~V-lfJ t~ wnoo an inrJVldJa/ 1WII a <XXlll1mted
sa tm rfXXNef9(J rama tundlon. If at ma tesl.Vlg IV cJerlCils c¥e ram, 1119
EnhancingAssessments&PlanningInTheNDIS190906.docHalm: M1X:1oo S1 Page 14imMdJ;jof 29is~ 11/acl; /he~ lmpijrmmt Scae wes rK1 wt. ISC....._..___...-.W.0,,aj---l'tlS
PEDI-CAT = Response Scales For Daily Activities, Mobility and Social/Cognitive:
- Unable
- A Little Hard
- Hard
- Easy
- I don’t know For Responsibility: EnhancingAssessments&PlanningInTheNDIS190906.doc Page 15 of 29
WHODAS 36 Item Assessment
PLEASE NOTE: When scoring WHODAS, the following numbers are assigned to responses: 0 = No Difficulty
1 = Mild Difficulty
2 = Moderate Difficulty
3 = Severe Difficulty
4 = Extreme Difficulty or Cannot Do
Score
Understanding & communicating
D1.1 Concentrating on doing something for ten minutes? 0
D1.2 Remembering to do important things? 0
D1.3 Analysing and finding solutions to problems in day-to-day life? 0
D1.4 Learning a new task, for example, learning how to get to a new 0
place?
D1.5 Generally understanding what people say? 0
D1.6 Starting and maintaining a conversation? 0
Getting around
D2.1 Standing for long periods such as 30 minutes? 0
D2.2 Standing up from sitting down? 0
D2.3 Moving around inside your home? 0
D2.4 Getting out of your home? 0
D2.5 Walking a long distance such as a kilometre [or equivalent]? 0
Self-care
D3.1 Washing your whole body? 0
D3.2 Getting dressed?? 0
D3.3 Eating? 0
D3.4 Staying by yourself for a few days?? 0
Getting along with people
D4.1 Dealing with people you do not know?? 0
D4.2 Maintaining a friendship? 0
D4.3 Getting along with people who are close to you? 0
D4.4 Making new friends?? 0
D4.5 Sexual activities? 0
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 16 of 29
Life activities
D5.1 Taking care of your household responsibilities? 0
D5.2 Doing most important household tasks well? 0
D5.3 Getting all the household work done that you needed to do? 0
D5.4 Getting your household work done as quickly as needed? 0
D5.5 Your day-to-day work/school? 0
D5.6 Doing your most important work/school tasks well? 0
D5.7 Getting all the work done that you need to do? 0
D5.8 Getting your work done as quickly as needed? 0
Participation in society
D6.1 How much of a problem did you have in joining in community 0
activities (for example, festivities, religious or other activities) in the same way as anyone else can?
D6.2 How much of a problem did you have because of barriers or 0
hindrances in the world around you?
D6.3 How much of a problem did you have living with dignity because 0
of the attitudes and actions of others?
D6.4 How much time did you spend on your health condition, or its 0
consequences?
D6.5 How much have you been emotionally affected by your health 0
condition?
D6.6 How much has your health been a drain on the financial 0
resources of you or your family?
D6.7 How much of a problem did your family have because of your 0
health problems?
D6.8 How much of a problem did you have in doing things by yourself 0
for relaxation or pleasure?
Overall Score 0.00%
H1 Overall, in the past 30 days, how many days were these difficulties present? H2 In the past 30 days, for how many days were you totally unable to carry out your usual activities or work because of any health condition? H3 In the past 30 days, not counting the days that you were totally unable, for how many days did you cut back or reduce your usual activities or work because of any health condition?
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 17 of 29
MODIFIED Patient Name:________________
RANKIN Rater Name: _________________
SCALE (MRS) Date: _______________________
Score Description
0 No symptoms at all 1 No significant disability despite symptoms; able to carry out all usual duties and activities 2 Slight disability; unable to carry out all previous activities, but able to look after own affairs without assistance 3 Moderate disability; requiring some help, but able to walk without assistance 4 Moderately severe disability; unable to walk without assistance and unable to attend to own bodily needs without assistance 5 Severe disability; bedridden, incontinent and requiring constant nursing care and attention 6 Dead
TOTAL (0–6): _______
References
Rankin J. “Cerebral vascular accidents in patients over the age of 60.” Scott Med J 1957;2:200-15
Bonita R, Beaglehole R. “Modification of Rankin Scale: Recovery of motor function after stroke.”
Stroke 1988 Dec;19(12):1497-1500
Van Swieten JC, Koudstaal PJ, Visser MC, Schouten HJ, van Gijn J. “Interobserver agreement for the assessment of handicap in stroke patients.” Stroke 1988;19(5):604-7
Provided by the Internet Stroke Center — www.strokecenter.org
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 18 of 29
CanChild Centre for Childhood Disability Research
Institute for Applied Health Sciences, McMaster University,
1400 Main Street West, Room 408, Hamilton, ON, Canada L8S 1C7 Tel: 905-525-9140 ext. 27850 Fax: 905-522-6095 E-mail: canchild@mcmaster.ca Website: www.canchild.ca
GMFCS – E & R
Gross Motor Function Classification System
Expanded and Revised
GMFCS - E & R © Robert Palisano, Peter Rosenbaum, Doreen Bartlett, Michael Livingston, 2007
CanChild Centre for Childhood Disability Research, McMaster University
GMFCS © Robert Palisano, Peter Rosenbaum, Stephen Walter, Dianne Russell, Ellen Wood, Barbara
Galuppi, 1997
CanChild Centre for Childhood Disability Research, McMaster University
(Reference: Dev Med Child Neurol 1997;39:214-223)
INTRODUCTION & USER INSTRUCTIONS
The Gross Motor Function Classification System (GMFCS) for cerebral palsy is based on self-initiated movement, with emphasis on sitting, transfers, and mobility. When defining a five-level classification system, our primary criterion has been that the distinctions between levels must be meaningful in daily life. Distinctions are based on functional limitations, the need for hand-held mobility devices (such as walkers, crutches, or canes) or wheeled mobility, and to a much lesser extent, quality of movement. The distinctions between Levels I and II are not as pronounced as the distinctions between the other levels, particularly for infants less than 2 years of age. The expanded GMFCS (2007) includes an age band for youth 12 to 18 years of age and emphasizes the concepts inherent in the World Health Organization’s International Classification of Functioning, Disability and Health (ICF). We encourage users to be aware of the impact that environmental and personal factors may have on what children and youth are observed or reported to do. The focus of the GMFCS is on determining which level best represents the child’s or youth’s present abilities and limitations in gross motor function. Emphasis is on usual performance in home, school, and community settings (i.e., what they do), rather than what they are known to be able to do at their best (capability). It is therefore important to classify current performance in gross motor function and not to include judgments about the quality of movement or prognosis for improvement. The title for each level is the method of mobility that is most characteristic of performance after 6 years of age. The descriptions of functional abilities and limitations for each age band are broad and are not intended to describe all aspects of the function of individual children/youth. For example, an infant with hemiplegia who is unable to crawl on his or her hands and knees, but otherwise fits the description of Level I (i.e., can pull to stand and walk), would be classified in Level I. The scale is ordinal, with no intent that the distances between levels be considered equal or that children and youth with cerebral palsy are equally distributed across the five levels. A summary of the distinctions between each pair of levels is provided to assist in determining the level that most closely resembles a child’s/youth’s current gross motor function. We recognize that the manifestations of gross motor function are dependent on age, especially during infancy and early childhood. For each level, separate descriptions are provided in several age bands. Children below age 2 should be considered at their corrected age if they were premature. The descriptions for the 6 to 12 year and 12 to18 year age bands reflect the potential impact of
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 19 of 29
environment factors (e.g., distances in school and community) and personal factors (e.g., energy demands and social preferences) on methods of mobility. An effort has been made to emphasize abilities rather than limitations. Thus, as a general principle, the gross motor function of children and youth who are able to perform the functions described in any particular level will probably be classified at or above that level of function; in contrast, the gross motor function of children and youth who cannot perform the functions of a particular level should be classified below that level of function.
OPERATIONAL DEFINITIONS
Body support walker – A mobility device that supports the pelvis and trunk. The child/youth is physically positioned in the walker by another person. Hand-held mobility device – Canes, crutches, and anterior and posterior walkers that do not support the trunk during walking. Physical assistance – Another person manually assists the child/youth to move. Powered mobility – The child/youth actively controls the joystick or electrical switch that enables independent mobility. The mobility base may be a wheelchair, scooter or other type of powered mobility device. Self-propels manual wheelchair – The child/youth actively uses arms and hands or feet to propel the wheels and move. Transported – A person manually pushes a mobility device (e.g., wheelchair, stroller, or pram) to move the child/youth from one place to another. Walks – Unless otherwise specified indicates no physical assistance from another person or any use of a hand-held mobility device. An orthosis (i.e., brace or splint) may be worn. Wheeled mobility – Refers to any type of device with wheels that enables movement (e.g., stroller, manual wheelchair, or powered wheelchair).
GENERAL HEADINGS FOR EACH LEVEL
LEVEL I - Walks without Limitations
LEVEL II - Walks with Limitations
LEVEL III - Walks Using a Hand-Held Mobility Device
LEVEL IV - Self-Mobility with Limitations; May Use Powered Mobility
LEVEL V - Transported in a Manual Wheelchair
DISTINCTIONS BETWEEN LEVELS
Distinctions Between Levels I and II - Compared with children and youth in Level I, children and youth in Level II have limitations walking long distances and balancing; may need a hand-held mobility device when first learning to walk; may use wheeled mobility when traveling long distances outdoors and in the community; require the use of a railing to walk up and down stairs; and are not as capable of running and jumping. Distinctions Between Levels II and III - Children and youth in Level II are capable of walking without a hand-held mobility device after age 4 (although they may choose to use one at times). Children and youth in Level III need a hand-held mobility device to walk indoors and use wheeled mobility outdoors and in the community. Distinctions Between Levels III and IV - Children and youth in Level III sit on their own or require at most limited external support to sit, are more independent in standing transfers, and walk with a hand held mobility device. Children and youth in Level IV function in sitting (usually supported) but self mobility is limited. Children and youth in Level IV are more likely to be transported in a manual wheelchair or use powered mobility.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 20 of 29
Distinctions Between Levels IV and V - Children and youth in Level V have severe limitations in head and trunk control and require extensive assisted technology and physical assistance. Self mobility is achieved only if the child/youth can learn how to operate a powered wheelchair.
Gross Motor Function Classification System – Expanded and Revised
(GMFCS – E & R)
BEFORE 2ND BIRTHDAY
LEVEL I: Infants move in and out of sitting and floor sit with both hands free to manipulate objects. Infants crawl on hands and knees, pull to stand and take steps holding on to furniture. Infants walk between 18 months and 2 years of age without the need for any assistive mobility device. LEVEL II: Infants maintain floor sitting but may need to use their hands for support to maintain balance. Infants creep on their stomach or crawl on hands and knees. Infants may pull to stand and take steps holding on to furniture. LEVEL III: Infants maintain floor sitting when the low back is supported. Infants roll and creep forward on their stomachs. LEVEL IV: Infants have head control but trunk support is required for floor sitting. Infants can roll to supine and may roll to prone. LEVEL V: Physical impairments limit voluntary control of movement. Infants are unable to maintain antigravity head and trunk postures in prone and sitting. Infants require adult assistance to roll.
BETWEEN 2ND AND 4TH BIRTHDAY
LEVEL I: Children floor sit with both hands free to manipulate objects. Movements in and out of floor sitting and standing are performed without adult assistance. Children walk as the preferred method of mobility without the need for any assistive mobility device. LEVEL II: Children floor sit but may have difficulty with balance when both hands are free to manipulate objects. Movements in and out of sitting are performed without adult assistance. Children pull to stand on a stable surface. Children crawl on hands and knees with a reciprocal pattern, cruise holding onto furniture and walk using an assistive mobility device as preferred methods of mobility. LEVEL III: Children maintain floor sitting often by “W-sitting” (sitting between flexed and internally rotated hips and knees) and may require adult assistance to assume sitting. Children creep on their stomach or crawl on hands and knees (often without reciprocal leg movements) as their primary methods of self-mobility. Children may pull to stand on a stable surface and cruise short distances. Children may walk short distances indoors using a hand-held mobility device (walker) and adult assistance for steering and turning. LEVEL IV: Children floor sit when placed, but are unable to maintain alignment and balance without use of their hands for support. Children frequently require adaptive equipment for sitting and standing. Self-mobility for short distances (within a room) is achieved through rolling, creeping on stomach, or crawling on hands and knees without reciprocal leg movement. LEVEL V: Physical impairments restrict voluntary control of movement and the ability to maintain antigravity head and trunk postures. All areas of motor function are limited. Functional limitations in sitting and standing are not fully compensated for through the use of adaptive equipment and assistive technology. At Level V, children have no means of independent movement and are transported. Some children achieve self-mobility using a powered wheelchair with extensive adaptations.
BETWEEN 4TH AND 6TH BIRTHDAY
LEVEL I: Children get into and out of, and sit in, a chair without the need for hand support. Children move from the floor and from chair sitting to standing without the need for objects for support. Children walk indoors and outdoors, and climb stairs. Emerging ability to run and jump. LEVEL II: Children sit in a chair with both hands free to manipulate objects. Children move from the floor to standing and from chair sitting to standing but often require a stable surface to push or pull up on with their arms.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 21 of 29
Children walk without the need for a handheld mobility device indoors and for short distances on level surfaces outdoors. Children climb stairs holding onto a railing but are unable to run or jump. LEVEL III: Children sit on a regular chair but may require pelvic or trunk support to maximize hand function. Children move in and out of chair sitting using a stable surface to push on or pull up with their arms. Children walk with a hand-held mobility device on level surfaces and climb stairs with assistance from an adult. Children frequently are transported when traveling for long distances or outdoors on uneven terrain. LEVEL IV: Children sit on a chair but need adaptive seating for trunk control and to maximize hand function. Children move in and out of chair sitting with assistance from an adult or a stable surface to push or pull up on with their arms. Children may at best walk short distances with a walker and adult supervision but have difficulty turning and maintaining balance on uneven surfaces. Children are transported in the community. Children may achieve self-mobility using a powered wheelchair. LEVEL V: Physical impairments restrict voluntary control of movement and the ability to maintain antigravity head and trunk postures. All areas of motor function are limited. Functional limitations in sitting and standing are not fully compensated for through the use of adaptive equipment and assistive technology. At Level V, children have no means of independent movement and are transported. Some children achieve self-mobility using a powered wheelchair with extensive adaptations.
BETWEEN 6TH AND 12TH BIRTHDAY
Level I: Children walk at home, school, outdoors, and in the community. Children are able to walk up and down curbs without physical assistance and stairs without the use of a railing. Children perform gross motor skills such as running and jumping but speed, balance, and coordination are limited. Children may participate in physical activities and sports depending on personal choices and environmental factors. Level II: Children walk in most settings. Children may experience difficulty walking long distances and balancing on uneven terrain, inclines, in crowded areas, confined spaces or when carrying objects. Children walk up and down stairs holding onto a railing or with physical assistance if there is no railing. Outdoors and in the community, children may walk with physical assistance, a hand-held mobility device, or use wheeled mobility when traveling long distances. Children have at best only minimal ability to perform gross motor skills such as running and jumping. Limitations in performance of gross motor skills may necessitate adaptations to enable participation in physical activities and sports. Level III: Children walk using a hand-held mobility device in most indoor settings. When seated, children may require a seat belt for pelvic alignment and balance. Sit-to-stand and floor-to-stand transfers require physical assistance of a person or support surface. When traveling long distances, children use some form of wheeled mobility. Children may walk up and down stairs holding onto a railing with supervision or physical assistance. Limitations in walking may necessitate adaptations to enable participation in physical activities and sports including self-propelling a manual wheelchair or powered mobility. Level IV: Children use methods of mobility that require physical assistance or powered mobility in most settings. Children require adaptive seating for trunk and pelvic control and physical assistance for most transfers. At home, children use floor mobility (roll, creep, or crawl), walk short distances with physical assistance, or use powered mobility. When positioned, children may use a body support walker at home or school. At school, outdoors, and in the community, children are transported in a manual wheelchair or use powered mobility. Limitations in mobility necessitate adaptations to enable participation in physical activities and sports, including physical assistance and/or powered mobility. Level V: Children are transported in a manual wheelchair in all settings. Children are limited in their ability to maintain antigravity head and trunk postures and control arm and leg movements. Assistive technology is used to improve head alignment, seating, standing, and and/or mobility but limitations are not fully compensated by equipment. Transfers require complete physical assistance of an adult. At home, children may move short distances on the floor or may be carried by an adult. Children may achieve selfmobility using powered mobility with extensive adaptations for seating and control access. Limitations in mobility necessitate adaptations to enable participation in physical activities and sports including physical assistance and using powered mobility.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 22 of 29
BETWEEN 12TH AND 18TH BIRTHDAY
Level I: Youth walk at home, school, outdoors, and in the community. Youth are able to walk up and down curbs without physical assistance and stairs without the use of a railing. Youth perform gross motor skills such as running and jumping but speed, balance, and coordination are limited. Youth may participate in physical activities and sports depending on personal choices and environmental factors. Level II: Youth walk in most settings. Environmental factors (such as uneven terrain, inclines, long distances, time demands, weather, and peer acceptability) and personal preference influence mobility choices. At school or work, youth may walk using a handheld mobility device for safety. Outdoors and in the community, youth may use wheeled mobility when traveling long distances. Youth walk up and down stairs holding a railing or with physical assistance if there is no railing. Limitations in performance of gross motor skills may necessitate adaptations to enable participation in physical activities and sports. Level III: Youth are capable of walking using a hand-held mobility device. Compared to individuals in other levels, youth in Level III demonstrate more variability in methods of mobility depending on physical ability and environmental and personal factors. When seated, youth may require a seat belt for pelvic alignment and balance. Sit-to-stand and floor-to-stand transfers require physical assistance from a person or support surface. At school, youth may self-propel a manual wheelchair or use powered mobility. Outdoors and in the community, youth are transported in a wheelchair or use powered mobility. Youth may walk up and down stairs holding onto a railing with supervision or physical assistance. Limitations in walking may necessitate adaptations to enable participation in physical activities and sports including self-propelling a manual wheelchair or powered mobility. Level IV: Youth use wheeled mobility in most settings. Youth require adaptive seating for pelvic and trunk control. Physical assistance from 1 or 2 persons is required for transfers. Youth may support weight with their legs to assist with standing transfers. Indoors, youth may walk short distances with physical assistance, use wheeled mobility, or, when positioned, use a body support walker. Youth are physically capable of operating a powered wheelchair. When a powered wheelchair is not feasible or available, youth are transported in a manual wheelchair. Limitations in mobility necessitate adaptations to enable participation in physical activities and sports, including physical assistance and/or powered mobility. Level V: Youth are transported in a manual wheelchair in all settings. Youth are limited in their ability to maintain antigravity head and trunk postures and control arm and leg movements. Assistive technology is used to improve head alignment, seating, standing, and mobility but limitations are not fully compensated by equipment. Physical assistance from 1 or 2 persons or a mechanical lift is required for transfers. Youth may achieve self-mobility using powered mobility with extensive adaptations for seating and control access. Limitations in mobility necessitate adaptations to enable participation in physical activities and sports including physical assistance and using powered mobility. © Palisano, Rosenbaum, Bartlett & Livingston, 2007
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 23 of 29
What is the accuracy of selecting a particular level from 5 predefined Levels of Disability? If the accuracy is not defined it is normally accepted that if there are 5 levels to select from then it could have an error of one level either side of the selected level. This is demonstrated in the following WHODAS 2.0 example.
PLEASE NOTE: When scoring WHODAS, the following numbers are assigned to responses: 0 = No Difficulty
1 = Mild Difficulty
2 = Moderate Difficulty
3 = Severe Difficulty
4 = Extreme Difficulty or Cannot Do
If Level 2 is selected ie “Moderate Difficulty” then allowing for an error of ± 1 Level the true level would lie between Level 1 Mild Difficulty and Level 3 Severe difficulty.
To put this in percentage terms it would equate to the following assessment:
“Moderate Difficulty ± 20%”
This means that the true degree of difficulty lies somewhere between “Mild Difficulty” and “Severe Difficulty”. If 10 opinions were obtained there is no way of determining what correct opinion should be applied in an assessment.
This degree of inaccuracy in collecting accurate information about a degree of disability indicates very poor science if other technologies can cost effectively provide measured accurate and quantifiable objective data. Where possible subjective interpretations should be replaced with unbiased measured data.
It is virtually impossible to audit “interpretations” provided by the subject or assessor, thus producing poorer accountability and lower reliance on decisions or plans upon which they are based.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 24 of 29
Documents relating to NDIS Planners & Assessments Extracts from
ndis.gov.au March 2017 | NDIA Submission to Productivity Commission
Question 24 How should the performance of planners be monitored and evaluated? In the NDIA’s view, the key considerations for monitoring and evaluating planner performance (and Local Area Coordinator performance when performing planner type functions) are: ■ 100% compliance with all statutory requirements such that every decision is a valid administrative decision; ■ Difference between approved plans and the benchmark (in terms of the difference being appropriate against the reasonable and necessary considerations); ■ The quality of plans; ■ Rate of unexpected plan reviews (that is, plan reviews requested by the participant due to the initial plan being inadequate rather than at the end of the plan term); ■ Efficiency metrics (time taken to develop and approve plans); and ■ Requests for internal review of decisions.
The NDIA also monitors performance through systemic measures such as: ■ Participant satisfaction – Rates of satisfaction have dropped from those recorded during trial to those recorded during transition (from approximately 95% to 85% of participants reporting they were satisfied or very satisfied). The NDIA is working to address the reasons for this drop. The NDIA’s community partners are expected to capture participant satisfaction and it is the NDIA’s intent to develop an independent participant satisfaction process; ■ Complaints – both volume and substance; and ■ The National Quality Framework – where monthly audits are conducted on planner records and feedback provided through coaching and supervision.
Question 25 Do NDIA assessment tools meet these criteria? What measures or evidence are available for evaluating the performance of assessment tools used by the NDIA? The NDIA’s identification and selection of clinically accepted and widely utilised assessment tools involved extensive consultation and engagement with key stakeholders and experts across the key disability types. Stakeholders comprised clinical experts and researchers, and disability associations. The following features were considered in the selection of tools to assess functional capacity: ■ Ease of collection of the individual’s rating or ease of use of the tool for NDIA staff to assist participants to use self-rating questionnaires; ■ Time required to undertake assessments; ■ Cost of assessments or tools; ■ Whether the tool was validated and reliable; and ■ Whether population data was available for assistance with modelling.
The NDIA acknowledges that the widespread use of NDIA assessment tools predominantly commenced post-trial, with the relevant research and consultation undertaken to inform original selection of tools. Participant assessment information was captured retrospectively through a targeted data back-capture initiative to validate the assessment tools for use within the NDIS cohort of participants. The lack of consistent assessment mechanisms during trial contributed to
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 25 of 29
some of the discrepancy between package allocations and subsequent benchmark costs. The NDIA undertakes ongoing monitoring and adjustment of tools, in terms of effectiveness and suitability, through continuous improvement processes, involving regular collection of feedback on the tools, reviewing the suite of tools, and making any necessary changes, is also embedded within the process. Further work on the testing and trialing of assessment tools is required to build confidence in the selected tools and their relevance to both NDIS practice and the participant population. In particular, the following activities relating to NDIA assessment tools are currently underway: ■ Continued development of a disability-specific assessment tool for psychosocial disability. While the WHODAS is currently being used, work continues in relation to identifying and testing suitable psychosocial disability assessment instruments. ■ A focus on assessment tools for intellectual disability and the use of the Vineland for this cohort.
The NDIA has also engaged external experts to develop national diagnostic guidelines for Autism. It is important to note that the functional assessment tools discussed above are only a component of determining the participant’s package of supports. Other factors, such as age, level of sustainable informal assistance, community and mainstream support available, and capacity building, are important inputs to making reasonable and necessary support decisions. Tools for individuals with complex disabilities could also be improved.
Question 26 What are the likely challenges for monitoring and refining the assessment process and tools over time? What implications do these have for scheme costs? The purpose of the assessment process and associated tools, is to provide the equitable, efficient, and sustainable allocation of resources, across comparable cohorts. Trial experience provides some tangible evidence of the challenges and impacts associated with not embedding an objective and independent assessment process. A number of cost drivers emerged from trial and have the potential to compound cost, if not addressed: ■ A mismatch between benchmark costs and actual packages, with higher plan costs than expected for participants with higher level of function and lower plan costs than expected for participants with lower level of function; ■ Consistent and objective assessment of participants will promote the alignment of expected (benchmark) and actual costs. Planners, Local Area Coordinators and other organisations (e.g. Disability Support Organisations) have a critical role to support and empower participants to effectively use their budgets to realise their goals and achieve meaningful outcomes;
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 26 of 29
Productivity Commission - Appendix B: Assessment tools - Inquiry report - Disability Care and Support 2011
B Assessment tools B.1 Introduction As set out in Chapter 7, there is no universally agreed tool for assessing the care and support needs of individuals. There is however, reasonable consensus about the attributes that any assessment tool must have. These include validity, reliability and rigour. This appendix provides further detail on the suite of assessment tools currently in use, including information about their reliability and validity. The appendix begins by briefly reviewing the concepts and measurement of validity and reliability, and the pitfalls associated with their interpretation (section B.2). Section B.3 discusses the assessment tools outlined in table 7.1 in greater detail. B.2 Validity and reliability Effective assessment tools must be both valid and reliable. It is important to examine these concepts closely because the various measures of the reliability and accuracy of different assessment tools may not be comparable and need to be interpreted carefully.
Reliability
A reliable measure is one that measures a construct consistently across time, individuals, and situations. A good measure should produce consistent results when the test is repeated within a reasonably short space of time and when different assessors use the instrument to assess the same individual. Test-retest reliability Test-retest reliability is used to assess the consistency of a test across time. It is measured by the correlation between results from tests administered to the same group of people over two or more periods.1 The test assumes the stability of the underlying phenomenon being measured and that the testing context is the same. If these assumptions do not hold, a test-retest measure may reflect confounding factors, and fail itself to provide a reliable measure of reliability. For instance, differences in test-retest results might reflect use of trained assessors at one time and untrained ones at a later time; assessment with and without an observer; or tests that are sufficiently far apart in time that the measured phenomenon has changed (for example, tests of depression or anxiety could be expected to change over a 12-month period, while that would generally not be true for intelligence tests). Accordingly, if there are confounding factors, a low test-retest score may not indicate an intrinsically unreliable test. Moreover, a high test-retest score may be obtained for a poorly performing test for two reasons. First, the initial test may provide a poor measure of functioning, with later tests simply finding consistent, but equally poor measures. An example is provided in box 7.2. This would produce a high test-retest level, but have poor validity. Second, a subsequent test may be highly correlated with an initial test, but nevertheless provide a very different average measure of functioning. As an illustration, suppose that functioning was measured on a scale of 1-10 and 5 people got the scores 2, 8, 6, 4 and 10 in the first test and the scores of 1, 4, 3, 2 and 5 on the second test. The correlation measure
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 27 of 29
would be 1, and therefore an indicator of ‘perfect’ test-retest reliability, despite the fact that the average level of functioning among this group would have halved. The implication is that measures of test-retest reliability should consider changes in averages as well as correlation between successive measures. Finally, a test may produce high test-retest scores in trials, but fail to achieve the same degree of consistency in its actual application in the field. This may reflect differences in the quality and capabilities of the assessors and the varying contexts in which the tests apply. Some tests may be less susceptible to this problem than others, but their superiority may not be evident in the empirical studies associated with the development of the tests.
1 It is typically measured using the Pearson product-moment correlation coefficient. A coefficient of around 0.7 is often regarded as ‘good’.
Inter-rater reliability Inter-rater reliability measures the extent to which two different assessors (or raters) agree when assessing the various degrees of a person’s functional capacity and support needs. It is important to the NDIS because assessments determine resource allocation — both in terms of aggregate amounts and in particular areas. Ideally, most of the differences in resources received by people would reflect differences between their underlying needs, and not differences between the assessors rating them. Low inter-rater reliability would result in inequity and inefficiency. It may also have adverse dynamic effects if assessors change their own ratings in response to perceptions of biased testing by other assessors — with the result being excessive and misdirected resourcing. Poor inter-rater reliability would lead to significant scheme inefficiencies.
Low inter-rater reliability could arise because of particular weaknesses in a test (for example, due to many subjective items). Or it could arise because facets of the implementation and oversight of tests are poor. For example, in the absence of audits it would be possible for an assessor to change a test score because it led to a better outcome for the rated party (for example, more resources).
Inter-rater reliability is usually tested by having several assessors rate the same group of people separately, and then examine the extent to which they give the same pattern and average of scores. Whereas correlation coefficients are the most common (if deficient) measure of test-retest reliability, there are many measures of inter-rater reliability (Cohen’s Kappa, Krippendorff’s alpha, generalizability coefficients, correlation coefficients and the intra-class correlation coefficient), each with different advantages and disadvantages. Unfortunately, not all studies report which instrument has been used or the exact context of the tests, which makes it difficult to make comprehensive comparisons between assessments of reliability between instruments.
A key question in considering inter-rater reliability measures is the use of the test. For example: •a test may be used as a screen for entry into the NDIS. In that instance, regardless of formal measures of inter-rater reliability based on the components of the test, the key issue is inter-rater reliability for eligibility (either a yes or a no). It is not necessarily problematic
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 28 of 29
if different assessors score specific components differently, if the decisions they reach are the same in the case of resource allocation, the test conclusion is not dichotomous. Instead, the tests seek to determine the amount and/or makeup of support packages. This represents a particular challenge for test instruments because higher inter-rater reliability requires assessors to provide nearly equivalent assessments for each of the dimensions of the test. This is one reason why self-directed funding has efficiency benefits beyond those described in chapter 8. It is likely that inter-rater reliability is greater for total resource allocation, than it is for components of packages. However, many assessment instruments are not tested for their consistency in measuring budget allocations despite this being the most important facet of any arrangement that gives people with disabilities significant choice •an assessment may assist a person with a disability to plan their lives and to specify their aspirations coherently (which is useful both to the people with disabilities and to scheme managers and DSOs). This can be a different goal from eligibility or resource allocation, which would have implications for the design of the instrument.
Validity
Validity is the extent to which a test measures what it claims to measure (Goodwin 2009). The key aspect of assessing validity is to pose the question: ‘what is the tool for?’ and ask whether it meets that purpose. Validity is not determined by a single statistic, but by extensive research that demonstrates the relationship between the test and that which it is intended to measure. There are a number of types of validity — content, construct and criterion. Content validity: When a test has content validity, the items on the test represent the entire range of possible items the test should cover. An illustration is a mathematics test that only tested people’s arithmetic skills and no other essential mathematical skills. The tested activities would not cover all the domains relevant to assessment of mathematical skill, and would therefore lack content validity. In the area of disability, an assessment tool that failed to measure some important areas where supports could be necessary (say mobility aids or community access), would also lack content validity. Criterion-related validity: This relates to the capacity of an assessment tool to: •accurately predict some future relevant behaviour (for instance, a ‘work’ test that provides an accurate prediction of a person’s capacity to be employed in the future, or an IQ test that measures future academic success) •meaningfully relate to some other measure of associated behaviour (for example, a work test should produce similar results to other measures of a person’s work skills). Construct Validity: A test has construct validity if it truly measures some theoretical construct. For instance, an intelligence test would have construct validity if it measured a person’s capacity for complex cognitive tasks and problem solving.
EnhancingAssessments&PlanningInTheNDIS190906.doc Page 29 of 29