FOI 25/26-1784 DOCUMENT 8
I-CAN research – Validity and reliability
A supports classification and assessment system was initially developed for the ACT Department of redacted in 1998-1999 for clients living in community residential and respite settings who had a primary diagnosis of intellectual/developmental disability. The aim was to enable accurate classification and resource allocation in the context of limited resources.
- Early research resulted in the development of an instrument tilted “SCAN – Supports: Classification and Assessment of Needs”. This consisted of 5 domain scales (basic physical care considerations, adaptive skills, communication and social skills, medical and allied health needs, and behavioural considerations) and a risk profile.
- Actual supports received, including staff hours over a 24-hour period and the presence and type of night support required to assist individuals in their current activities and living environments was also recorded.
- The SCAN was then administered to 116 clients who had an intellectual disability, many with additional disabilities: psychiatric conditions (22%), epilepsy or seizures (38%), a progressive degenerative condition (35%), visual impairment (22%), and/or hearing impairment (8%). Additionally, 23% experienced mobility problems, with 6% totally immobile. Some individuals had multiple disabilities requiring constant care and support of a life sustaining nature. A total of 59% of the sample were considered to require support at all times, and only 11% were able to travel independently.
- Ages ranged from 13-50 years average age = 29 years 10 months
- A multi-dimensional scoring model and multiple regression analyses and discriminant function analyses were used to evaluate how well 5 domain scales and the risk profile predicted current support, measured by current staff hours per 24-hour period.
- the R squared was .58, meaning that jointly, the 5 predictor variables explained 58% of the variation in results. Basic physical care and adaptive behaviour were identified as the significant predictor variables in this analysis.
- A discriminant analysis using 4 levels of staff support hours as the independent variable accounted for 88% of the variance and identified the 3 domains of adaptive behaviour, basic physical care and communication as significant predictors of staff support hours (Chi square = 76.35, df=15, p= .000). A second function included both the behaviour and medical and allied health needs domains and accounted for another 7% of the variance.
- The SCAN was compared with 2 other instruments, the Functional Independence Measure (FIM) that assesses and the Inventory for Client and Agency Planning (ICAP) for concurrent validity for N=30. The FIM concentrates upon physical care and mobility issues, and as predicted, there was a high negative correlation (r=0.96) between the FIM Motor Sub Scale and the SCAN basic physical care domain scale. The ICAP measures both adaptive behaviour and problem behaviour and these scales correlated highly and in the expected direction with similar SCAN domains. Thus SCAN measured content similar to parts of the FIM and ICAP in a reliable and consistent way.
- 3 month Test-retest reliability results (N=30) proved very strong for the areas of adaptive behaviour, communication and basic physical care, with correlations all above r= 0.90, psychiatric (r=0.98) and epilepsy (r=0.81) conditions. Spearman
FOI 25/26-1784
rho correlation coefficient for behaviour was r= 0.77. This was considered respectable, since behaviour is an area that can be subject to change.
- SCAN proved easy and practicable to use and meaningful to service providers, clients, family members, advocates and guardians.
- Requests resulted in the development of SCAN-D for use in Day programs and was well received but lack of research funding prevented analysis of results
I-CAN
- The SCAN was the basis of I-CAN v1-I-CANv3 which further refined and trialed extensively under an ARC Linkage Grant 2001- 2004.
- Involved CDS/ The university of Sydney and 16 partner organisations supporting people with various disabilities in residential and day program settings across the Australian Capital Territory, New South Wales, Queensland and Victoria.
- A total of N=5071 individuals participated in trialing successive versions of the I-CAN: N=1012 people with disabilities, N= 3093 staff from disability organizations, N= 1012 facilitators, 339 parent/s, relatives or guardians, 23 advocates, and 72 others such as friends.
- Was aligned with the ICF classification system (WHO, 2001).
- The instrument was analysed using factor analytic and scale reliability techniques, and relationships were explored between sub scales and between scale results and independent variables such as gender, age and type of disability.
- Reliability studies were also conducted to assess internal consistency of scales, test-retest reliability and inter-rater reliability.
- All test-retest correlations on the Activity and Participation Scales (A&P) and the A&P Total Scale were significant and in the moderate to high range with r=0 .46 to 0.93
- Test-retest results on the Behaviour Scale r=0 .66 to 0.70 at the one year retest and .25 to .32 at the two year retest.
- Test-retest reliability coefficients for the Physical Health Scale and the Mental Emotional Health Scale ranged from r=-0.22 to r=0.51 at the one and two year retest – measured actual changes.
- An inter-rater reliability study using a 10% stratified random sampling technique for participants who completed Version 2 of the instrument found the reliability for a total of 3228 items ranged from r=0.96 to r=1.00, and the overall inter-rater reliability was very high at r= 0.99.
- Concurrent validity was investigated in several small studies, by exploring the relationship between the I-CAN and the Inventory for Client and Agency Planning (ICAP) (Bruininks, et al., 1986) and the Quality of Life Questionnaire (QOL-Q) (Schalock & Keith, 1993).
- The correlation coefficients between the I-CAN Health and Well Being(HWB) and A&P Scales and the ICAP Service Level Scores were all negative and significant at p<.05. Sixty percent of the correlation coefficients were significant at p<.05 between the I-CAN A&P scales and the ICAP Adaptive Behaviour Scales, while 38% of the I-CAN HWB Scales and the ICAP Maladaptive Indexes correlated significantly. The I-CAN Mental Emotional Health Scale correlated significantly
FOI 25/26-1784
with the ICAP Internalised, Externalised and General Maladaptive Indexes, while the I-CAN Behaviour Scale correlated significantly with the Externalised and General Maladaptive Indexes.
- Low to moderate correlation coefficients were evident between the I-CAN Scales and the Quality of Life Questionnaire (QOL-Q) Total, indicating little relationship between the amount of support required and quality of life generally
- Predictive validity studies were undertaken using amount of day time and night time support required and support hours using a 24 hour clock to measure where and when support was currently provided to participants. Regression analyses were run using these measures with I-CAN scale scores.
- A multiple regression analysis with daytime support continuous dependent variable (0-16 hours can be left alone) against the six activity and participation A7P) Scales and the A&P Total, found three scales measuring community participation, ability to make daily decisions and problem solve, and communication together explained 40% of the variance of the daytime support hours required, F (3, 1005)= 221.78, p<.001. Low % variance due to little discrimination between participants regarding the number of daytime support hours recorded, with the majority of the sample (68%) receiving almost total day time support and over 90% receiving at least eight hours support each day. Outlying cases were those individuals (8%) who only required drop- in support rather than daily support.
- Health and Well Being factors, as assessed using Version 2.1-2.2 for 338 participants included medical conditions, epilepsy, mental illness and known behaviour. These factors combined only accounted for 8% of the variance for daytime support. This increased only marginally to 11% for the 76 participants who were assessed according to the Physical Health, Mental Emotional Health, Behaviour, and Health Services Scales on Version 3
- Multiple regression analyses for I-CAN scales of HWB and A&P against current support hours best fit model using a stepwise entry explained only 27% of the variance, (F (2, 275) =3 4.37, p<.001), with significant contributions made by the scales measuring community participation, (22.1%), ability to make daily decisions and problem solve (3.4%), and Medical Condition (1.8%).
- Multiple regression analysis found low variance according to gender, age and disability grouping.
- Additional factors to consider apart from hours of support identified as - organizational factors relating to group supervision practices, concern for occupational health and safety and risk factors, and duty of care issues.
Lack of research funding has prevented further research.