OVERVIEW
The IMPACT Social Communication Rating Scale is a norm-referenced pragmatic language rating scale for children and young adults ages 5 through 21 years old. It is composed of 35-40 test items, and has three separate forms to be completed by clinician, parent(s), and teacher(s). It is an accurate and reliable assessment that yields valid results on informal observations of pragmatic language such as intent to socialize, nonverbal language (e.g., facial expressions, tone of voice), theory of mind, social reasoning and cognitive flexibility. Normative data of this test is based on a nationally representative sample of 1006 children and young adults in the United States.
Standardization and Normative Information
The normative data for the IMPACT Social Communication Rating Scale test are based on the test performance of 1006 examinees across 11 age groups (shown in Table 2.1) in 17 states (Arizona, California, Colorado, Nevada, Idaho, Illinois, Iowa, Kansas, Ohio, Minnesota, Florida, New York, Pennsylvania, Florida, South Carolina, Texas, Washington).
The data were collected during the school-year of 2018-2019 by 34 state licensed speech-language pathologists recruited through Go2Consult Speech and Language Services, a certified special education staffing company. All standardization project procedures were reviewed and approved by IntegReview IRB, an accredited and certified independent institutional review board. To ensure representation of the national population, the he IMPACT Social Communication Rating Scale standardization sample was selected to match the US Census data reported in the ProQuest Statistical Abstract of the United States, 2017 (ProQuest, 2017). The sample was stratified within each age group by the following criteria: gender, race or ethnic group and geographic region. The demographic table below (Table 2.2) specifies the distributions of these characteristics and shows that on the whole, the sample is nationally representative.
Criteria for inclusion in the normative sample
A good assessment is one that yields results that will benefit the individual being tested or society as a whole (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, and NCME], 2014). One way we can tell if an assessment is a good test, is if it includes adequate norms. Previous research has suggested that utilizing a normative sample can be beneficial in the identification of a disability and that the inclusion of children with disabilities may negatively impact the test’s ability to differentiate between children with disorders and children who are typically developing (Peña, Spaulding, & Plante, 2006). Since the purpose of the IMPACT Social Communication Rating Scale is to help to identify students who present with social communication deficits, it was critical to exclude students from the normative sample who have diagnoses that are known to influence social communication (Peña, Spaulding, & Plante, 2006). Students who had previously been diagnosed with a specific language impairment or learning disability were not included in the normative sample. Further, students were excluded from the normative sample if they were diagnosed with autism spectrum disorder, intellectual disability, hearing loss, neurological disorders, or genetically syndromes. Students who present with articulation disorders or attention-deficit/hyperactivity disorders were allowed to be included in the normative sample, as long as there was no co-occurring pragmatic language disorder. To sum up, in order for students to be included in the present normative sample, students must have met criteria of having normal language development, and show no evidence of a social language communication disorder. Students used in the present normative sample had no other diagnosed disabilities and were not receiving speech and language support or any other services. Thus, the normative sample for the IMPACT Social Communication Rating Scale provides an appropriate comparison group (i.e., a group without any known disorders that might affect social communication) against which to compare students with suspected disorders.
*Note: The IMPACT Social Communication Rating Scale is designed for students who are native speakers of English and/or are English language learners (ELL) who have demonstrated a proficiency in English based on state testing scores and school district language evaluations. Additionally, students who were native English speakers and also spoke a second language were included in this sample.
Norm-referenced testing is a method of evaluation where an individual’s scores on a specific test are compared to scores of a group of test-takers (e.g., age norms) (AERA, APA, and NCME, 2014). Clinicians can compare clinician, teacher, and parent ratings on the IMPACT Social Communication Rating Scale to this normative sample to determine whether a student is scoring within normal limits or, if their scores are indicative of a social communication disorder. Administration, scoring, and interpretation of the IMPACT Social Communication Rating Scale must be followed in order to make comparisons to normative data. This manual provides instructions to guide examiners in the administration, scoring, and interpretation of the rating scale.
Validity
Single-cut Scores
It is often common practice to use single cut scores (e.g., -1.5 standard deviations) to identify disorders, however, this is not evidence-based and there is actually evidence that advises against using this practice (Plante & Farinella, 2006). When using single cut scores (e.g., -1.5 SD, -2.5 SD, etc.) we may under identify students with impairments on tests for which the best-cut score is higher and over identify students impairments on tests for which the best-cut score is lower. Additionally, using single cut scores may go against IDEA’s mandate, which states assessments must be valid for the purpose for which they are used.
Sensitivity and Specificity
Table 3.1 shows the cut scores needed to identify social communication disorders within each age range. Additionally, this table demonstrates the sensitivity and specificity information that indicates the accuracy of identification at these cut scores. Sensitivity and specificity are diagnostic validity statistics that explain how well a test performs. Vance and Plante (1994) set forth the standard that for a language assessment to be considered clinically beneficial, it should reach at least 80% sensitivity and specificity.
Thus, strong sensitivity and specificity (i.e., 80% or stronger) is needed to support the use of a test in its identification of the presence of a disorder or impairment. Sensitivity measures how well the assessment will accurately identify those who truly have a social language disorder (Dollaghan, 2007). If sensitivity is high, this indicates that the test is highly likely to identify the pragmatic language disorder, or, there is a low chance of “false positives.” Specificity measures the degree to which the assessment will accurately identify those who do not have a pragmatic language disorder, or how well the test will identify those who are “typically developing” (Dollaghan, 2007).
Criterion Validity
In assessing criterion validity, the IMPACT Social Communication Rating Scale was correlated to other measures of social communication: the Clinical Assessment of Pragmatics (CAPs), the Social Responsiveness Scale -2 (SRS-2) and the Pragmatic Language Skills Inventory (PLSI). Time between test administrations ranged from the same day to 5 days.
The concurrent validity was assessed using Pearson’s correlation among all measures. Correlation coefficients of ≥0.7 are recommended for same-construct instruments while moderate correlations of ≥ 0.4 to ≤0.70 are acceptable. The level of significance was set at p≤0.05. When assessing validity, the IMPACT Social Communication Rating Scale was substantially correlated with the CAPs test and the SRS-2: 0.93, and 0.82 respectively, p<0.001. The correlations are the lowest with the Pragmatic Language Skills Inventory (Table 3.3). While there is an apparent relationship between performance on all three measures, the IMPACT Social Communication Rating Scale evaluates social language from a conceptually different framework.
Group Differences
Since a social communication assessment tool is designed to identify those examinees with social language deficits, it would be expected that individuals identified as likely to exhibit pragmatic language deficits would score lower than those who are typically developing. Mean standard scores for two clinical groups of examinees (autism spectrum disorder and social communication disorder) who were administered the IMPACT Social Communication Rating Scale are listed in Table 3.4. The mean for the outcome variables (Clinician, Teacher and Parent ratings) were compared among the two clinical groups and the typically developing group of examinees using Kruskal Wallis analysis of variance (ANOVA). Further comparisons in mean scores between the groups were examined using Mann- Whitney U test. The level of significance was set at p≤0.05. Further comparisons using Mann- Whitney U test showed that there was a significant difference among all the study groups (p<0.001, refer to Table 3.4)
Inclusion/Exclusion Criteria for the Group Differences Study
Typically developing participants were selected based on the following criteria: 1) exhibited hearing sensitivity within normal limits; 2) presented with age-appropriate speech and language skills; 3) successfully completed each school year with no academic failures; and 4) attended public school and placed in general education classrooms. Typically developing participants were excluded if they presented with conditions as defined by a DSM- V diagnosis of mental health problems such as clinical disorders, personality disorders and general medical conditions.
Inclusion criteria for the autism spectrum disorder (ASD) group was: 1) having a current diagnosis within the autism spectrum disorder as defined by a DSM- V (based on medical records and school-based special education eligibility criteria); and 2) currently attending a local public school, and enrolled in the general education classroom for at least 3 hours per day. Participants were excluded if they presented with comorbid conditions as defined by a DSM- V diagnosis of mental health problems such as clinical disorders, personality disorders and general medical conditions.
Finally, the inclusion criteria for the social communication disorder (SCD) group were: 1) having a current diagnosis within the social communication disorder as defined by a DSM- V (based on medical records and school-based special education eligibility criteria, and a score of 76 or below on the Clinical Assessment of Pragmatics test and displaying inappropriate or inadequate usage of pragmatic language as documented by medical or special educational records); 2) being enrolled in the general education classroom for at least 4 hours per day. Students from the SCD group were excluded from the study if the following were identified: 1) intellectual disability, learning disability, emotional disturbance; 2) comorbid conditions where the student has a DSM- V diagnosis of mental health problems including clinical disorders, personality disorders and general medical conditions.
