00smith2Frontmatter.i_xii
Technical Appendix
2012 ADDENDUM TO THE TECHNICAL APPENDIX
Since the publication of the ELLCO Pre-K in 2008, psychometric analyses of the ELLCO Pre-K were performed using data collected from 2008 through 2010 as part of a U.S. Department of Education–funded Early Reading First project: Reading to Nurture Excellence in Worcester (RENEW). ELLCO Pre-K observations were conducted twice annually in 35 classrooms over the course of 3 years. Due to teacher turnover, the total number of classroom observations conducted during this 3-year period equals 203. The RENEW project, and Early Reading First projects in general, are concerned with improving language and literacy outcomes for disadvantaged children. The data in these analyses were therefore collected in classrooms that serve a low-income, at-risk population.
Interrater Reliability
Research use of the ELLCO Pre-K is predicated on the appropriate training of observers. Observers are selected based on knowledge of early childhood classrooms and experience in conducting observations. Prospective observers participate in a daylong training session on using the ELLCO Pre-K, which includes background information on language and literacy development, explanation of how to use the instrument, and scoring practice using written examples and video footage. Observers then participate in a second day of training that consists of supervised practice using the tool in a classroom setting followed by independent scoring and discussion of scoring decisions. When observers are trained and supervised appropriately, we have achieved an average interrater reliability of 74%.
Table A.1. Descriptive statistics for the ELLCO Pre-K (n = 203)
| Composite variable | Mean | Standard deviation | Minimum | Maximum |
|---|---|---|---|---|
| Classroom Structure | 3.54 | 0.71 | 1.50 | 5.00 |
| Curriculum | 3.14 | 0.72 | 1.67 | 4.67 |
| The Language Environment | 2.85 | 0.72 | 1.00 | 4.75 |
| Books and Book Reading | 3.24 | 0.87 | 1.20 | 5.00 |
| Print and Early Writing | 3.15 | 0.99 | 1.00 | 5.00 |
| General Classroom Environment subscale | 3.37 | 0.67 | 1.57 | 4.86 |
| Language and Literacy subscale | 3.09 | 0.75 | 1.50 | 4.92 |
Reliability Analysis
Reliability analysis was conducted to examine the internal consistency of the ELLCO Pre-K. Table A.2 shows alphas obtained for the five ELLCO Pre-K sections as well as for the two subscales. Cronbach’s alphas for the five sections were high, ranging from .723 for the Curriculum section to .894 for the Print and Early Writing section. Item-total correlations for each section were moderate to high and ranged as follows: Classroom Structure section from .519 for Item 4, Personnel, to .657 for Item 1, Organization of the Classroom.
Table A.2. Cronbach’s alpha for the five sections and two subscales that compose the ELLCO Pre-K (n = 203)
| Composite variable | $\alpha$ |
|---|---|
| Classroom Structure | .785 |
| Curriculum | .723 |
| The Language Environment | .786 |
| Books and Book Reading | .871 |
| Print and Early Writing | .894 |
| General Classroom Environment subscale | .864 |
| Language and Literacy subscale | .922 |
Measuring Stability and Change
Using data collected from intervention and comparison group classrooms from the RENEW project, we can report on the ability of the ELLCO Pre-K to measure both stability and change over time. The intervention group consisted of Head Start teachers from the Worcester Community Action Council in Worcester, Massachusetts, who engaged in an intensive professional development process designed to improve language and literacy instruction and classroom environments. The comparison group was composed of Head Start teachers from nearby communities who were not engaged in a professional development intervention. In both groups, classrooms were observed biannually, in the fall and in the spring, over the course of 3 years. The pre- and post-observations for each year were paired to create a dataset of 128 fall and spring cases (69 intervention cases, and 59 comparison cases).
Although means for RENEW classrooms were consistently higher than those for comparison classrooms, comparison means remained relatively stable from fall to spring; difference-of-means analysis (t-tests) indicate a statistically significant difference between the fall and spring scores for RENEW classrooms across all sections and subscales of the ELLCO Pre-K. The stability of the ELLCO Pre-K is a good indicator of the tool’s test–retest reliability.
Table A.3. Stability and change scores: mean (standard deviation) for the ELLCO Pre-K (n = 69 RENEW; 59 comparison)
| Composite variable | Fall | Spring | ||
|---|---|---|---|---|
| RENEW Intervention | Comparison | RENEW Intervention | Comparison | |
| Classroom Structure | 3.87(0.63) | 3.10(0.48) | 4.11***(0.55) | 3.13(0.51) |
| Curriculum | 3.35(0.71) | 2.75(0.57) | 3.71***(0.45) | 2.72(0.54) |
| The Language Environment | 3.10(0.66) | 2.57(0.60) | 3.31*(0.51) | 2.48(0.65) |
| Books and Book Reading | 3.66(0.61) | 2.59(0.61) | 4.04**(0.70) | 2.70(0.54) |
| Print and Early Writing | 3.68(0.72) | 2.28(0.58) | 4.14***(0.55) | 2.50(0.57) |
| General Classroom Environment subscale | 3.65(0.61) | 2.95(0.48) | 3.94***(0.45) | 2.96(0.46) |
| Language and Literacy subscale | 3.48(0.56) | 2.51(0.47) | 3.82***(0.45) | 2.58(0.46) |
- p < .05, ** p < .01, *** p < .001
2008 TECHNICAL APPENDIX
This technical appendix reports data that were collected from 1997 to 2002 as part of the development of the ELLCO Toolkit, Research Edition, as well as additional data that were collected from 2002 to 2007 using the Research Edition. The most significant changes are the integration of the Literacy Environment Checklist and the Literacy Activities Rating Scale into the architecture of the observation, and the inclusion of detailed descriptive indicators for each of the five scale points. Specific psychometric analyses on the current ELLCO Pre-K will be reported as the tool is used.
Psychometric Properties of the Literacy Environment Checklist
The psychometric properties presented for the Literacy Environment Checklist (ELLCO Toolkit, Research Edition) are based on data from Year 4 of the NEQRC project combined with data from Years 1–3 of the LEEP project. The total sample size was 255, although the actual subsample sizes vary depending on the analyses conducted. Many of the classrooms included were in Head Start programs.
Table A.4. Descriptive statistics for subscale and total score data for the Literacy Environment Checklist in the ELLCO Toolkit, Research Edition (n = 255)
| Composite variable | Mean | Standard deviation | Minimum | Maximum |
|---|---|---|---|---|
| Books subscale | 11.13 | 3.90 | 2.00 | 20.00 |
| Writing subscale | 10.44 | 4.22 | 1.00 | 20.00 |
| Literacy Environment Checklist Total score | 21.57 | 7.37 | 5.00 | 40.00 |
Reliability Analysis
Reliability analysis was conducted to examine the internal consistency of the Literacy Environment Checklist. Cronbach’s alpha of .84 for the Total score shows good internal consistency. All item–total correlations were moderate to high (ranging from .54 to .55).
Measuring Stability and Change
Using the data collected from the LEEP classrooms, we reported preliminary findings on the ability of the Literacy Environment Checklist to measure both stability and change over time. When looking at mean scores across the 3 years of the LEEP project, the fall scores of the intervention are slightly higher on the three dimensions of the Literacy Environment Checklist than the comparison group.
Table A.6. Stability and change in Literacy Environment Checklist scores (ELLCO Toolkit, Research Edition), fall and spring means, for Years 1–3 of the Literacy Environment Enrichment Program (LEEP)
| Composite variable | Fall | Spring | ||
|---|---|---|---|---|
| Comparison | LEEP Intervention | Comparison | LEEP Intervention | |
| Books subtotal | 9.25 | 10.53 | 10.45 | 14.84 |
| Writing subtotal | 8.77 | 11.21 | 9.12 | 14.26 |
| Literacy Environment Checklist | 18.12 | 20.86 | 19.52 | 29.03 |
- p < .05, ** p < .01, *** p < .001
Psychometric Properties of the Classroom Observation
The Classroom Observation has been used for research for the NEQRC and LEEP. The psychometric properties presented in the sections that follow come from various analyses of data from Year 4 of the NEQRC research project combined with data collected from Years 1–3 of the LEEP project. The total sample size was 308 classrooms, though the actual subsample size varies depending on the analyses conducted.
Table A.7. Descriptive statistics for data for the Classroom Observation in the ELLCO Toolkit, Research Edition (n = 308)
| Composite variable | Mean | Standard deviation | Minimum | Maximum |
|---|---|---|---|---|
| General Classroom Environment subtotal | 3.44 | 0.79 | 1.20 | 5.00 |
| Language, Literacy, and Curriculum subtotal | 3.02 | 0.75 | 1.13 | 5.00 |
| Classroom Observation Total score | 3.15 | 0.71 | 1.29 | 5.00 |
Reliability Analysis
Reliability analysis was conducted to examine the internal consistency of the Classroom Observation using data from 308 classrooms. Cronbach’s alpha of .83 for the General Classroom Environment shows good internal consistency for this composite.
Measuring Stability and Change
In the LEEP project, classrooms were observed in the fall and in the spring of Years 1–3, yielding the ability to measure change over time using the Classroom Observation. These changes resulted in intervention group scores that were statistically significantly different from the comparison group scores in every category, indicating both stability and instructional sensitivity.
Table A.10. Stability and change in Classroom Observation scores (ELLCO Toolkit, Research Edition), fall and spring means, for Years 1–3 of the Literacy Environment Enrichment Program (LEEP)
| Composite variable | Fall | Spring | ||
|---|---|---|---|---|
| Comparison | LEEP Intervention | Comparison | LEEP Intervention | |
| General Classroom Environment subtotal | 3.26 | 3.61 | 3.42 | 3.91 |
| Language, Literacy, and Curriculum subtotal | 2.85 | 3.01 | 2.93 | 3.75 |
| Classroom Observation Total score | 2.97 | 3.19 | 3.08 | 3.74 |
Correlation with Another Widely Used Measure
As part of the NEQRC project, the Classroom Observation has been used in conjunction with the Classroom Profile, a widely used tool for assessing the overall quality of early childhood classrooms. The findings provide evidence of divergent validity for the Classroom Observation.
REFERENCES
Abbott-Shim, M., & Sibley, A. (1998). Assessment Profile for Early Childhood Programs. Atlanta, GA: Quality Assist.
Dickinson, D.K., & Chaney, C. (1998). Profile of Early Literacy Development. Newton, MA: Education Development Center, Inc.
Dickinson, D.K., Sprague, K., et al. (2000). Classroom factors that foster literacy and social development of children from different language backgrounds.
Dickinson, D.K., & Tabors, P.O. (Eds.). (2001). Beginning literacy with language: Young children learning at home and school. Baltimore: Paul H. Brookes Publishing Co.