# Technical Appendix

This technical appendix reports data that were collected from 1997 to 2002 as part of the development of the ELLCO Toolkit, Research Edition, as well as additional data that were collected from 2002 to 2007 using the Research Edition. As described in Chapter 1, on the basis of this data, along with feedback from the field, we made numerous changes that serve to make the ELLCO Pre-K easier to use and score. The most significant changes are the integration of the Literacy Environment Checklist and the Literacy Activities Rating Scale into the architecture of the observation, and the inclusion of detailed descriptive indicators for each of the five scale points. Specific psychometric analyses on the current ELLCO Pre-K will be reported as the tool is used. For reasons that we outline at the end of this appendix, however, we believe that the ELLCO Pre-K will prove to be as reliable, if not more reliable, than the Research Edition. The ELLCO Toolkit, Research Edition, was pilot tested and used in several research studies from its development, with the minor revisions that typically occur through practical use and feedback, through its original publication in 2002; this included research conducted in more than 150 preschool classrooms for the Head Start–funded New England Quality Research Center (NEQRC) (funder: U.S. Department of Health and Human Services, Administration for Children and Families; 1995–2000) and the Literacy Environment Enrichment Project (LEEP) (funder: U.S. Department of Education, Office of Educational Research and Improvement; 2000–2003), both based in the Center for Children & Families (CC&F) at Education Development Center, Inc., in Newton, Massachusetts. Since its initial publication, researchers at CC&F have used the ELLCO Toolkit, Research Edition, in more than 250 classrooms as part of six different projects:

- The New England Quality Research Center: The Next Generation (funder: U.S. Department of Health and Human Services, Administration for Children and Families; 2001–2006)
- Examining the Efficacy of Two Models of Preschool Professional Development in Language and Literacy (funder: U.S. Department of Education, Institute of Education Sciences; 2005–2007)
- Child Care Quality: Does Partnership Make a Difference—an Extension of the Partnership Impact Project (funder: U.S. Department of Education, 2004–2007)
- Evaluation of the Newport Early Reading First Collaborative (funder: U.S. Department of Education, 2003–2006)
- Evaluation of the Springfield Early Reading First Initiative (funder: U.S. Department of Education, 2003–2007)
- Connecticut Is Reading First (funder: U.S. Department of Education, 2005 to present)

All of these projects are concerned with the language and literacy development of children from lower income families and communities. Because of this, data reported here are based on centers and classrooms in lower income communities. The data reported in the main body of the appendix come from 30 classrooms from the NEQRC study and a total of 117 classrooms for the LEEP study. Each of the NEQRC classrooms were observed on one occasion, whereas most of the LEEP classrooms were observed on two occasions (fall and spring), and a few LEEP classrooms were visited a total of three times. In the data used to calculate means, correlations, and Cronbach’s alpha, each visit to a classroom is counted as a separate observation. In the data used to report stability and change (LEEP classrooms only), each LEEP classroom is counted only once, and fall and spring scores are treated as distinct variables. An addendum to the original research at the end of this technical appendix describes Cronbach’s alpha analyses performed using a larger sample of data collected between 2001 and 2007. These data are from a total of 259 classrooms from the following projects:

- The New England Quality Research Center: The Next Generation (n 57)
- Examining the Efficacy of Two Models of Preschool Professional Development in Language and Literacy (n 67)
- Child Care Quality: Does Partnership Make a Difference—an Extension of the Partnership Impact Project (n 66)
- Evaluation of the Newport Early Reading First Collaborative (n 26)
- Evaluation of the Springfield Early Reading First Initiative (n 22)
- Connecticut Is Reading First (n 21)

# PSYCHOMETRIC PROPERTIES OF THE LITERACY ENVIRONMENT CHECKLIST

The psychometric properties presented for the Literacy Environment Checklist (ELLCO Toolkit, Research Edition) are based on data from Year 4 of the NEQRC project combined with data from Years 1–3 of the LEEP project. Data from the NEQRC project were collected during the winter of 1998–1999 (n 29). The data from Year 1 of the LEEP project were collected in the fall of 1998 (n 26) and the spring of 1999 (n 26). Data from Year 2 of the LEEP project were collected in the fall of 1999 (n 42) and spring of 2000 (n 38). Data from Year 3 of the LEEP project were collected in the fall of 2000 (n 47) and spring of 2001 (n 47). Together, the projects resulted in a total sample size of 255, although the actual subsample sizes vary depending on the analyses conducted. Many of the classrooms included were in Head Start programs. Unlike the Classroom Observation, the Literacy Environment Checklist and the Literacy Activities Rating Scale have been used for research only in preschool classrooms and were designed specifically to help identify the impact of our literacy intervention in those classrooms. They have not been used to predict children’s growth; rather, they have been used in conjunction with the Classroom Observation to pinpoint the specific effects of a literacy intervention. The items from the Literacy Environment Checklist of the ELLCO Toolkit, Research Edition, have been incorporated into the main body of the ELLCO Pre-K. The Literacy Activities Rating Scale is not included in the ELLCO Pre-K. (See Table A.15 later in this appendix for item-level correspondences between the ELLCO Pre-K and the ELLCO Toolkit, Research Edition.)

# Interrater Reliability

Research use of the Literacy Environment Checklist was predicated on the appropriate training of observers. We have required that prospective observers be familiar with theories of early literacy development and have an understanding of the range of instructional methods that are typically used in classrooms.

Prospective observers received a daylong training session on using the ELLCO Toolkit, Research Edition, which included background information on language and literacy development, explanation of the toolkit, and videotape examples; then training session participants received a second day of supervised practice in using the toolkit. When observers were trained and supervised appropriately, we achieved an average interrater reliability of 88% with relative ease. (This interrater reliability rate is for agreements within 1 point of each other on the rating scale.)

# General Statistics

On the basis of our theoretical beliefs and preliminary analysis of the data, we created three summary variables for the Literacy Environment Checklist: the Books subtotal, the Writing subtotal, and the Total score. The Books subtotal includes all items from the Book Area, Book Selection, and Book Use sections of the checklist. The Writing subtotal includes all items from the Writing Materials and Writing Around the Room sections. Table A.1 reports descriptive statistics for Literacy Environment Checklist data gathered as part of the NEQRC and LEEP studies (n 255).

# Reliability Analysis

Reliability analysis was conducted to examine the internal consistency of the Literacy Environment Checklist. Table A.2 shows the alphas obtained for the Total score as well as for the two subtotals. Cronbach’s alpha of .84 for the Total score shows good internal consistency. All item–total correlations were moderate to high (r.54 to r.55).

| Composite variable | Mean | Standard deviation | Minimum | Maximum |
| --- | --- | --- | --- | --- |
| Books subscale | 11.13 | 3.90 | 2.00 | 20.00 |
| Writing subscale | 10.44 | 4.22 | 1.00 | 20.00 |
| Literacy Environment Checklist Total score | 21.57 | 7.37 | 5.00 | 40.00 |

# Measuring Stability and Change

Using the data collected from the LEEP classrooms, we reported preliminary findings on the ability of the Literacy Environment Checklist to measure both stability and change over time (see Table A.3). When one looks at mean scores across the 3 years of the LEEP project, the fall scores of the intervention are

| Composite variable | Fall |  | Spring |  |
| --- | --- | --- | --- | --- |
| Comparison group(n=38) | LEEP intervention(n=40) | Comparison group(n=38) | LEEP intervention(n=40) |  |
| Books subtotal | 9.25 | 10.53 | 10.45(t=3.27,p<.01) | 14.84(t=7.18,p<.001) |
| Writing subtotal | 8.77 | 11.21 | 9.12(t=1.14,p=n.s.) | 14.26(t=5.72,p<.001) |
| Literacy Environment Checklist Total score | 18.12 | 20.86 | 19.52(t=2.87,p<.01) | 29.03(t=7.82,p<.001) |

slightly higher on the three dimensions of the Literacy Environment Checklist than the comparison group. (For the fall scores in the LEEP study, differences between the intervention group and comparison group on the Literacy Environment Checklist were statistically significant for the Writing subtotal only (t –2.62, p.05). In the spring, the comparison group showed significant change on the Total score as well as on the Books subtotal yet remained stable on the Writing subtotal. As hoped, the intervention group scores changed significantly from fall to spring in all categories. These changes resulted in intervention group scores that were statistically significantly different from the comparison group scores in every category and statistically significantly different from the intervention group fall scores in every category.

# PSYCHOMETRIC PROPERTIES OF THE CLASSROOM OBSERVATION

Like the other parts of the ELLCO Toolkit, Research Edition, the Classroom Observation has been used for research for the NEQRC and LEEP. The Classroom Observation also has been used as a part of a school improvement project in the Philadelphia public school system in classrooms that range from kindergarten through grade 5. It has also been introduced to school systems in Connecticut and Maine. In these settings, it is being used both to collect data on and to provide a basis for discussions about classroom quality.
The psychometric properties presented in the sections that follow come from various analyses of data from Year 4 of the NEQRC research project combined with data collected from Years 1–3 of the LEEP project. Data from the NEQRC project were collected during the winter of 1998–1999 (n 29). The data from Year 1 of the LEEP project were collected in the fall of 1998 (n 27) and the spring of 1999 (n 27). Data from Year 2 of the LEEP project were collected in the fall of 1999 (n 42) and spring of 2000 (n 38). Data from Year 3 of the LEEP project were collected in the fall of 2000 and spring of 2001 in New England (fall: n 34; spring: n 37) and North Carolina (fall: n 37; spring: n 37). Together, the projects resulted in a total sample size of 308 classrooms, though the actual subscale size varies depending on the analyses conducted. As with the other parts of the ELLCO Toolkit, the data reported here for the Classroom Observation come from centers and classrooms in lower income communities.

Research use of the Classroom Observation was predicated on appropriate training of observers, as explained in the section of this appendix on the Literacy Environment Checklist. Novice observers’ initial observations were conducted with an experienced observer in order to ensure appropriate calibration to the rubrics in the Classroom Observation. When observers were trained and supervised appropriately, we consistently achieved interrater reliabilities of 90% and better for this part of the ELLCO Toolkit.

# General Statistics

On the basis of our theoretical beliefs and preliminary analyses of the data, we chose to create three summary variables for the Classroom Observation: the General Classroom Environment subtotal, the Language, Literacy, and Curriculum subtotal, and the Total score. One item (Item 3), Presence and Use of Technology, was problematic and was excluded from all summaries and analyses. Items included in the two subtotals in the ELLCO Toolkit, Research Edition, are as follows.

## General Classroom Environment subtotal:

1. Organization of the Classroom
2. Contents of the Classroom
4. Opportunities for Child Choice and Initiative
5. Classroom Management Strategies
6. Classroom Climate

## Language, Literacy, and Curriculum subtotal:

7. Oral Language Facilitation
8. Presence of Books 9P. Approaches to Book Reading (Prekindergarten and Kindergarten Version)
10P. Approaches to Children’s Writing (Prekindergarten and Kindergarten Version)
11. Approaches to Curriculum Integration
12. Recognizing Diversity in the Classroom
13. Facilitating Home Support for Literacy
14. Approaches to Assessment 1

By problematic, we mean that scores for Presence and Use of Technology did not cluster with scores for the other items, suggesting that effective use of technology reflects capabilities somewhat distinct from those captured by the other items in the Classroom Observation. In addition, the scores for Presence and Use of Technology did not relate clearly to the General Classroom Environment subtotal or to the Language, Literacy, and Curriculum subtotal; therefore, it was not included in the reported averages or calculations of Cronbach’s alpha for the total tool. For this item in 308 classrooms, the mean was 2.45, with a standard deviation of 1.09 and a minimum of 1.0 and a maximum of 5.0.

| Composite variable | Mean | Standard deviation | Minimum | Maximum |
| --- | --- | --- | --- | --- |
| General Classroom Environment subtotal | 3.44 | 0.79 | 1.20 | 5.00 |
| Language, Literacy, and Curriculum subtotal | 3.02 | 0.75 | 1.13 | 5.00 |
| Classroom Observation Total score | 3.15 | 0.71 | 1.29 | 5.00 |

Using these subtotals, we obtained data from classrooms throughout New England that provide some indication of observed levels of performance in classrooms that serve low-income children. As with the Literacy Environment Checklist, many of the classrooms included were in Head Start programs. Tables A.4 and A.5 report descriptive statistics for the Classroom Observation data gathered as part of the NEQRC and LEEP studies (n 308).

# Reliability Analysis

Reliability analysis was conducted to examine the internal consistency of the Classroom Observation using data from 308 classrooms.

| Composite variable | Alpha |
| --- | --- |
| General Classroom Environment subtotal | .83 |
| Language, Literacy, and Curriculum subtotal | .86 |
| Classroom Observation Total score | .90 |

Cronbach’s alpha of .83 for the General Classroom Environment shows good internal consistency for this composite. All of the item–total correlations were high—with correlation coefficients ranging from .60 for Item 1, Organization of the Classroom, to .75 for Item 6, Classroom Climate—with the exception of Item 2, Contents of the Classroom. This item had the lowest item–total correlation, which was nonetheless a moderate correlation (r.53). The internal consistency of the Language, Literacy, and Curriculum composite is very good, with an alpha of .86. All of the item–total correlations were moderate to high, ranging from .55 for Item 8, Presence of Books, to .65 for Item 13, Facilitating Home Support for Literacy. Cronbach’s alpha of .90 also shows very good internal consistency for all items combined on the Classroom Observation. All of the item–total correlations for the Classroom Observation Total were moderate to high (r.39 to r.68).

# Measuring Stability and Change

Again, in the LEEP project, classrooms were observed in the fall and in the spring of Years 1–3, yielding the ability to measure change over time using the Classroom Observation. Some of the teachers were taking a yearlong course that focused on early language and literacy (our intervention group); the remaining teachers were not (our comparison group). Using data from the comparison group classrooms, we have data on the ability of the Classroom Observation to measure both stability and change over time.

| Composite variable | Fall |  | Spring |  |
| --- | --- | --- | --- | --- |
| Comparison group(n=65) | LEEP intervention(n=42) | Comparison group(n=65) | LEEP intervention(n=42) |  |
| General Classroom Environment subtotal | 3.26 | 3.61 | 3.42(t=1.96,p=n.s.) | 3.91(t=2.26,p<.05) |
| Language, Literacy, and Curriculum subtotal | 2.85 | 3.01 | 2.93(t=1.13,p=n.s.) | 3.75(t=5.50,p<.0001) |
| Classroom Observation Total score | 2.97 | 3.19 | 3.08(t=1.53,p=n.s.) | 3.74(t=4.88,p<.0001) |

n.s., not significant.

# Correlation with Another Widely Used Measure

From our comparison group data, we were able to conclude that the Classroom Observation is able to capture stability in classroom quality. This is a good indicator of the Classroom Observation’s test–retest reliability. Our data also show that the Classroom Observation is able to capture changes in classroom quality associated with a literacy-focused intervention. These findings come from two sources: evidence of fall-to-spring growth and differences between the intervention and comparison groups. These data provide evidence of the instructional sensitivity of this tool. The concept of instructional sensitivity is an important factor in determining the quality of research instruments. Our data suggest that the Classroom Observation is both stable and sensitive to interventions that target literacy in ways that are consistent with its assumptions about what constitutes appropriate early literacy practices.

# PSYCHOMETRIC PROPERTIES OF THE LITERACY ACTIVITIES RATING SCALE

Like the Classroom Observation and the Literacy Environment Checklist, the Literacy Activities Rating Scale of the ELLCO Toolkit, Research Edition, has been used to conduct research as part of the NEQRC and LEEP, and the data presented here are from centers and classrooms in lower income communities. The psychometric properties presented for the rating scale are based on data from Year 4 of the NEQRC project, combined with data from Years 1–3 of the LEEP project. Data from the NEQRC project were collected during the winter of 1998–1999 (n 30). The data from Year 1 of the LEEP project were collected in the fall of 1998 (n 28) and the spring of 1999 (n 28). Data from Year 2 of the LEEP project were collected in the fall of 1999 (n 42) and spring of 2000 (n 40). Data from Year 3 of the LEEP project were collected in the fall of 2000 (n 47) and spring of 2001 (n 47). Together, the projects resulted in a total sample size of 262, although actual subsample size varies depending on the analyses conducted. As with the Literacy Environment Checklist, the rating scale has been used for research only in preschool classrooms, many of which are in Head Start programs. Some items from the Literacy Activities Rating Scale have been incorporated into the ELLCO Pre-K (see Table A.15 later in this appendix), but there is no Literacy Activities Rating Scale in the ELLCO Pre-K.

# Interrater Reliability

Observers underwent a training process explained in the section of this appendix that describes interrater reliability for the Literacy Environment Checklist and the Classroom Observation. As with those parts of the ELLCO Toolkit, Research Edition, novice observers’ initial observations are conducted with an experienced observer. Because the Literacy Activities Rating Scale was used simply to describe activities observed during the classroom visit, we did not maintain formal records of interrater reliability. Observers who visited classrooms together, however, had little difficulty arriving at the same ratings for the classrooms they visited. When observers were trained and supervised appropriately, we achieved an average interrater reliability of 81% with relative ease.

| Composite variable | Mean | Standard deviation | Minimum | Maximum |
| --- | --- | --- | --- | --- |
| Full-Group Book Reading subtotal | 2.86 | 1.95 | 0 | 6.00 |
| Writing subtotal | 2.10 | 1.39 | 0 | 5.00 |
| Literacy Activities Rating Scale Total score | 5.80 | 2.63 | 0 | 13.00 |

# Reliability Analysis

Reliability analysis was conducted to examine the internal consistency of the Literacy Activities Rating Scale.

| Composite variable | Alpha |
| --- | --- |
| Full-Group Book Reading subtotal | .92 |
| Writing subtotal | .73 |
| Literacy Activities Rating Scale Total score | .66 |

The Full-Group Book Reading alpha of .90 shows very good internal consistency for this subtotal. The Writing subtotal and Literacy Activity Rating Scale Total score both show good internal consistency with alphas of .74 and .72, respectively. Item–total correlations for the Full-Group Book Reading subscale were high, ranging from .75 to .85. The items that compose the Writing subtotal showed moderate to high item–total correlations of .47 to .64. The Literacy Activities Rating Scale Total score also had moderate to high item–total correlations that ranged from .30 to .56. On the basis of the psychometric properties of the ELLCO Toolkit, Research Edition, as well as the theoretical and practical considerations outlined in Chapter 1, we revised the ELLCO to include more specificity, in the form of detailed descriptive indicators for each scale point, as well as a broader range of measures of quality in early literacy, such as phonological awareness, efforts to build vocabulary, opportunities for extended conversations, and environmental print. Although the ELLCO Pre-K is more thorough and expansive than the Research Edition, it does include the same content covered by the earlier version, with the exception of Research Edition Item 3: Presence and Use of Technology. Cronbach’s alpha analyses on the ELLCO Toolkit, Research Edition, described earlier in this technical appendix, indicated that this construct was not statistically related to the other items; therefore, it was not included in the ELLCO Pre-K.