{"id":271,"date":"2012-01-13T16:14:26","date_gmt":"2012-01-13T22:14:26","guid":{"rendered":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/?page_id=271"},"modified":"2014-07-21T15:16:04","modified_gmt":"2014-07-21T21:16:04","slug":"researchdesign","status":"publish","type":"page","link":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/homepage\/researchdesign\/","title":{"rendered":"Research Design, Methods, and Measures"},"content":{"rendered":"

Design and Randomization:<\/h2>\n

Theory of Change<\/span><\/h4>\n

In order to benefit from preschool and master the literacy, math, and social skills they will need to participate in formal schooling, young children must be able to think, attend, and remember in an intentional and purposeful way.\u00a0Tools of the Mind <\/em>is designed to initiate and support the change from reactivity to active and focused thinking. This change is\u00a0facilitated through specific teacher practices that should help children move from assisted to independent learning. Tools of the Mind is based on an interactive sequence of change\u00a0whereby teachers use assessment and scaffolding to tailor their use and modeling of specific tactics. These tactics are internalized by their students as cognitive tools, which are then used independently and manifested in observable behaviors in the classroom.\u00a0These learning-related self-regulatory skills then become the foundation that supports the learning of the preschool \u201ctarget skills\u201d and the academic content that follows preschool. Below is a graphical representation of the Theory of Change model used to guide the selection of child assessment and classroom observation measures used in this study.<\/p>\n

\"\"<\/a><\/p>\n

Click to view a pdf of the Theory of Change<\/a><\/p>\n

Study Design<\/span><\/h4>\n

The research sample from the 2010-11 school year was drawn from five school districts representing a range of urban, rural and suburban locations and serving demographically diverse children in two states (60 classrooms, 32 implementing Tools<\/em>). A sixth school district provided a 2nd cohort during the 2011-12 school year (20 classrooms, 10 implementing Tools<\/em>). <\/span>Experimental condition was randomized at the school level. This scheme helped minimize interactions between experimental and control teachers that might compromise the comparison. The schools were blocked by district, with the largest district divided into two blocks. Within each block, approximately half the schools were assigned to the Tools<\/em> condition and half to the practice as usual control condition.\u00a0The teachers in the classrooms assigned to the control condition continued to practice as usual with whatever curriculum they were using and whatever professional development was typical in their setting. The teachers in the Tools<\/em> condition\u00a0began the professional development sequence for Tools<\/em> and\u00a0implementation of\u00a0the curriculum during the first year of the study (school year 2009-10 for cohort 1 and school year 2010-11 for cohort 2). The first year was a training and practice year for the teachers and no study data was collected. The research team used this period to develop and practice the classroom observational scheme for collecting fidelity data on the Tools<\/em> curriculum. Evaluation of the Tools<\/em> curriculum took place the following school year. Replacement teachers for any of the original teachers that left were trained and their classrooms remained in the study. During the third year of the study (2011-12)<\/span>, training on Tools<\/em> was offered to each school system in cohort 1\u00a0for\u00a0the control teachers\u00a0with 3 of the 5 systems choosing to participate in this opportunity. The following year, training on Tools <\/em>was offered to cohort 2 for the control teachers and this system choose to participate in this opportunity.<\/p>\n

Child outcome measures were collected at the beginning and end of the pre-k year, at the end of kindergarten, and at the end of first grade.<\/span><\/span><\/p>\n

Randomization <\/span><\/h4>\n

Random assignment was successful at producing condition groups that did not differ significantly on any demographic variable\u00a0(ethnicity, gender, IEP status, ELL status, age, teacher education level, years teaching, etc.)\u00a0as well as entering self regulation and academic achievement scores for cohort 1.<\/p>\n

Participants:<\/span><\/h2>\n

Children<\/span><\/h4>\n

During the 2010-11 school year, 828 children (Tools <\/em>= 477) were seen at the beginning of pre-k and 821 children (Tools<\/em> = 472) at the end of pre-k. The table below provides demographic information about the participant children from cohort 1.<\/p>\n\n\n\n\n
\"\"<\/a><\/td>\n<\/td>\n<\/tr>\n
\n
Note: Cohort 1 participants are from Franklin Special School District, Lebanon Special School District, Wilson County School District, and Cannon County School District in Tennessee as well as Guilford County School System in North Carolina.<\/address>\n<\/td>\n
 <\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/address>\n
<\/address>\n
<\/address>\n

During the 2011-12 school year, 267 children (Tools=147) were seen at the beginning of pre-k and 257 children (Tools=142) at the end of pre-k. The table below provides demographic information about the participant children from cohort 2.<\/span><\/span><\/p>\n\n\n\n\n
\"\"<\/a><\/td>\n<\/td>\n<\/tr>\n
\n
Note: Cohort 2 participants\u00a0are from Alamance-Burlington School System in North Carolina.<\/address>\n<\/td>\n
<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Teachers<\/span><\/h4>\n

During the 2010-11 school year,\u00a060 classrooms (Tools<\/em> = 32) in 45 schools (Tools<\/em> = 25) participated in the study. The table below provides demographic information about the participating classroom teachers from cohort 1.<\/span><\/span><\/p>\n\n\n\n
\"\"<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

During the 2011-12 school year,\u00a020 classrooms (Tools<\/em> = 10) in\u00a012 schools (Tools<\/em> = 5) participated in the study. The table below provides demographic information about the participating classroom teachers from cohort 2.<\/p>\n\n\n\n
\"\"<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Procedures:<\/h2>\n
\n
\n

Overall Data Collection<\/span><\/h4>\n

All child assessment and classroom observational data for this project were collected using a paperless tablet data collection system.\u00a0 This system allowed assessors to record children’s responses on the direct assessment measures while administering the items.\u00a0 The system had built in checks to notify assessors instantly of missed items, stop rules, ceiling errors, etc.\u00a0 Similarly, the system for classroom observation\u00a0measures featured built in access to manuals and allowed observers to\u00a0navigate quickly to specific portions of the measure depending on what was occurring in the classroom at any given moment.\u00a0 Assessors\/Observers saved their data onto thumb drives and the data was then imported into\u00a0a\u00a0main database.\u00a0 This paperless system eliminated the need for data entry and double entry\u00a0which allowed for quicker access to the dataset at the conclusion of the assessment and observation\u00a0cycles.<\/p>\n

Click to view\u00a0more information\u00a0regarding the Paperless Data Collection System <\/a><\/p>\n

Direct Assessments<\/span> <\/span><\/h4>\n

Direct assessments with the participant children were administered in two sessions with each session lasting approximately 25-35 minutes.\u00a0\u00a0The assessments were administered to the participant children\u00a0individually inside a quiet room within their school.<\/p>\n

A team of 16 assessors\u00a0was utilized to complete the collection of the child level data during the 2010-11 school year, 24 during the 2011-12 school year, 26\u00a0during the 2012-13 school year, and 5 during the 2013-14 school year.<\/span> Most assessment staff were trained to administer one session of the assessment battery with 3 trained in both sessions.\u00a0\u00a0Assessors participated in formal in office training sessions to learn the measures and were required to practice with each other as well as\u00a0in office staff prior to administering the session to a consented non-study child at a\u00a0non-study school.\u00a0 Assessors were required to submit a video of\u00a0their session\u00a0with a consented non-study child in order to become certified in the administration of that assessment session.\u00a0 Videos were scored using a scoring rubric and assessors had to obtain a minimum score of 85 in order to become certified.<\/p>\n

Click to view an example of the Assessment Video Scoring Rubric<\/a><\/p>\n

To measure educational achievement, the Woodcock-Johnson III Tests of Achievement (WJ-III) was used.\u00a0 To measure self regulatory skills, a battery of self regulation measures was used.<\/p>\n

\n
Session 1 (in order as given):<\/h5>\n

Navy Blue<\/span><\/strong> text denotes SR measures, Black<\/strong> text denotes WJ-III subtests<\/p>\n

    \n
  1. Peg Tap<\/span><\/li>\n
  2. Head Toes Knees Shoulders<\/span><\/li>\n
  3. Copy Design<\/span><\/li>\n
  4. Oral Comprehension<\/span><\/li>\n
  5. Applied Problems<\/span><\/li>\n
  6. Quantitative Concepts A & B<\/span><\/li>\n
  7. Picture Vocabulary<\/span><\/li>\n
  8. Passage Comprehension (only administered during the 1st grade followup)<\/span><\/li>\n<\/ol>\n
    Session 2:<\/h5>\n
      \n
    1. Dimensional Change Card Sort<\/span><\/li>\n
    2. Corsi Blocks<\/span><\/li>\n
    3. Letter Word<\/li>\n
    4. Academic Knowledge A, B, & C<\/li>\n
    5. Spelling<\/li>\n<\/ol>\n

      Classroom Observations<\/span><\/h4>\n

      Participating pre-k classrooms were observed three times during the 2010-11 (cohort 1) and 2011-12 (cohort 2)\u00a0school years. <\/span>During each classroom observation,\u00a0two observers were present.\u00a0 One observer completed the Narrative Record form, Tools of the Mind Fidelity, and Environmental Scan.\u00a0 The other observer completed the Teacher Observation in Preschool (TOP)\u00a0and Child Observation in Preschool (COP).\u00a0 Both observers met together immediately following the observation to complete the Post Observation Rating Scale (PRS).<\/p>\n

      Classroom observers were trained in one set of the classroom observations\u00a0(either the COP and TOP or the Narrative Record, Fidelity, and Environmental Scan)\u00a0by attending in-office training over multiple days.\u00a0Observers were then required to observe in\u00a0non-study classrooms with experienced project staff to solidify the material discussed in training and ensure\u00a0coding reliability.\u00a0 Once reliable in a non-study classroom, observers were certified to observe in study classrooms.\u00a0 At the beginning of each\u00a0of the three observation cycles, reliability was established among all observers in the field.<\/p>\n

      Cohort 1, 2010-11 Observation Schedule:<\/h4>\n

      Observation 1 Window:<\/h5>\n

      TN (N=30):\u00a0 10\/11\/10 through 12\/9\/10<\/p>\n

      NC (N=30):\u00a0 11\/8\/10 through 12\/14\/2010<\/p>\n

      Observation 2 Window:<\/h5>\n

      TN (N=30):\u00a0\u00a0 1\/18\/2011 through 2\/18\/2011<\/p>\n

      NC (N=30):\u00a0 1\/19\/2011 through 2\/25\/2011<\/p>\n

      Observation\u00a03 Window:<\/h5>\n

      TN (N=30):\u00a0\u00a0 3\/8\/2011 through 4\/14\/2011<\/p>\n

      NC (N=30):\u00a0 3\/8\/2011 through 4\/14\/2011<\/p>\n

      Cohort 2, 2011-12 Observation Schedule:<\/h4>\n

      Observation 1 Window:<\/h5>\n

      NC (N=20):\u00a0 10\/25\/2011 through 12\/2\/2011<\/span><\/p>\n

      Observation 2 Window:<\/h5>\n

      NC (N=20):\u00a0\u00a0 1\/20\/2012 through 2\/6\/2012<\/span><\/p>\n

      Observation\u00a03 Window:<\/h5>\n

      NC (N=20):\u00a0\u00a0 3\/13\/2012 through 3\/29\/2012<\/span><\/p><\/blockquote>\n<\/blockquote>\n<\/blockquote>\n

      Measures:<\/h2>\n

      Academic Achievement\u00a0Assessments<\/span><\/h4>\n
      Woodcock Johnson III Tests of Achievement (WJ-III):<\/strong><\/h5>\n

      Woodcock Johnson III Tests of Achievement (Woodcock, McCrew, & Mather, 2001).\u00a0 The WJ-III is a standardized measure with established reliability and validity and applicability to a wide age range, beginning at age 3. The selected scales for language\/literacy focus on emergent literacy and span the areas of decoding letters and words, vocabulary, and comprehension. The selected measures for math\/numeracy span number recognition, simple problem solving, and simple math concepts.<\/p>\n

      WJ subtests used in this study:\u00a0 Letter Word, Spelling, Picture Vocabulary, Oral Comprehension, Applied Problems, Quantitative Concepts, Academic Knowledge, and *Passage Comprehension (*1st gr only)<\/p>\n

      Self Regulation Assessments<\/span><\/h4>\n
      Peg\u00a0Tapping:<\/h5>\n

      Peg Tapping (Diamond & Taylor, 1996). In this game, children are asked to tap a peg twice when the experimenter taps once and vice versa. The task requires children to inhibit a natural tendency to mimic the experimenter while remembering the rule for the correct response. Sixteen trials are\u00a0conducted with 8 one-tap and 8 two-tap trials in random sequence.<\/p>\n

      Click to view more information regarding the Peg Tapping task<\/a><\/p>\n

      Head Toes Knees Shoulders (HTKS):<\/h5>\n

      Head-to-Toes Task (Cameron et al., in press). In this task, which requires inhibitory control, attention, and working memory (though inhibitory control is the main focus) children are asked to play a game in which they must do the opposite of what the experimenter says. The experimenter instructs children to touch their head (or their toes), but instead of following the command, the children are supposed to do the opposite and touch their toes. The measure is available from Claire Cameron, who gave permission for us to use the measure for this project.<\/p>\n

      Click to view more information regarding the HTKS task<\/a><\/p>\n

      Copy Design:<\/h5>\n

      Copying Designs Test (Osborn et al., 1984). Children must copy \u201cexactly\u201d a series of simple geometric designs. Error scores are used to assess attention.<\/p>\n

      Click to view more information regarding the Copy Design task<\/a><\/p>\n

      Dimensional Change Card Sort (DCCS):<\/h5>\n

      Dimensional Change Card Sort (Zelazo, 2006). In this reverse categorization task, children must sort a set of cards based on different sorting criteria given by the examiner. For example, the first sort involves color, the second sort involves shape, and the final sort is a mix of color and shape depending on whether a card has a border or not. An additional level was added during the first grade\u00a0follow up\u00a0that involved reversing the rules on the final sort listed above.\u00a0This task is used to assess attention-shifting.<\/p>\n

      Click to view more information regarding the DCCS task<\/a><\/p>\n

      Corsi Blocks:<\/h5>\n

      Corsi Blocks Task (Corsi, 1972). The Corsi Blocks task is a visuospatial short-term memory task. Children point to a series of blocks in a pattern both forward and backward. The task has two forward series of patterns and two backward series of patterns. The length of the block pattern increases until the recalled pattern is no longer correct.<\/p>\n

      Click to view more information regarding the Corsi Block task<\/a><\/p>\n

      Classroom Observations<\/span><\/h4>\n
      Narrative Record:<\/h5>\n

      Narrative Record (Bilbrey &\u00a0Farran, 2004).\u00a0 The Narrative Record Form is an open-ended format for recording narrative data notes and rating the activities occurring in the classroom as a whole.\u00a0The Narrative Record is an accounting of the content, activity type, duration, engagement, and instructional level of each segment over the full observational period.<\/p>\n

      Click to view more\u00a0information regarding the Narrative Record<\/a><\/p>\n

      Tools of the Mind Curriculum Fidelity:<\/h5>\n

      Tools of the Mind Curriculum Fidelity (Vorhaus & Meador, 2010).\u00a0 The Tools Fidelity captures the specific Tools<\/em> curriculum activities that occur within a classroom observation period along with information about the specific implementation steps that occur, and mediators that are used.\u00a0 In addition, the curriculum developers furnished a list of behaviors that \u201cshould not\u201d happen during each activity that are also captured by observers. The Tools<\/em> Fidelity Measure provides an in-depth look at the degree of curriculum implementation across the year within experimental classrooms. \u00a0Although this instrument was used in both Tools<\/em> and comparison classrooms, relatively few Tools<\/em> activities were ever coded in comparison rooms.<\/p>\n

      Click to view more information regarding the Tools of the Mind Fidelity measure<\/a><\/p>\n

      Environmental Scan:<\/h5>\n

      Environmental Scan (Vorhaus, Meador, & Farran, 2010).\u00a0 The Environmental Scan is an observational tool to gauge a classroom\u2019s environment and materials. It is derived from a list of early childhood materials the Tools of the Mind<\/em> developers indicate should be available in the classroom. The scan focuses on the play centers and materials accessible to children.<\/p>\n

      Click to view more information regarding the Environmental Scan<\/a><\/p>\n

      POST Observation Rating Scale (PRS):<\/h5>\n

      Post Observation Rating Scale (Yun, Farran, Lipsey, Vorhaus, & Meador, 2010).\u00a0 The PRS is completed immediately after a classroom is observed and is a 5-point Likert-type researcher-developed scale for rating classroom-level characteristics. This instrument was developed following extensive discussions with the Tools of the Mind<\/em> developers during which they identified classroom attributes that were most likely to be different between Tools<\/em> classrooms and other early childhood classrooms. The PRS includes items regarding general classroom characteristics as well as teacher practices, classroom activities, and children\u2019s social and academic behaviors. \u00a0Both observers complete the PRS together following the visit.<\/p>\n

      Click to view more information regarding the PRS<\/a><\/p>\n

      Teacher Observation in Preschool (TOP):<\/h5>\n

      Teacher Observation in Preschool (Bilbrey,Vorhaus, & Farran, 2007).\u00a0 The TOP is a system for observing teacher\/assistant\u2019s behavior in preschool classrooms and is based on a series of snapshots of the teacher\u2019s and assistant\u2019s behavior across a period of time.\u00a0 Each snapshot may be by itself an unreliable piece of information but collectively they combine to provide a picture of how the teacher and assistant are spending their time in a classroom. The teacher\u2019s behavior is observed for a 3 second window before scoring. Once scoring has been completed for the teacher, the same procedure is followed for the assistant in the classroom. Teacher and assistant are coded at the beginning of a \u201csweep\u201d\u00a0with\u00a0children\u00a0coded immediately afterward. At the end of an observation, 20 sweeps have been collected on the teacher and the assistant.<\/p>\n

      Click to view more information regarding the TOP<\/a><\/p>\n

      Child Observation in Preschool (COP):<\/h5>\n

      Child Observation in Preschool (Farran et al. (2006, 2008 revision).\u00a0The COP is a system for observing children\u2019s behavior in preschool classrooms and is based on a series of snapshots of children\u2019s behavior across a period of time.\u00a0Each snapshot may be by itself an unreliable piece of information but collectively they combine to provide a picture of how children are spending their time in a classroom (as an aggregate) as well as information about individual differences among children in their preferences. A specific child is observed during a 3 second window and then coded across 9 dimensions before the observer moves to the next child. At the end of an observation, 20 sweeps were collected on each child in the classroom. Consented children are indentified by name; all others are identified as \u201cExtra boy\u201d or \u201cExtra girl.\u201d<\/p>\n

      Click to view more information regarding the COP<\/a><\/p>\n

      Child Report Forms (completed by each teacher)<\/span><\/h4>\n

      Teachers were asked to complete a child report form on each of their consented children during fall and spring of the prekindergarten year, spring of the kindergarten year, and spring of the first grade year (cohort 1 only).<\/p>\n

      Cooper-Farran Behavior Rating Scales (CFBRS):<\/h5>\n

      Cooper-Farran Behavior Rating Scales (Cooper & Farran, 1991).\u00a0 The Cooper-Farran is composed of 37 items in two subscales. The Interpersonal Skills subscale (IPS) includes 21 items and the Work-Related Skills (WRS) subscale includes 16 items. The IPS subscale measures how well children get along with peers and the teacher. The WRS subscale includes items about independent work, compliance with instructions, and memory for instructions. Items are rated on a 1-7 scale with descriptive phrases to \u201canchor\u201d points 1, 3, 5, and 7.<\/p>\n

      Click to view more information regarding the CFBRS<\/a><\/p>\n

      Adaptive Language Inventory (ALI):<\/h5>\n

      Adaptive Language Inventory (Feagans & Farran, 1983; Feagans, Fendt, & Farran, 1995).\u00a0 The ALI focuses on children\u2019s comprehension and use of language in classroom settings in comparison to their peers and has been used both at the preschool and elementary levels. The measure consists of 18 items that focus on comprehension, production, rephrasing, spontaneity, listening, and fluency. Children are rated on a 1-5 scale.<\/p>\n

      Click to view more information regarding the ALI<\/a><\/p>\n

      Academic Classroom Behavior Record (ACBR):<\/strong><\/p>\n

      Academic Classroom Behavior Record (Farran, Bilbrey & Lipsey, 2003).\u00a0 The Academic Classroom Behavior Record is composed of 9 items.\u00a0 This scale includes items about school readiness, liking of school, and behavior problems.\u00a0 The first six items are rated on a 1-7 scale with descriptive phrases to \u201canchor\u201d points 1,3,5,7.\u00a0 Items 7 and 9 are multiple choice items.\u00a0 Item 8 is an item in which the rater checks all answers that apply. This instrument\u00a0was only used with participating kindergarten and first grade teachers.<\/p>\n

      Click to view more information regarding the ACBR<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

      Design and Randomization: Theory of Change In order to benefit from preschool and master the literacy, math, and social skills they will need to participate in formal schooling, young children must be able to think, attend, and remember in an intentional and purposeful way.\u00a0Tools of the Mind is designed to initiate and support the change…<\/p>\n","protected":false},"author":549,"featured_media":0,"parent":3,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"tags":[],"class_list":["post-271","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/pages\/271","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/users\/549"}],"replies":[{"embeddable":true,"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/comments?post=271"}],"version-history":[{"count":201,"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/pages\/271\/revisions"}],"predecessor-version":[{"id":1227,"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/pages\/271\/revisions\/1227"}],"up":[{"embeddable":true,"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/pages\/3"}],"wp:attachment":[{"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/media?parent=271"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/my.dev.vanderbilt.edu\/toolsofthemindevaluation\/wp-json\/wp\/v2\/tags?post=271"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}