{"id":940,"date":"2019-11-04T00:17:17","date_gmt":"2019-11-04T05:17:17","guid":{"rendered":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/?page_id=940"},"modified":"2019-11-04T18:31:06","modified_gmt":"2019-11-04T23:31:06","slug":"quality-matters-research-design-magnitude-and-effect-sizes","status":"publish","type":"page","link":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/quality-matters-research-design-magnitude-and-effect-sizes\/","title":{"rendered":"Quality Matters – Research Design, Magnitude and Effect Sizes"},"content":{"rendered":"

\"Quality<\/p>\n

 <\/p>\n

The keystone of evidence-based practices (EPBs) is research.\u00a0 Through research, we are able to create a solid foundation for which an EPB can stand on.\u00a0 However, not all research is created equal.\u00a0 There are studies that are the cornerstone for other researchers, studies of such high-quality that they can be cited and reference with not qualms.\u00a0 Then there is research that is high-quality but may not meet all of the criteria for a EBP.\u00a0 Then there are studies that are missing key elements of a quality study and while they do show effects, those effects may be impacted by reliability. validity or implementation problems.\u00a0 Then there are the studies that are not high-quality.\u00a0 These studies leave many of the essential elements of a high-quality out and therefore the results cannot be validated.<\/p>\n

 <\/p>\n

\"This<\/p>\n

 <\/p>\n

The Council for Exceptional Children has developed quality indicators to help identify studies that examine interventions and their effects on children with disabilities.\u00a0 These indicators look to examine if a study is methodologically sound.\u00a0 This is another term used to describe the quality of research.<\/p>\n

To access The Council for Exceptional Children’s Research Quality Indicators, click here.\u00a0<\/a><\/p>\n

 <\/p>\n

Here is a simplified chart of the CEC quality indicators.\u00a0 We will revisit these indicators later to see what is required of studies to impact the status of evidence-based practices.<\/p>\n

 <\/p>\n

CEC Quality Indicators<\/h2>\n(Adapted from the Council for Exceptional Children Standards for Evidence-Based Practices in Special Education, 2014<\/span>\n\n\n\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
Indicator:<\/td>Indicator:<\/td>Additional Information:<\/td>\n<\/tr>\n
1.0 \u2013 Context and Setting<\/td>The study provides sufficient information regarding the critical features of the context or setting <\/td>School information, curriculum, geographical location, socioeconomic status, physical layout<\/td>\n<\/tr>\n
2.0 \u2013 Participants <\/td>The study provides sufficient information to identify the population of participants to which results may be generalized and to determine or confirm whether the participants demonstrated the disability or difficulty of focus.<\/td>Gender, age, race\/ethnicity, socioeconomic status, language status, etc.
\n
\nDisability status and how determination for disability status was obtained
\n<\/td>\n<\/tr>\n
3.0 \u2013 Intervention Agent<\/td>The study provides sufficient information regarding the critical features of the intervention agent.<\/td>Who is providing the intervention (teacher, researcher, etc.) and the information about the provider (age, educational background, etc.)
\n
\nHow much training was provided to the those delivering intervention and what are the qualifications needed to implement intervention
\n<\/td>\n<\/tr>\n
4.0 \u2013 Description of Practice <\/td>The study provides sufficient information regarding the critical features of the practice (intervention), such that the practice is clearly understood and can be reasonably replicated.<\/td>Describes intervention procedures, materials, dosage, prompts, proximity, etc. <\/td>\n<\/tr>\n
5.0 \u2013 Implementation Fidelity <\/td>The practice is implemented with fidelity.<\/td>Is there information about implementation fidelity (checklist, etc.) how often was fidelity looked at and how was it reported, was fidelity checked throughout the entire study, interventionist, participant, etc. <\/td>\n<\/tr>\n
6.0 \u2013 Internal Validity <\/td>The independent variable is under the control of experimenter. The study describes the services provided in control and comparison conditions and phases. The research design provides sufficient evidence that the independent variable causes change in the dependent variable or variables. Participants stayed with the study, so attrition is not a significant threat to internal validity.<\/td>Independent variable is manipulated with descriptions of baseline, control groups, interventions (duration, length, etc.)
\n
\nDescribes I there was randomization and if no randomization was done, why and within what limits
\n
\nHow many experimental effects were reported?
\n
\nSingle subject \u2013 how many baseline points and descriptions on performance
\n
\nThreats to internal validity and how these apply to single-subject designs
\n
\nWas there attrition \u2013 did they lose any participants from the beginning to the end of the study
\n
\nIs attrition controlled for?
\n<\/td>\n<\/tr>\n
7.0 \u2013 Outcomes Measures\/Dependent Variables <\/td>Outcome measures are applied appropriately to gauge the effect of the practice on study outcomes. Outcome measures demonstrate adequate psychometrics.<\/td>Outcomes are socially important
\n
\nDescriptions of measurement of dependent variable
\n
\nReporting of p values and effect sizes
\n
\nFrequency and outcome of measures \u2013 for single subject three points per phase change
\n
\nThere is reliability information among observers
\n
\nProvides information about validity \u2013 content, social, criterion, etc.
\n<\/td>\n<\/tr>\n
8.0 \u2013 Data Analysis <\/td>Data analysis is conducted appropriately. The study reports information on effect size.<\/td>Statistical techniques for comparison are reported and more that one effect size statistic is reported <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

 <\/p>\n

\"Evidence-Based<\/p>\n

(Please note that all graphs and infographics are original.)<\/p>\n

 <\/p>\n

 <\/p>\n

Experimental Designs<\/strong><\/p>\n

 <\/p>\n

One of the key quality feature of\u00a0EBP research is that studies employ an experimental design.\u00a0 An experimental design looks to compare results from two different groups to determine if an intervention was effective or not.\u00a0 The participants in these designs are either assigned to the treatment groups (getting the intervention) or to the control group (business as usual).\u00a0 Business as usual refers to students receiving their normal instruction in the classroom with no change applied.\u00a0 All of the participants have the same key features, for example all the participants are in the 2nd grade and have been identified as having a math disability.\u00a0 The participants are given the same pre-test assessments and the same post-test assessments on the same time schedule.\u00a0 Since one group received the treatment and one did not, the effects that the intervention has on participants can confidently be determined for the interventions effectiveness.\u00a0 Experimental designs are also quantitative which means the data can be measured and interpreted through different statistical manipulations.<\/p>\n

 <\/p>\n

\"Infographic<\/p>\n

 <\/p>\n

Quasi-Experimental Designs<\/strong><\/p>\n

 <\/p>\n

There are studies that are quasi-experimental.\u00a0 These studies do not have randomization of their groups.\u00a0 They still have the same participants as pre and post assessments but their is no “business as usual” group.\u00a0 These studies look to see which of two interventions were more impactful on student outcomes.\u00a0 \u00a0For example, a group of 3rd graders may be receiving two different reading interventions.\u00a0 One looks at phonics instruction only and the other at fluency only.\u00a0 The two groups receive the same intervention over the same number of weeks and at the end of intervention treatment, post tests are given.\u00a0 Based off the results, researchers can determine which intervention was more impactful.<\/p>\n

 <\/p>\n

\"Infographic<\/p>\n

 <\/p>\n

Single-Subject Designs<\/strong><\/p>\n

Single-subject designs are just as the name suggests…a single subject.\u00a0 This design type looks to examine the effects of an intervention on a single subject but the intervention could be looking at its effects on different behaviors or skills.\u00a0 You could have the same intervention (single-subject) across participants which looks at the interventions effects across participants of the same age, level or skill set.\u00a0 You could also have a single-subject designs that looks at alternating treatments and their effects on a participant.\u00a0 All of these different single subject designs demonstrate a interventions effects.\u00a0 Note that single-case design data looks most like data that teachers collect in their own classrooms.<\/p>\n

 <\/p>\n

\"Example<\/p>\n

 <\/p>\n

 <\/p>\n

Qualitative Designs\u00a0<\/strong><\/p>\n

Qualitative designs are different from quantitative in that the data is not measurable.\u00a0 Qualitative studies involve things like interviews, surveys and anecdotal information.\u00a0 These studies look at things like teachers’ perceptions of the learning environment, survey information about why a teacher left the classroom or descriptive information like effective teachers have these qualities.\u00a0 While the information from qualitative studies can be used for determining an evidence-based practice, they cannot be the only type of research in the body of evidence.\u00a0 This is because in these studies, there is no measurable data to compare and intervention and it’s effects on students or teachers.<\/p>\n

 <\/p>\n

\"Example<\/p>\n

 
\n\"Effect<\/p>\n

What is magnitude and what are effect sizes?\u00a0\u00a0Magnitude refers to the amount of\u00a0studies that show a strong, positive cause-and-effect relationship between an intervention and improved academic or behavioral outcomes.\u00a0 Basically the more studies you have that meet the quality indicators, the larger the magnitude of the effects of the intervention.\u00a0 A intervention with 3 quality studies versus an intervention with 13 studies means that the intervention with 13 studies has a greater magnitude for effectiveness.<\/p>\n

Magnitude is also applied to effect sizes.\u00a0 Effect sizes report statistical information about how impactful an intervention was on student performance.\u00a0 For example an effect size of .15 demonstrates that there was some effectiveness to the intervention but not enough to be statistically significant.\u00a0 However an effect size of 1.5 shows a greater amount of effectiveness of the intervention on student outcomes.\u00a0 Rule of thumb for effect sizes is the closer you get to 1.0, the more statistically significant the effects.\u00a0 Oh and remember too that the closer to zero or negative numbers show a negative outcome for the intervention on student outcomes. The larger the effect size, the greater the magnitude of its impact.<\/p>\n

 <\/p>\n

\"Qualification<\/p>\n

So what do these research designs have to do with a practice being evidence-based?\u00a0 Based on the rigor and quality of each of the design types, there is a minimum numbers of quality studies needed to identify a practice or intervention as evidence-based.<\/p>\n

Here is the general guidelines that are used in special education:\u00a0<\/strong><\/p>\n

 <\/p>\n

General Guidelines for Evidence -Based Practices in Special Education<\/h2>\nAdapted from Cook, Farley, and Cook, 2012<\/span>\n\n\n\n\n\t\n\t\n\t\n\t
Design Type:<\/td>Guidelines:<\/td>\n<\/tr>\n
Group Design and quasi-experimental <\/td>At least two high-quality or four acceptable-quality studies must support the practice as effective, with a weighted effect size across studies significantly greater than zero. High-quality studies must meet criteria in at least three of
\nthe following areas: description of participants, intervention and comparison condition, outcome measures, data analysis, and at least four of eight \u201cdesirable\u201d indicators, such as attrition rate (Gersten et al., 2005).
\n<\/td>\n<\/tr>\n
Single Subject<\/td>Five or more high-quality studies with 20 or more participants that meet criteria related to participants and setting; independent and dependent variables; baseline; and internal, external, and social validity must support the practice as effective (Horner et al., 2005).<\/td>\n<\/tr>\n
Qualitative<\/td>Although qualitative studies are useful for many purposes, they are not designed to determine whether a practice causes improved student outcomes (McDuffie & Scruggs, 2008).<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

 <\/p>\n

Now that we have looked at design\u00a0types, magnitude and effect sizes, let’s look at some resources for selecting Evidence-Based Practices.<\/b><\/p>\n

 <\/p>\n

\"To<\/a><\/p>\n

 <\/p>\n

References:<\/strong><\/p>\n

Council for Exceptional Children (CEC). (2014<\/span>).\u00a0Council for Exceptional Children standards for evidence-based practices in special education<\/span>.<\/em> Retrieved from\u00a0http:\/\/www.cec.sped.org\/~\/media\/Files\/Standards\/Evidence%20based%20Practices%20and%20Practice\/CECs%20EBP%20Standards.pdf<\/a><\/p>\n

Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71<\/em>, 149\u2013164.<\/p>\n

Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71<\/em>, 165\u2013179.<\/p>\n

Kratochwill, T. R., Hitchcock, J. H., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2013). Single-case intervention research design standards. Remedial and Special Education, 34<\/em>, 26\u201338.<\/p>\n

Torres, C., Farley, C. A., & Cook, B. G. (2012). A special educator\u2019s guide to successfully implementing evidence-based practices. TEACHING Exceptional Children, 45(1)<\/em>, 64\u201373.<\/p>\n","protected":false},"excerpt":{"rendered":"

  The keystone of evidence-based practices (EPBs) is research.\u00a0 Through research, we are able to create a solid foundation for which an EPB can stand on.\u00a0 However, not all research is created equal.\u00a0 There are studies that are the cornerstone for other researchers, studies of such high-quality that they can be cited and reference with…<\/p>\n","protected":false},"author":8585,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"tags":[],"class_list":["post-940","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/pages\/940","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/users\/8585"}],"replies":[{"embeddable":true,"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/comments?post=940"}],"version-history":[{"count":20,"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/pages\/940\/revisions"}],"predecessor-version":[{"id":1082,"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/pages\/940\/revisions\/1082"}],"wp:attachment":[{"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/media?parent=940"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/my.dev.vanderbilt.edu\/spedteacherresources\/wp-json\/wp\/v2\/tags?post=940"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}