Exploring the limits of quantitative measures: The case for including qualitative studies in educational research. By Doctor Sam Bommarito

reading creatuve commons

 

One recent post on twitter indicated that the only real criteria for judging the worth of instructional methods are that they pass the test of randomized controlled trials. This randomized control statement is one important way of looking at things. However, like all things in educational research, there are limits and limitations to the use of “pure” science. This is especially true when looking at the instrumentation often used in conducting educational research.  Let me give two examples that demonstrate this point.

The first is the use of the Lexile system for measuring readability.  I have long maintained that measures of readability are really measures of decodability. They should really be called decodability formulas. For instance, a 4th-grade decodability means that a student performing at a 4th-grade reading level can decode it. In no way does it guarantee that the material in the book itself is actually 4th-grade material, more on that thought in a minute.

The components that go into Lexile measurements are fairly straightforward. See this link for an explanation of how they are determined: https://lexile.com/educators/measuring-growth-with-lexile/lexile-measures-grade-equivalents/ .

Key components include sentence complexity and vocabulary.  Overall the system does a fairly decent job of doing what it is supposed to do. It does give us a sense of what grade level a child’s reading ability needs to be to decode it.

There are critics of the Lexile system. I’m providing a link to this anti-Lexile post not because I agree with its conclusion (I don’t) but because it contains good examples of times when Lexile really gets it wrong. http://www.unleashingreaders.com/?p=8891.

TIMES WHEN LEXILES GET IT WRONG

Examples of Of where Lexile scores get it wrong.

 

 

I’ve seen the Grapes of Wrath example in many places. The content of Grapes of Wrath clearly does not fit the typical content for a 4th-grade classroom.  This particular example seems to be a favorite of the critics of Lexile measurements.

What to do, what to do?

Here is what to do. Do exactly what systems like Fountas and Pinnell do when they assign readabilities to books. In addition to the usual things (sentence complexity/vocab load) they look at the actual content of the book, how the book is structured, specific characteristics of the book and the degree to which the book that is measured matches the usual characteristics of books at that reading level. F & P’s system adds qualitative information that makes their product far more useful to classroom teachers.  When talking to teachers about which system to use when choosing a leveled text, I always recommend systems like F & P.  Fewer “surprises” like saying the Grapes of Wrath or the Color Purple are appropriate books for 4th graders.  The overall point here is that adding the qualitative element to the decodability measure makes It much more useful for educators. Quantitative measures can benefit from the addition of qualitative supplements.

Another example is the whole idea around measuring fluency. As I mentioned last week, Tim Rasinski just did an excellent post about that. The post was from the Robb Review.

https://therobbreviewblog.com/uncategorized/making-kids-read-fast-is-not-the-goal-of-fluency-instruction-making-meaning-is/

Last week’s post is germane to this week’s topic because unlike measures like the Dibels which measures fluency solely as speed, Tim’s measurement is much more complete since it includes the use of a rubric based on the acronym E.A.R.S. (Expression, Automatic Word Recognition, Rhythm and Phrasing, and Smoothness). Dibels focuses only on speed.  Teachers using Dibels for instruction use a shallow view of the oral reading process that too often results in the creation of Robot Readers. These readers read in a monotone, lifeless manner. Such readers are not likely to use oral reading as a window to comprehension.

Tim had this to say in a Twitter post this week:

TIM RAZ

 

 

Tim’s post clearly indicates that when most of the literacy instructional time is used on phonics instruction with little or no time spent on meaning making the result can be the creation of word callers. This dove tales completely with my own experiences working with children placed in a basal that focused too much of its time on phonics and not enough of its time on things like Tim’s style of fluency (shall we call it prosody?), writing, and comprehension.  This isn’t to say there should be no time spent on decoding. This is simply saying that there should also be SIGNIFICANT amounts of time spent on the other components of the reading process.

Overall, Tim’s way adds a definite qualitative aspect to the mix.  Instead of just doing simple speed measures, teachers must carry out a somewhat more complex measure using his rubric. Does that make it less scientific?  For those who have taken courses in qualitative analysis, you know that if one takes the time and follows the procedures, one can get interrater reliability. It is completely scientific. Qualitative measures and qualitative statistics are perfectly capable of answering the question of whether or not the results of a study are simply the result of chance or the result of the experimental treatment.

This leads me to my big question. Is there a reason to also include qualitative studies when evaluating the efficacy of reading approaches?  I think there is. Qualitative methods pick up on things pure quantitative methods simply can’t.  Is this a case of advocating my science against your science? Not really. It’s a case of advocating my science (qualitative studies) IN ADDITION TO your science. It’s an acknowledgment that empirical approaches have limits and limitation. Qualitative approaches can make important value-added contributions to the study of literacy approaches. In sum, I’m not saying to replace purely empirical studies. I am saying that we should consider supplementing their findings with qualitative studies as we determine the efficacy of various approaches in the teaching of literacy. Tim has said in the past that the teaching of reading is both science and art. I’ve given the link to my post about his views around this previously. Here it is again.

https://doctorsam7.blog/2018/05/04/the-teaching-of-reading-as-both-science-and-art-a-report-evaluation-of-rasinkis-recent-presentation-in-st-louis-by-dr-sam-bommarito/

I wholeheartedly concur with Tim. The teaching of reading really is both art and science. I think the study of the efficacy of literacy approaches would benefit from the inclusion of qualitative studies. What do you think?

Til next time, Happy Reading and Writing

Dr. Sam Bommarito (a.k.a. the quantitative PLUS qualitative guy)

Copyright 2019 by Dr. Sam Bommarito. Views/interpretations expressed here are solely the view of this author and do not necessarily reflect the views of any other person or organization.

P.S. If you found the blog through Facebook or Twitter, please consider following the blog to make sure you won’t miss it.  Use the “follow” entry on the sidebar of the blog.

2 thoughts on “Exploring the limits of quantitative measures: The case for including qualitative studies in educational research. By Doctor Sam Bommarito

  1. DeGee Brown

    We lose so much when we just use “cold, dry” numbers. When we use the additional information, we get a clearer, more complete picture of the reader.
    Maybe this is part of why teachers have balked so hard at all these tests that are supposed to tell us which students are “good” or “bad” readers—because, we are dealing with people and “The teaching of reading really is both art and science.”

    Reply

Leave a Reply to doctorsam7 Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.