Monthly Archives: June 2019

Exploring the limits of quantitative measures: The case for including qualitative studies in educational research. By Doctor Sam Bommarito

reading creatuve commons

 

One recent post on twitter indicated that the only real criteria for judging the worth of instructional methods are that they pass the test of randomized controlled trials. This randomized control statement is one important way of looking at things. However, like all things in educational research, there are limits and limitations to the use of “pure” science. This is especially true when looking at the instrumentation often used in conducting educational research.  Let me give two examples that demonstrate this point.

The first is the use of the Lexile system for measuring readability.  I have long maintained that measures of readability are really measures of decodability. They should really be called decodability formulas. For instance, a 4th-grade decodability means that a student performing at a 4th-grade reading level can decode it. In no way does it guarantee that the material in the book itself is actually 4th-grade material, more on that thought in a minute.

The components that go into Lexile measurements are fairly straightforward. See this link for an explanation of how they are determined: https://lexile.com/educators/measuring-growth-with-lexile/lexile-measures-grade-equivalents/ .

Key components include sentence complexity and vocabulary.  Overall the system does a fairly decent job of doing what it is supposed to do. It does give us a sense of what grade level a child’s reading ability needs to be to decode it.

There are critics of the Lexile system. I’m providing a link to this anti-Lexile post not because I agree with its conclusion (I don’t) but because it contains good examples of times when Lexile really gets it wrong. http://www.unleashingreaders.com/?p=8891.

TIMES WHEN LEXILES GET IT WRONG

Examples of Of where Lexile scores get it wrong.

 

 

I’ve seen the Grapes of Wrath example in many places. The content of Grapes of Wrath clearly does not fit the typical content for a 4th-grade classroom.  This particular example seems to be a favorite of the critics of Lexile measurements.

What to do, what to do?

Here is what to do. Do exactly what systems like Fountas and Pinnell do when they assign readabilities to books. In addition to the usual things (sentence complexity/vocab load) they look at the actual content of the book, how the book is structured, specific characteristics of the book and the degree to which the book that is measured matches the usual characteristics of books at that reading level. F & P’s system adds qualitative information that makes their product far more useful to classroom teachers.  When talking to teachers about which system to use when choosing a leveled text, I always recommend systems like F & P.  Fewer “surprises” like saying the Grapes of Wrath or the Color Purple are appropriate books for 4th graders.  The overall point here is that adding the qualitative element to the decodability measure makes It much more useful for educators. Quantitative measures can benefit from the addition of qualitative supplements.

Another example is the whole idea around measuring fluency. As I mentioned last week, Tim Rasinski just did an excellent post about that. The post was from the Robb Review.

https://therobbreviewblog.com/uncategorized/making-kids-read-fast-is-not-the-goal-of-fluency-instruction-making-meaning-is/

Last week’s post is germane to this week’s topic because unlike measures like the Dibels which measures fluency solely as speed, Tim’s measurement is much more complete since it includes the use of a rubric based on the acronym E.A.R.S. (Expression, Automatic Word Recognition, Rhythm and Phrasing, and Smoothness). Dibels focuses only on speed.  Teachers using Dibels for instruction use a shallow view of the oral reading process that too often results in the creation of Robot Readers. These readers read in a monotone, lifeless manner. Such readers are not likely to use oral reading as a window to comprehension.

Tim had this to say in a Twitter post this week:

TIM RAZ

 

 

Tim’s post clearly indicates that when most of the literacy instructional time is used on phonics instruction with little or no time spent on meaning making the result can be the creation of word callers. This dove tales completely with my own experiences working with children placed in a basal that focused too much of its time on phonics and not enough of its time on things like Tim’s style of fluency (shall we call it prosody?), writing, and comprehension.  This isn’t to say there should be no time spent on decoding. This is simply saying that there should also be SIGNIFICANT amounts of time spent on the other components of the reading process.

Overall, Tim’s way adds a definite qualitative aspect to the mix.  Instead of just doing simple speed measures, teachers must carry out a somewhat more complex measure using his rubric. Does that make it less scientific?  For those who have taken courses in qualitative analysis, you know that if one takes the time and follows the procedures, one can get interrater reliability. It is completely scientific. Qualitative measures and qualitative statistics are perfectly capable of answering the question of whether or not the results of a study are simply the result of chance or the result of the experimental treatment.

This leads me to my big question. Is there a reason to also include qualitative studies when evaluating the efficacy of reading approaches?  I think there is. Qualitative methods pick up on things pure quantitative methods simply can’t.  Is this a case of advocating my science against your science? Not really. It’s a case of advocating my science (qualitative studies) IN ADDITION TO your science. It’s an acknowledgment that empirical approaches have limits and limitation. Qualitative approaches can make important value-added contributions to the study of literacy approaches. In sum, I’m not saying to replace purely empirical studies. I am saying that we should consider supplementing their findings with qualitative studies as we determine the efficacy of various approaches in the teaching of literacy. Tim has said in the past that the teaching of reading is both science and art. I’ve given the link to my post about his views around this previously. Here it is again.

https://doctorsam7.blog/2018/05/04/the-teaching-of-reading-as-both-science-and-art-a-report-evaluation-of-rasinkis-recent-presentation-in-st-louis-by-dr-sam-bommarito/

I wholeheartedly concur with Tim. The teaching of reading really is both art and science. I think the study of the efficacy of literacy approaches would benefit from the inclusion of qualitative studies. What do you think?

Til next time, Happy Reading and Writing

Dr. Sam Bommarito (a.k.a. the quantitative PLUS qualitative guy)

Copyright 2019 by Dr. Sam Bommarito. Views/interpretations expressed here are solely the view of this author and do not necessarily reflect the views of any other person or organization.

P.S. If you found the blog through Facebook or Twitter, please consider following the blog to make sure you won’t miss it.  Use the “follow” entry on the sidebar of the blog.

Oral Reading Fluency: Exploring fluency practices suggested by the work of Dr. Tim Rasinski (Part Two) by Dr. Sam Bommarito

Fluent Reading

Oral Reading Fluency: Exploring fluency practices suggested by the work of Dr. Tim Rasinski (Part Two) by Dr. Sam Bommarito

 

While I was away on vacation, Dr. Tim Rasinski made a post in the Robb review that is quite relevant to our topic of reading fluency (I follow the Robb review- lots of good stuff there!). Here is a link to his post:

https://therobbreviewblog.com/uncategorized/making-kids-read-fast-is-not-the-goal-of-fluency-instruction-making-meaning-is/

==============================================================

Key ideas included:

  • The real goal of phonics instruction should be to “get kids to the point where they don’t have to use phonics.”
  • Fluency is not about reading fast; fluency is about reading in a manner that promotes meaning-making. He did an ORF score (speed test score) on recordings of arguably two of the most fluently read speeches in American history – Martin Luther King’s “I Have a Dream Speech” and John F. Kennedy’s inaugural address “Ask Not What Your Country…”. He found that “in both cases, Dr. King and President Kennedy’s readings of their speeches may have landed them in a remedial reading class based on their very low ORF scores.”
  • He urged caution in using the scores, e.g., DIBELS and AimsWeb ORF scores, or Hasbrouck and Tindal’s norms (Words Correct per Minute).
  • He also said that “For fluency instruction to truly work, we need to see the goal of fluency as expressive oral (and silent) reading that reflects the meaning of the text.

=============================================================

Which brings us round to the story I was telling you before I left for vacation.  I volunteer one day a week doing literacy work at an elementary school. I had been using the idea of teaching kids to read like a storyteller in the after-school group all year.  About 20 1st and second graders participated.  We used one of the several techniques Rasinski reported on when he spoke to us in St. Louis. We paired off readers and had them read/reread poems and passages from favorite books.  We used several Eric Litwin’s books since they were at the right readability, were engaging and predictable.  When the after-school program ended, I asked one of the first-grade teachers if she would be interested in trying the reading practice with the whole class for a couple of weeks. She agreed.

We paired readers with somewhat higher scores with readers with somewhat lower reading scores. We allowed the children to self-select from poems that they were familiar with. This included some from trade books and some from the basal program. Partners could select the same poem OR they could each select a poem of their own.  Students practiced reading to each other daily. The teacher circulated in the room as the oral reading went on. The entire process took 10 minutes or so each day. I was present for this activity each Tuesday.

Students knew there would be a performance event at the end of the week. The classroom was using SeeSaw. The kids made audio recordings. The purpose of this little trial was to see if the primary teachers would be interested in trying this out during the school year next year. I’m happy to report that the results were such that both the 1st and 2nd grade is going to do this next year.

Here are some of the highlights:

The classroom teacher was surprised to see how quickly some of the “average” students improved in prosody. The stronger of the two partners were showing improvement as well.  The kids coached each other in how to sound more like storytellers.   One interesting anecdote came when I gave my usual talk about reading like a storyteller NOT like a robot. “Never read like a robot!” I said. One of the 1st graders raised his hand and said, “But Dr. B., what if the character is a robot? Then what?”  Hmmm. Guess you know how I answered.  You know, sometimes the kids learn more than you teach them!

I have parent permission to listen to the recordings, and over the summer I’ll be using the rubrics from Rasinski and Smith’s The Megabook of Fluency to assess those recordings.  I’ve talked about that rubric before. It uses the acronym E.A.R.S. Expression, Automatic Word Recognition, Rhythm and Phrasing, and Smoothness.  By the fall I expect to be able to use the rubric with some degree of reliability.

All four of the classes (2 first grades and 2-second grades) use the Raz Kids computer program.  I already have used the message feature of that program to talk to the after-school kids and give them advice about reading. The program allows both written and recorded messages. I think feedback on oral reading might prove useful.  Stay tuned- by the end of next year, I may have an action research project to report on!

Next time I want to take up the topic of the advantages of supplementing quantitative research with qualitative research.  I’ll explain how the startup I just described does exactly that.  I’ll also explore some of the limits and limitations of using only quantitative information.

Until next time, this is Dr. Sam signing off.

 

Dr. Sam Bommarito

Copyright 2019 by Dr. Sam Bommarito. Views/interpretations expressed here are solely the view of this author and do not necessarily reflect the views of any other person or organization.

P.S. If you found the blog through Facebook or Twitter, please consider following the blog to make sure you won’t miss it.  Use the “follow” entry on the sidebar of the blog.

Back from vacation- Some thoughts about the “Bookends” of WWII found at Pearl Harbor By Dr. Sam Bommarito

Back from vacation- Some thoughts about the “Bookends” of WWII found at Pearl Harbor

 

Pearl Harbor “Bookends”                                                Plaque Marking Surrender Spot

Just back from vacation. Letting everyone know my regular posts will resume next Friday morning. My last day in Hawaii was spent touring the bookends of WWII. The bookends include the Arizona Memorial (the beginning) and the Battleship Missouri (the end). Our guide for the Battleship Missouri tour told the moving story of how General MacArthur handled the speech he gave at the end of the war.  He spoke at the surrender ceremony that took place on the deck of the Battleship Missouri. The guide noted MacArthur could have talked about all the horrors and injustices of the war. Instead here is the heart of what he said:

“It is my earnest hope, and indeed the hope of all mankind, that from this solemn occasion a better world shall emerge out of the blood and carnage of the past — a world founded upon faith and understanding, a world dedicated to the dignity of man and the fulfillment of his most cherished wish for freedom, tolerance, and justice.”

So, he chose to offer a message of hope at the end of the war. A message of healing for all sides. Perhaps this is a message worth pondering.  See you next Friday morning.

 

Dr. Sam