The Research Gap By Jamie McKenzie about the author This article first appeared in the February, 2000 issue of eSchool News. © 2000, JMcKenzie, all rights reserved.

Four teachers and two administrators have flown two thousand miles to attend a national conference on educational uses of new information technologies. They boarded the plane confident that they would return home armed with research findings to guide decision-making. They are united in their faith that decisions should be guided by data.

• What works?
• What fails?
• How can we do our best by the students?
• How do we get the optimal return on our technology investments?

Sadly, they are disillusioned by the lack of significant research being conducted by most educational communities rushing to network schools.

"There's a big gap here," complains Sally, the veteran technology enthusiast of the team. So committed is she to the vision that she is spending evenings, weekends and summers picking up an advanced degree in educational technology. "There is so little research to help us think about our choices."

The team is camped out in an espresso Internet cafe several blocks from the Convention Center. It is the final morning of the final day, and they are having trouble mobilizing for the closing keynote.

"I'm tired of all these promises and visions and the whole 'new new thing,'" adds Gregory, the team's hardened skeptic. "I haven't found a single session where they reported sound research."

At this point, one of the administrators joins in. "Well, I did find one session - the one by Hank Becker from UCI - and his reports are a good start, but I agree that there were slim pickings, unless, of course, we include the corporation funded reports."

"I tried a few of those," laughed Sally, "and they failed to meet any of the research design standards I am expected to observe in my graduate classes. They were quite biased. Seems like we're going home empty-handed."

Answering Key Questions

Educational research is meant to help us to explore the most important questions we have at the end of each day. In the best of worlds, research would also provide guidance for educators to make wise choices from the sometimes bewildering menu being thrust toward them by eager vendors.

Providing Guidance

Given dozens of ways to use laptops or desktop computers once they arrive in our classrooms, which uses are most productive and which professional development experiences are needed to bring about the most return on investment?

If a district hoped to improve student scores on a demanding state writing test, for example, planners would want to know which of the following would have the greatest payoffs . . .

• Strategy 1 - Frequent guided writing practice with pencil and paper and no special strategy.
• Strategy 2 - Frequent guided writing practice with pencil and paper and an emphasis upon "writing as process" supported by cluster diagramming and the use of the Six Traits approach to writing. Teachers spend two weeks in the summer in writers workshops learning how to do "writing as process" with the Six Traits approach.
• Strategy 3 - Frequent guided writing practice with 10 hours of weekly computer time for each student using desktop computers and no special strategy.
• Strategy 4 - Frequent guided writing practice with 10 hours of weekly computer time for each student using desktop computers paper and an emphasis upon "writing as process" supported by cluster diagramming using the Inspiration™ software program and the Six Traits approach to writing. Teachers spend two weeks in the summer in writers workshops learning how to do "writing as process" as well as the Six Traits approach.
• Strategy 5 - Frequent guided writing practice with each student having their own laptop computers and no special strategy.
• Strategy 6 - Frequent guided writing practice with each student having their own laptop computers and an emphasis upon "writing as process" supported by cluster diagramming using the Inspiration software program and the Six Traits approach to writing. Teachers spend two weeks in the summer in writers workshops learning how to do "writing as process" and use Inspiration ™ plus the Six Traits approach on their own laptop computers.

A properly designed study of these six strategies would prove enlightening, but no credible, comprehensive, comparative studies exist on writing or any other major educational question related to the impact of new technologies.

Reliability Issues

The design of reliable educational research is beyond the scope of this article, but there are classic books, like Campbell and Stanley Experimental and Quasi-Experimental Designs for Research (1966, Houghton Mifflin College; ISBN: 0395307872) that outline the traits of a credible study. Unfortunately, many of the reports gaining national attention these days do not meet those specifications.

A recent example would be an Educational Testing Service study, "Does it Compute? The Relationship Between Educational Technology and Student Achievement in Mathematics," by Harold Wenglinsky, exploring relationships between computer experiences and math performance on the NAEP tests (National Assessment of Educational Progress). (http://www.ets.org/research/pic/technolog.html )

This study found that certain teacher strategies were associated with lower math performance while others were associated with higher performance (but only at some grade levels).

The researchers did what is called ex post facto research. Rather than setting up a study to test hypotheses with all confounding conditions controlled, they take data from a broader study and "mine" the data to find links between various events. This kind of research shows association but does not show cause.

We can see a correlation between phone ownership in African countries and lower infant mortality, but that doesn't mean we could reduce infant mortality by purchasing more phones for everybody. Phone ownership is an indicator of wealth. Wealthy countries have better inoculation programs, better nutrition and better water quality — some likely causes of lower infant mortality. Using ex post facto research we could (erroneously) claim that people should buy more phones to save children’s lives!

Objectivity Issues

A standard issue in research is the possibility that researchers might have a vested interest in the outcome of the study. Validity and credibility is undermined when the authors and funders are closely allied and likely to benefit if the study comes out in a manner that makes their product look good. This basic rule is frequently ignored today as companies pay for research without observing the objectivity standards and are able to win national press attention by virtue of their superior public relations resources while more credible academic research fails to gain notice.

Theory Testing

What we need is more theory testing like the writing example provided earlier. Schools need well constructed, publicly funded, academically sound research to determine which strategies are most likely to produce the learning outcomes required to meet performance and curriculum goals. Instead, we are seeing an increase in national reports attempting to describe what is happening across the land.

Before a public opinion pollster can claim to describe behaviors and attitudes of the general public, they must discuss how their sample is representative. Most of the recent reports have failed to meet that standard, but are still quick to make generalizations. Their findings about teachers use of the Internet may be based on one district employee’s speculation, for example.

The New York Cybertimes of December 1, 1999 summarizes a report, "Technology in Education 1999," a Market Data Retrieval about computers in the schools.

One finding . . .

"And 54 percent of schools report that a majority of their teachers are now using the Internet in instruction." This conclusion contrasts dramatically with previous reports by Becker and others that show much smaller percentages of use by teachers in instruction.

". . . a minority of teachers had students use Web browsers during the last school year (36%) . . ." (Becker, "Internet Use by Teachers," p. 6)

http://www.crito.uci.edu/TLC/FINDINGS/internet-use/startpage.htm

"What might be called "regular" use — using the World Wide Web to do research on at least 10 occasions — was a practice of nearly one-quarter of all teachers with a modem in their classrooms and 30% of those with direct high-speed connections." (Becker, p. 6)

The reporter does an excellent job of adding interviews from skeptics who raise the point that the type of use is more important than the amount of access, but the 54% statistic stands pretty much unchallenged, even though it is highly speculative in its origins, being based on self reports of schools and teachers with sample sizes often quite limited and possibly skewed.

http://www.schooldata.com/publications3.html

In one Market Data Retrieval report for Technology Counts’99, for example, the responses were quite low . . .

On April 28, surveys were mailed to a stratified sample of 15,000 teachers--classroom teachers at the elementary school level and English, mathematics, science, and social studies teachers at the middle and high school levels. 1,407 teachers responded by the cutoff date of May 21.

An educational researcher would be expected to comment on how respondents might differ from non-respondents, but Market Data Retrieval remains silent on this important issue.

What Research is Needed

We need much more research comparing learning strategies for both teachers and students. We see little that would help us to make choices. So far the research on teachers has done little but count the hours of training. There is little differentiating between software training courses and adult learning strategies such as study groups and mentor programs. They all are lumped together.

We also need to know more about the consequences of different delivery systems and types of equipment deployment. Are we better off spreading computers evenly across all classrooms, for example, than moving them about in strategic clusters?

Most important of all, we need government funding for robust, independent research that will free schools from reliance upon research driven by marketing concerns and interests.