|
|
|
Vol 8|No 7|April|1999 |
|
|
|
|
|
|
Vol 8|No 7|April|1999 |
|
by Jamie McKenzie Not all studies of schools stand up to careful scrutiny. Some of them are badly done. Some ignore the most basic rules governing research design. And some are freighted with conflicts of interest and marketing motives. Despite these serious weaknesses, such studies still manage to gain the notice of the national press. Often as not, their findings are cited as if reliable and valid. Conjecture becomes headline. Inference becomes fact. We are flooded with virtual truths which take on more and more certainty as they are passed from one news story to the next. Even though the media may A leading example is the CEO FORUM Report and Star Chart which purports to describe the current state of technology use in American schools. The report lists the following individuals as the actual CEO Forum:
When the Year 2 Report first surfaced, I was heartened by its emphasis upon professional development, but when I probed below the surface and read the report's methodology section, I became concerned. Sadly, this kind of report has elements of the "fox in the hen house." "Mr. Fox, how would you suggest we lock the hen house?" Much of the Year 2 Report is a reasonable summary and rehashing of other groups' findings, recommendations and suggestions. But there are three major problems with the report . . . Problem One - "Technology for Technology's Sake" The report emphasizes "technology" throughout its recommendations and suggestions as if technology were an end rather than a delivery system, as if technology were a subject. There is far too little mention of literacy or learning. Technology should not be the end. In fact, an overemphasis of technology standards and programs is one of the biggest obstacles schools face to true program integration. When states such as North Carolina emphasize separate technology standards, many schools in those states create special "cram" classes to make sure their students pass the state test, diverting attention from appropriately integrated experiences. The overall effect of this bias is to sell "technology for technology's sake." This slant is also a serious misappropriation of the term "technology" because it implies that technologies must be plugged into walls and run on electricity while we know that books are technologies and thinking skills are technologies. We see too little emphasis upon strategic teaching and too much on equipment. The report speaks far too little of learning and far too much about training, about workplaces and about skills. As we read through the recommendations of the report, we need to notice the dollar signs emerging between the lines, as acceptance of the report would represent even larger sales of products the benefits of which remain virtually unproved. We have here a modern example of the Emperor's New Clothes. The assumption (unsubstantiated) is that putting networked computers in all classrooms will enhance the learning capabilities of students. Problem Two - Lack of Credible Data The report offers a new model to describe how many schools fall into various categories of technology usage. This model, the Star Chart, claims to take us beyond the mere counting of items. While the Star Chart purports to assess schools' commitments to content, professional development and integration as well as ownership, when From Now On examined the research base advanced to substantiate the claims, we found the research data thin or not available for scrutiny (see explanation later in this article). Problem Three - Buy First, then Get Ready The report fails to address leading strategic issues which could hurt computer sales but which might lead to better technology implementations. This silence on issues such as pacing of purchases in concert with development of teacher readiness is self serving and ingenuous. The implicit message of the report is "Buy first" and then see if you can convince teachers to use what has been installed. We see many schools putting networked computers into rooms where there is no readiness or predisposition to employ the new equipment as an important part of the learning program. Business literature such as Crossing the Chasm1 (Moore, 1991) has established clearly that we have early adopters and late adopters in the technology marketplace, each of which requires special attention, yet the wholesale networking and equipping of classrooms suggested by computer companies and various government officials ignores what we know about teacher readiness. The CEO Forum Report shares the Apple Classroom of Tomorrow stages of Teacher Adoption of Technology (1. entry, 2. adoption, 3. adaptation, 4. appropriation, 5. invention) but never points out the folly of equipping teachers' rooms with lots of computers when they are still in the early stages. The proper role of business in developing research? These three problems raise a more serious issue . . . which is the proper role of business in developing research reports for schools. Some of us would like to see these companies concentrate on building their products and inventing educational solutions with the assistance of knowing educators. We would also applaud their funding of serious, well designed educational research directed by credible educational researchers such as Henry Becker and Barbara Means, but it is important that research be conducted with a respect for truth uncolored by corporate interests. In direct contrast to the CEO Forum reports, we have the excellent data being collected and reported by Henry Becker and his colleagues from the University of California at Irvine. One need only compare the two with regard to methodology to appreciate the difference between research and virtual truth. "Internet Use by Teachers" is the first in a series of twelve reports that UCI will produce from their NSF/OERI-jointly funded grant on Computer Technology and Instructional Reform. (See http://www.crito.uci.edu/TLC for more details.) Becker's findings and methodology are explored in some depth in an accompanying article in this issue . . . "Scoring High" (click here to view article) Failure to Honor Basic Research Design Standards In its first year, the CEO FORUM limited its report primarily to data describing how much equipment schools owned and how many were connected to the Internet. The data seemed quite reliable because it was a matter of counting up survey responses. This year the CEO FORUM made a giant leap by trying to characterize actual technology use in schools (content, professional development, integration & use, educational benefits) - a number of leaps and assumptions not sustained by reliable and credible data. Their report would never pass the scrutiny of a dissertation committee. The methodology section for the "Star Chart" is remarkable for its brevity and superficiality. A single sentence is all we see for each of the major sections of the chart:
"That's all she wrote!" Considering the serious tone of the rest of the report which purports to speak authoritatively about the state of education in 1998/99, the lack of a substantial foundation for such statements and conclusions is surprising. When From Now On pressed for the actual interview questions, the data, the number of people interviewed and the missing substance, one of the report's authors stated that those items could not be made available. The author confirmed that much of the material within the sections listed above (such as content, professional development, integration & use and educational benefits) was "based on extrapolation from the hardware data from QED." He stated that interview data was from a McKinsey study of Texas school districts gathered in 1997. Yet the report purports to describe current national usage! "24% of the schools effectively integrate technology while more than 50% remain in the Low Tech category," the report states authoritatively. How do they know that? They don't. They mainly know how much equipment folks have bought and they are tying usage to purchase and ownership. ". . . logical assumptions about the activities that different technology infrastructures allow." Henry Becker's data conflict with this grandiose claim. In addition, the annual Technology in Education 1998 Report from Market Data Retrieval reports that Internet access has increased dramatically while just seven percent of schools claim that the majority of their teachers are at an Advanced skill level (able to integrate technology use into the curriculum). (http://www.schooldata.com) Who should we believe? Many of the generalizations and conclusions of this report regarding technology do not stand up to scrutiny. The author could not provide data to substantiate many of the claims and findings. The report is available at http://www.ceoforum.org as two PDF (Adobe Acrobat) files. This report fails to meet basic standards of educational research . . .
1. Does the report name its authors? Serious and credible scientific and educational research requires naming of authors along with information regarding their qualifications and possible conflicts of interest. The CEO Forum does not list its authors and does not provide any evidence that they have a background which equips them to comment upon educational policy issues or conduct serious and credible educational research. This information should be publicly reported to all who wish to question the credibility of the report, but it is concealed from view. After a number of phone calls, From Now On was able to establish that the report was primarily written by two consultants hired by a consulting company, InfoTech Strategies, who do not have academic credentials directly related to conducting or interpreting educational research, although one consultant does have a M.A. in public policy from the Kennedy School of Government. In the educational world we'd expect a Ph.D. or an Ed.D. at least from the primary authors. We'd expect extensive training in educational research design and statistical analysis. 2. Is the report and data generated by disinterested and independent researchers? When Henry Becker at the University of California asks how teachers are using technology, he is unlikely to be accused of a conflict of interest which might "color" his findings, but when technology companies pay consultants to write a report and these consultants remain nameless and anonymous, we have no way of judging the independence of their thinking. When educational research is conducted by university researchers funded by federal grants, there is a considerably higher level of credibility - a sense that their inquiry is driven by a search for truth and understanding rather than profits and sales. Even then, one would want to know if the educational researchers have often served on the payroll of vendors who have an interest in the outcome of the research. In the medical community, research scientists who have worked for the pharmaceutical industry are expected to disclose those relationships and those possible conflicts of interest when they report their findings. 3. Do we have a chance to see actual data? The data is not displayed. We do not know how many people made various responses. The data was not available to this journalist interviewing the primary author of the report. 4. Do we have a chance to see the surveys which generated the data? The survey and interview questions are not displayed. We do not know if the questions pass various tests of reliability. The survey and interview questions were not available to this journalist interviewing the primary author of the report. 5. Do we know how many people were interviewed, how they were selected and who they are? The number of people interviewed is not disclosed and we cannot learn how they were selected or who they were. This is a critically important issue because various data groups have interviewed technology directors and asked them how many people in their schools are using computers in various ways. This choice of interviewee tends to skew the findings in an optimistic direction since technology directors are paid to increase staff usage and are unlikely to report failures. Valid data would include a sample of randomly selected classroom teachers as well as others with no vested interest in reporting high usage such as parents and students. The information regarding interviewees was not available to this journalist interviewing the primary author of the report. 6. Do we know when the data was collected? It is unclear from the report when the interviews and data was collected. One would assume from the way the report makes statements about "current practice" that the data was gathered in 1998 or 1999. It turns out that most came from a 1997 McKinsey study. 7. Do we have a clear picture of the way data was translated into findings? There is a very big leap from the methodology section to the conclusions and ratings of schools. The phrase "logical assumptions" is never explained. It appears that the authors (and funders) associate good technology usage in schools with abundant purchase. That is an assumption one would expect to read in a report published by a group of vendors and merchants, but it hardly stands up to logical analysis. There is far too much credible data on teacher use of technologies from researchers like Henry Becker to buy this leap of the CEOs. 8. Do we have data on reliability, validity and statistical significance? We are given no data on reliability, validity and statistical significance. Conclusion The advent of networked computers to school during the past 3-4 years has been accompanied by tremendous fanfare from the companies supplying the "goods." Unfortunately, what we have bought is half a product . . . a delivery system which remains virtually unproven and without a clearly substantiated educational benefit. We suffer from a surplus of marketing hype which confuses the decision-making process as claims, promises and ersatz data flood the market and the schools. What we have is business people offering up very thin research findings which do not merit serious attention. It turns out that the CEO Forum's observations about content, professional development and technology usage were based primarily on what the group calls "an extrapolation." The most serious "hidden assumption" is that possession of resources leads to "effective technology integration." They seem to be saying "Buy and it will be used." But they have no evidence to back up this claim and the data collected by Becker and reported in this month's "Scoring High" article contradicts their claims. Even in the well equipped classrooms of Becker's study, the level of use is pretty dismal. When we read the marketing books of the CEO's colleagues such as Crossing the Chasm1 (Moore, 1991) we learn that there is normally a huge "gap" or "chasm" between early and late technology adopters. The assumption that late adopters follow the lead of early adopters has proven to be wrong-minded and dangerous, according to Moore. Crossing the chasm between these groups, states Moore, requires a mammoth campaign which includes special attention to the vastly different needs, perspectives and demands of the late adopters. What works for pioneers does not work for the later group. When schools place lots of networked computers in the rooms of reluctants and late adopters they are courting disaster. They are placing the cart before the horse. We have no convincing evidence that possession translates into dramatic, consistent and effective use. Good policy would argue for restraint and pacing . . . no classrooms before their time. Good policy would argue that strong professional development programs should be timed to match the introduction of networking computers to a school but that the extra levels of comfort required to make profitable use of 4-5+ computers in each classroom represent a huge challenge which should precede widespread dispersal into all rooms. Henry Becker's research shows that access to computers helps, but he also finds that certain types of teachers (those with constructivist leanings) are far more likely to make important use of these new technologies than more traditional teacher types. To spread such technologies throughout a school's classrooms willy-nilly without taking into account what we have learned from twenty years of experience with technology is foolhardy. We can hardly blame the technology companies for urging full speed ahead here. That is, after all, their business. But we must respect that age old warning "Buyer beware!"
Credits: The photographs were shot by Jamie McKenzie. Icons from Jay Boersma. Copyright Policy: Materials published in From Now On may be duplicated in hard copy format if unchanged in format and content for educational, nonprofit school district and university use only and may also be sent from person to person by e-mail. This copyright statement must be included. All other uses, transmissions and duplications are prohibited unless permission is granted expressly. Showing these pages remotely through frames is not permitted. |
From Now On