Reprinted with the kind permission of the author, Dr. Arnold Kling.
First released at TCS Daily.com
"When Consumer Reports rates cars, they drive them; [U.S. News and World Report] does the equivalent of measuring the amount of steel used in cars, rather than their performance." — Richard Vedder
In the fields of health care, education, and assistance to poor countries, we rarely measure value properly. It seems as though we prefer to be ignorant about what succeeds and what fails. We know shockingly little about the cost-effectiveness of very expensive programs.
The gold standard for cost-effectiveness analysis is a controlled experiment, also known as a randomized trial. Randomized trials are commonly used in determining the effectiveness of new pharmaceuticals. However, they are almost never used in determining the effectiveness of a new education method, a foreign aid program, or a non-pharmaceutical medical protocol, such as the decision to send a patient with a back injury for an MRI or to follow patients with a particular heart problem with visits to a cardiologist, say, monthly vs. annually.
MIT’s Abhijit Vinayak Banerjee wrote recently about the difficulties in measuring value of foreign aid. One problem is that researchers try to shortcut the process of randomized trials. For example, he wrote,
"A study of schools in western Kenya by Paul Glewwe, Michael Kremer, Sylvie Moulin and Eric Zitzewitz compared the performance of children in schools that used flip charts for teaching science and schools that did not and found that the former group did significantly better in the sciences… An intuitive assessment might have readily ascribed the difference to the educational advantages of using flip charts, but… Perhaps the parents of children attending these schools were particularly motivated and this motivation led independently both to the investment in the flip charts and, more significantly, to the goading of their children to do their homework. Perhaps these schools would have done better even if there were no such things as flip charts.
"Glewwe and company therefore undertook a randomized experiment: 178 schools in the same area were sorted alphabetically, first by geographic district, then by geographic division, and then by school name. Then every other school on that list was assigned to be a flip-chart school. This was essentially a lottery, which guaranteed that there were no systematic differences between the two sets of schools. If we were to see a difference between the sets of schools, we could be confident that it was the effect of the flip charts. Unfortunately, the researchers found no difference between the schools that won the flip-chart lottery and the ones that lost."
This is a classic illustration of what in statistics is known as the difference between an observational study and an experiment. In an observational study, the researcher does not determine which group gets the "treatment" (flip charts in this instance) and which is the control group (no flip charts). In an experiment, the researcher does determine who gets what.
Observational studies often result in incorrect attributions of causal relationships. For example, the press recently reported a study showing a relationship among teenagers between violent behavior and watching pro wrestling on television, as well as another study showing a relationship between teen sexual activity and the time they spent watching television programs with sexual content. However, as observational studies, these cannot demonstrate causality. It could be that a teen with a propensity to be violent would prefer to watch pro wrestling, and a teen with a propensity not to be violent would prefer to watch something else. All else equal, watching pro wrestling could turn out to have no effect, or even a dampening effect, on teens’ propensity for violence. To establish a causal relationship between what teens watch on television and how they behave, one would need to conduct a randomized trial in which the programs that teens watch are determined by the experimenter.
Costs and Benefits
The tendency in health, education, and foreign aid is to take the view that more is always better. Instead, cost-benefit analysis could substantially improve the return on investment. For example, Banerjee wrote,
"The cheapest strategy for getting children to spend more time in school, by some distance, turns out to be giving them deworming medicine so that they are sick less often. The cost, by this method, of getting one more child to attend primary school for a year is $3.25. The most expensive strategy among those that are frequently recommended (for example by the World Bank, which also recommends deworming) is a conditional cash-transfer program, such as Progresa in Mexico, where the mother gets extra welfare payments if her children go to school. This costs about $6,000 per additional child per year, mainly because most of the mothers who benefit from it would have sent their children to school even if there were no such incentive. This is a difference of more than 1,800 times."
In the process of writing Crisis of Abundance, my book on U.S. health care policy, I came across study after study that indicated that intensive utilization of medical services has little effect on aggregate outcomes. For example, Dartmouth Professor John Wennberg and colleagues have found very different levels of utilization by Medicare patients in different regions, but with similar health outcomes.
What these studies suggest is that we are sending patients to specialists, to hospitals, and for expensive diagnostic tests without knowing when this is cost-effective and when it is not. In a nation where health care spending as a share of income has roughly doubled over the past thirty years, and where consumers are more than 85 percent insulated from the cost of health care (because 85 percent of personal health care spending is paid by either private insurance or government), ignorance about cost-effectiveness is a major economic concern.
Ignorance in Education
We seem to be most determined to remain ignorant in the field of education. The testing in No Child Left Behind is based on observational studies, rather than experiments. That is, we look at school quality solely in terms of outcomes. We label a school as good or bad without having any idea whether the school actually adds value.
Imagine what might happen if one were to run a controlled experiment, pooling a group of students and randomly assigning them to different schools. Would the "good" suburban school really do better than the "failing" urban school, once the population of students is similar?
I suspect that controlled experiments in education would show shockingly little value added. That is, if you were to randomly assign students to schools, the children of good parents would do well regardless of where you send them, and conversely.
I suspect that controlled experiments in higher education would also show little value added. Everyone can cite the differences in earnings between college graduates and non-graduates, but those differences do not come from controlled experiments. What if you were to undertake a randomized trial, in which a cross section of high-school graduates is sent to college and then compared with a similar cross-section that is not sent to college? My guess is that the earnings differences will not be so large.
I think that many people would prefer not to have the answers to these sorts of questions. For the most part, consumers and taxpayers would rather not know whether education, health care, and foreign aid are cost-effective. Instead, people would rather "trust the experts" and attribute high skill levels to educators, doctors, and aid agencies. And, of course, the experts would like us to continue to pay their salaries without questioning their results. As on many other issues, in seeking cost-benefit analysis economists are fighting an uphill battle.
Arnold Kling is a TCS Daily contributing editor and author of Learning Economics.
The views expressed are those of the author, and not necessarily those of the Nassau Institute (which has no corporate view), or its Advisers or Directors.
