Wednesday, December 08, 2010
In the late nineteen-nineties, John Crabbe, a neuroscientist at the Oregon Health and Science University, conducted an experiment that showed how unknowable chance events can skew tests of replicability. He performed a series of experiments on mouse behavior in three different science labs: in Albany, New York; Edmonton, Alberta; and Portland, Oregon. Before he conducted the experiments, he tried to standardize every variable he could think of. The same strains of mice were used in each lab, shipped on the same day from the same supplier. The animals were raised in the same kind of enclosure, with the same brand of sawdust bedding. They had been exposed to the same amount of incandescent light, were living with the same number of littermates, and were fed the exact same type of chow pellets. When the mice were handled, it was with the same kind of surgical glove, and when they were tested it was on the same equipment, at the same time in the morning.
The premise of this test of replicability, of course, is that each of the labs should have generated the same pattern of results. "If any set of experiments should have passed the test, it should have been ours," Crabbe says. "But that's not the way it turned out." In one experiment, Crabbe injected a particular strain of mouse with cocaine. In Portland the mice given the drug moved, on average, six hundred centimetres more than they normally did; in Albany they moved seven hundred and one additional centimetres. But in the Edmonton lab they moved more than five thousand additional centimetres. ...
The disturbing implication of the Crabbe study is that a lot of extraordinary scientific data are nothing but noise.
Tuesday, December 07, 2010
Although verbal overshadowing remains a widely accepted theory [the theory that witnesses' verbal descriptions of events decrease the accuracy of the witnesses' identifications and descriptions] -- it's often invoked in the context of eyewitness testimony, for instance -- [Professor Jonathan] Schooler [of UCSB, a principal originator of the theory] is still a little peeved at the cosmos. "I know that I should move on already," he says. "I really should stop talking about this [that his repeated efforts to replicate earlier results of empirical tests show decreasing support for the hypothesis of verbal overshadowing]." That's because he is convinced that he has stumbled on a serious problem, one that afflicts many of the most exciting new ideas in psychology.Following the report of this near-confession by Schooler, the article recounts the story of the Duke psychologist who in the 1930s conducted experiments that suggested that one of his human subjects [a student, Adam Linzmayer] had extra-sensory perception: The student seemed to have an uncanny ability to guess hidden cards in a special deck of cards: "[H]e averaged nearly 50% [accuracy in guessing cards correctly instead of the expected 20%], and pulled off several uncanny streaks, such as guessing nine cards in a row. The odds of this happening by chance are about one in two million. Linzmayer did it three times." However, the Duke psychologist, before publishing his experimental results, tested the student some more -- quite a bit more. What happened? After two years of additional experiments with the student and the deck of cards, this happened: "[T]he student lost his spooky talent." The Duke psychologist (Joseph Banks Rhine) was "forced to conclude that the student's 'extra-sensory perception ability has gone through a marked decline.'"