“Fatally Flawed” — Internal Britannica Review Tackles Nature Methods
Below is a letter that Encyclopedia Britannica sent out today to some of its customers, in response to the December Nature article comparing the accuracy of articles in Wikipedia and Britannica. A more detailed review of the Nature study, including responses to each alleged error and omission, is linked from the front page of www.eb.com; you can also see an HTML version of the review here (thanks to Ben Yates).
that claimed to compare the accuracy of the online Encyclopædia Britannica
with Wikipedia, the Internet database that allows anyone, regardless of
knowledge or qualifications, to write and edit articles on any subject.
Wikipedia had recently received attention for its alleged inaccuracies, but
Nature’s article claimed that Britannica’s science coverage was only
slightly more accurate than Wikipedia’s.
Arriving amid the revelations of vandalism and errors in Wikipedia, such a
finding was, not surprisingly, big news. Perhaps you even saw the story
yourself. It’s been reported around the world.
Those reports were wrong, however, because Nature’s research was invalid. As
our editors and scholarly advisers have discovered by reviewing the research
in depth, almost everything about the Nature’s investigation was wrong and
misleading. Dozens of inaccuracies attributed to the Britannica were not
inaccuracies at all, and a number of the articles Nature examined were not
even in the Encyclopædia Britannica. The study was so poorly carried out and
its findings so error-laden that it was completely without merit.
Since educators and librarians have been among Britannica’s closest
colleagues for many years, I would like to address you personally with an
explanation of our findings and tell you the truth about the Nature study.
Almost everything Nature did showed carelessness and indifference to basic
research standards. Their numerous errors and spurious procedures included
the following:
* Rearranging, reediting, and excerpting Britannica articles. Several
of the “articles” Nature sent its outside reviewers were only sections of,
or excerpts from Britannica entries. Some were cut and pasted together from
more than one Britannica article. As a result, Britannica’s coverage of
certain subjects was represented in the study by texts that our editors
never created, approved or even saw.
* Mistakenly identifying inaccuracies. The journal claimed to have
found dozens of inaccuracies in Britannica that didn’t exist.
* Reviewing the wrong texts. They reviewed a number of texts that were
not even in the encyclopedia.
* Failing to check facts. Nature falsely attributed inaccuracies to
Britannica based on statements from its reviewers that were themselves
inaccurate and which Nature’s editors failed to verify.
* Misrepresenting its findings. Even according to Nature’s own
figures, (which grossly exaggerated the number of inaccuracies in
Britannica) Wikipedia had a third more inaccuracies than Britannica. Yet the
headline of the journal’s report concealed this fact and implied something
very different.
Britannica also made repeated attempts to obtain from Nature the original
data on which the study’s conclusions were based. We invited Nature’s
editors and management to meet with us to discuss our analysis, but they
declined.
The Nature study was thoroughly wrong and represented an unfair affront to
Britannica’s reputation.
Britannica practices the kind of sound scholarship and rigorous editorial
work that few organizations even attempt. This is vital in the age of the
Internet, when there is so much inappropriate material available. Today,
having sources like Britannica is more important than ever, with content
that is reliable, tailored to the age of the user, correlated to curriculum,
and safe for everyone.
Whatever may have prompted Nature to do such careless and sloppy research,
it’s now time for them to uphold their commitment to good science and
retract the study immediately. We have urged them strongly to do so.
Nature responded with a polite but firm declination.
sj
March 24, 2006 @ 1:33 am
Here’s a response from Nature: