The murmur of the snarkmatrix…

August § The Common Test / 2016-02-16 21:04:46
Robin § Unforgotten / 2016-01-08 21:19:16
MsFitNZ § Towards A Theory of Secondary Literacy / 2015-11-03 21:23:21
Jon Schultz § Bless the toolmakers / 2015-05-04 18:39:56
Jon Schultz § Bless the toolmakers / 2015-05-04 16:32:50
Matt § A leaky rocketship / 2014-11-05 01:49:12
Greg Linch § A leaky rocketship / 2014-11-04 18:05:52
Robin § A leaky rocketship / 2014-11-04 05:11:02
P. Renaud § A leaky rocketship / 2014-11-04 04:13:09
Jay H § Matching cuts / 2014-10-02 02:41:13

'Pedia Still Astonishingly Awesome
 / 

wikitanica.jpg

Many of you may have already caught this Nature article posted on Boing Boing. Nature conducted a peer review of 42 entries from Wikipedia and the Encyclopedia Britannica. The results:

Only eight serious errors, such as misinterpretations of important concepts, were detected in the pairs of articles reviewed, four from each encyclopaedia. But reviewers also found many factual errors, omissions or misleading statements: 162 and 123 in Wikipedia and Britannica, respectively.

I’m pretty darn awed by that.

If you’ve been watching Romenesko’s letters this week, you might have caught Karen Heyman’s letter about Wikipedia’s problems. A snippet:

Unless you already know a field, you can have no idea that an apparently definitive entry presents only one side of an ongoing fight between specialists. That it may be changed, and changed back again, hardly helps matters. This, btw, is the best explanation as to why simply sitting back and saying, “It’s okay now, it’s changed,” ultimately would not have worked for Seigenthaler. Chances are high that later somebody would have come along to “fix” the correction.

Wikipedia is a fantastic idea, a wonderful service, with entries that often reflect great effort and care. Unfortunately, inevitably, as it’s grown, the flaws built into its original design have become more obvious. Egalitarian editing may be a noble goal, but the reality is that if Wikipedia is to truly fulfill its promise, it needs a way to vet contributors, to let users know whether an entry on neuroscience was written and edited by a senior professor, a student who just took Psych 101, or a layperson who’s paraphrasing an old issue of Scientific American. Certainly prankster Brian Chase’s initial belief that Wikipedia was a joke site says a great deal about how some of its entries appear to the general public. If Seigenthaler’s complaint actually leads to more accountability, far from hurting Wikipedia, he may ultimately have saved it.

I’ll cross-post my reply to Ms. Heyman below:

If Karen Heyman’s letter were a wiki, I’d edit the beginning of her concluding paragraph to read: “The newspaper is a fantastic idea, a wonderful service, with articles that often reflect great effort and care. Unfortunately, inevitably, as it’s grown, the flaws built into its original design have become more obvious.”

Why is it that people, especially journalists, always seem to propose fixing Wikipedia’s problems by dispatching more editors to the site? News organizations are rife with editors, yet I’d trust Wikipedia’s coverage of most complex science and tech issues at least as much as your average news report. Which is to say, cum serious grano salis.

Wikipedia’s problem is not that it has become more (or more obviously) flawed as it’s grown, as Heyman says. But it has become such an incomparably useful resource for so many that its flaws are more regrettable. Look back two or three years, and you’d find it informative and accurate in spots, but plainly lacking in key areas. Today, it’s astonishingly comprehensive, with standards of accuracy and balance that very often match or exceed media that are far more established, and a regard for transparency that would put any news outlet I know to shame. That growth couldn’t have happened if Wales and co. had a system in place to vet contributors. Such a system couldn’t scale.

All that is not to dispute the problems that Heyman points out. But Wikipedia is a strange new beast, and old-school measures aren’t going to cut it.

One greater problem that the Seigenthaler incident brings to light is our collective lack of basic media literacy. A prankster like Brian Chase has tons of options to introduce false information into an official-looking public record. It’s a very simple matter to create a fake CNN mockup and disguise a link to it. If Chase’s friend could be shocked by a random sentence on a Web page featuring the words “edit this page” in bold type across the top, chances are it wouldn’t have taken much to fool him.

We should continue thinking of ways to improve Wikipedia; it’s a phenomenal public resource. But we must turn our attention to educating people how to evaluate all sources of information as the amount of it proliferates.

By the way, don’t miss this tidbit from the Nature article:

Wales also plans to introduce a ‘stable’ version of each entry. Once an article reaches a specific quality threshold it will be tagged as stable. Further edits will be made to a separate ‘live’ version that would replace the stable version when deemed to be a significant improvement. One method for determining that threshold, where users rate article quality, will be trialled early next year.

This is a fantastic idea that Wales has been discussing for a while now. I imagine it will work in much the same way the featured article process does today.

When editing Wikipedia becomes more mainstream, especially within the academic community, it will be such an amazing project. A solitary tear runs down my cheek, just thinking about it.

December 14, 2005 / Uncategorized

8 comments

I really like Wales’ ideas. I’ve been thinking for a while now that it would be great to download a snapshot of Wikipedia, upload the mirror to a secured site, and unleash a team of editors and fact checkers onto that, say once a year. So you can either check the uptotheminute live and lovely Wiki, or, if you’re a bit less intrepid, go back to the last edited/factchecked version that’s held on a secure different server. Don’t like their take? Make your own team.

The other thing that a stable vs. live distinction within Wikipedia might do is remind people that, unlike Brittanica, a live Wikipedia entry isn’t ostensibly final, authoritative, etc., at least in any “mission-critical” sense. Wikipedia’s virtues are different from Brittanica’s. Every entry is a beta entry. Part of the problem in the Seigenthaler case was the fact that his entry appeared on sites that mirrored Wikipedia, apparently without reference to this. And that the ethoi, practices, and expectations of the whole “open-source” thing are only now trickling down to most of the rest of the world. It’ll get there, though.

Just wanted to say I loved your opening volley at the beginning of your Romenesko-letter. I feel like you have set up a little cottage industry in knocking down assumptions about the usefulness of the traditional printed newspaper.

Also, re: Saheli’s point… in the world of open-source software there is the concept of a ‘forked’ codebase — I guess it happens when some people want to go in a totally different direction w/ a program or something. Part of Wikipedia’s strength (I think?) is that things DON’T get forked, of course — they get hashed out — but all the same, it might be interesting.

Recently I read something from Wales that said there are ~1,500 really solid Wikipedia editors. I think that’s interesting. Show of hands — how many of us have edited the ‘pedia? I did, but only once.

Howard says…

I think Matt’s comparison with newspapers is apt, and telling. “First rough draft of history,” and all that. Obviously, there’s a lot of defensiveness involved in how people react to Wikipedia.

But I do have one nit to pick. Everybody seems very impressed that the study found Wikipedia was roughly as accurate as Britannica. I’m a newsroom guy, so check my math on this:

The article I read says “But reviewers also found many factual errors, omissions or misleading statements: 162 and 123 in Wikipedia and Britannica, respectively.”

Doesn’t that mean that 750 errors in Britannica = 1,000 errors in Wikipedia?

Isn’t that a hell of a lot of difference?

No, you’re right, Howard — that’s 30% more errors in Wikipedia.

But I’ll bet a lot of Wiki-critics would have guessed an order of magnitude — or more — difference in error rates at the outset, so 30% probably still beats a lot of expectations.

I’ve edited a couple-three Wikipedia entries. What Wales is proposing would be different from typical open-source forking in that you wouldn’t have two different live versions of an entry, you’d have one frozen, peer-reviewed version, and one live version available at any given time.

I think it’s an awesome idea for content in general. Imagine if news sites allowed people to edit wiki versions of their stories (while still being able to access the version published in the prior day’s paper).

Howard, to answer your question, I added this comment over at Bayosphere:

I think making such a strict comparison on the strength of this study would be difficult. Another way to parse the data we’ve been given is that Britannica features an average of 3 errors per entry, while Wikipedia features an average of 4. But even that data is somewhat useless when you consider that the scope of each entry may be entirely different; this Wikipedian estimates that Wikipedia entries include almost triple the amount of information in their Britannica counterparts, on average. Parsing the data that way makes Wikipedia look much more accurate than Britannica.

A much broader study would have to be done to get any sort of concrete picture about the relative accuracy of Wikipedia and Britannica. But as an anecdotal check on our assumptions, I think the study packs a wallop.

Howard says…

Ah, the glories of lowered expectations. Since we thought it would be much worse, simply being 30 percent less accurate is a big win.

Far more important, it seems to me, is Matt’s observation about the potential power of displaying parallel versions with separate criteria for alteration and editing. That seems like a great idea. It’s humbling to think about how much better almost all my journalism would have been had that kind of fact-checking been available (especially before the fact, or simultaneously, as with the Star Tribune’s “Big Question” project).

I for one wholeheartedly embrace the end of pretended omniscience; I was never that comfortable with it anyhow. I believe I may have mentioned before what one of my heroes, Bertrand Russell, had to say when asked if he would be willing to die for his ideas: “Well, of course not. I may be wrong.”

I wish an independent content review like this was performed on some other established publications. We all know The New York Times has problems, but we generally refer to it as an impeccable resource, as accurate as anyone out there. But almost everyone I know who’s been featured in a New York Times article has complained of some mistake: a misspelled name, a truncated quote, a mistaken detail.

It’s not that surprising in this study to hear that Wikipedia’s science entries each contain an average of 4 errors, but it was surprising that the vaunted Encyclopedia Britannica’s entries each contain 3. What would we find if we probed further?

The snarkmatrix awaits you

Below, you can use basic HTML tags and/or Markdown syntax.