Subtle Inaccuracies and Reader Trust

The most recent Vanity Fair has a long piece on Facebook and their new algorithims and processes for serving better ads. In it, there’s this paragraph:

Moreover, the company has created a computer framework that allows it to constantly revise the Facebook site. At any time of day, hundreds of different versions of Facebook are running on the Internet—with a changed color here, a moved button there—and the user response to each variation is measured. And the same is done with advertising. Through what executives call A/B tests, a company can run different ads at the same time, have Facebook engineers measure the response, then ditch whichever ones had less of an impact.

Vanity Fair is a general interest publication, so it’s fine that they explain A/B tests to the uninitiated. But the piece loses credibility with a single phrase: “…what executives call A/B tests..” Wait, did Facebook execs invent A/B tests? No way. The sentence should have read: “Through what’s called A/B tests…” so as not to imply that A/B testing is in any way unique to Facebook.

It’s a tiny word thing. But it caused me pause, and made me not trust the rest of the article in terms of insight. To be sure, had the thrust of the piece been a profile of an exec or an examination of the company’s public policy efforts or any number of other things — it’d be a forgivable slip up. But it in fact centers on the company’s ability to serve targeted ads and monetize user behavior, and the A/B testing point is held up as evidence of innovation. And the author subtly overstates the newness and interestingness of that point, which conveniently supports his broader thesis. I ask myself, “What else could he be getting wrong?” and switch from reading to skimming.

10 Responses to Subtle Inaccuracies and Reader Trust

  1. Michael Cohen says:

    Yep. I stopped reading at that phrase.

  2. DaveJ says:

    In my view this example is less about reader “trust” and more about the lack of domain knowledge of the journalist. It simply demonstrates that the writer does not know that this is a term of art in the business. Thus he is relying on what he learns from the interviews to report, which means (first) as you point out he is unlikely to provide any new insights directly and (second) the reporting is subject to manipulation by Facebook since that’s probably the main source of information. However, there is the possibility that some techniques that Facebook is using could be described in the article. In any case, what it means is that it’s an article written for people who do not already know what A/B testing is, just like the journalist.

    By the way, you spelled “algorithms” incorrectly. I’ll let that one slide as a typo. ;-)

  3. I agree this example is less about reader “trust” and more about the lack of domain knowledge of the journalist, as DaveJ says.

    I had a visceral reaction to your own remark about subtly overstating a point to conveniently support a broader thesis, however. I have a similar response to the writing of many of your friends and mentors, especially that of such luminaries as Arnold Kling and Tyler Cowen. I don’t trust the motives of anything they say, even if I should happen to agree with a particular point.

    This is only roughly analogous, but I read a recent post at Kling’s new blog, linked to by his partner-in-crime Cowen (one does need to keep abreast of what the enemy is up to), in which he uses rather weaselly language to slip in an assertion that “Probably more borrowers were “victimized” by Freddie Mac, Fannie Mae, and FHA than by Wall Street” in a screed ostensibly about housing and wealth destruction.

    It may sound innocuous, but this Big Lie, promulgated in the right-wing disinformation campaign waged by Wall Street and its allies to deflect blame for the financial crisis to Fannie Mae, Freddie Mac and the Community Reinvestment Act, has been thoroughly discredited – yet Kling and his like keep resurrecting the zombie meme that federal affordable housing policies were responsible for the proliferation of high-risk mortgages over the last decade.

    This sort of skullduggery committed by a credentialed economist (one who was also a senior economist at Freddie Mac from 1986 to 1994) is a far more egregious violation of the implicit trust more naive readers might place in his writing than any slip-ups in “tiny word things” a journalist contributing to Vanity Fair might commit, short of outright libel or slander.

    PS:
    My God, your comments are being overwhelmed by spam. It’s getting more and more difficult to find the legit responses in the comments RSS feed.

    • Ben Casnocha says:

      Marked a whole bunch of comments as spam — hopefully that will get WordPress to block them going forward. Sorry and thanks!

  4. Dan S says:

    “the lack of domain knowledge of the journalist”

    I’m not sure what’s worse: the lack of domain knowledge or the journalist deliberately taking the position of lacking the knowledge to emulate the presumed lack of knowledge of the reader. How many times have you seen a competent, knowledgable journalist suddenly go “stupid” and play the part of the (presumed to be) ignorant reader/viewer?

    Ignorance vs dumbing-down should be more of a false choice than it is.

  5. LemmusLemmus says:

    Dave, Vince: Surely if the journalist displays ignorance of his topic, this should decrease the rational reader’s trust in the quality of the (rest of the) article?

  6. An on point quote from Michael Crichton’s 2002 speech, “Why Speculate?”:

    Media carries with it a credibility that is totally undeserved. You have all experienced this, in what I call the Murray Gell-Mann Amnesia effect. (I call it by this name because I once discussed it with Murray Gell-Mann, and by dropping a famous name I imply greater importance to myself, and to the effect, than it would otherwise have.)

    Briefly stated, the Gell-Mann Amnesia effect works as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward-reversing cause and effect. I call these the “wet streets cause rain” stories. Paper’s full of them.

    In any case, you read with exasperation or amusement the multiple errors in a story-and then turn the page to national or international affairs, and read with renewed interest as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know.

    That is the Gell-Mann Amnesia effect. I’d point out it does not operate in other arenas of life. In ordinary life, if somebody consistently exaggerates or lies to you, you soon discount everything they say. In court, there is the legal doctrine of falsus in uno, falsus in omnibus, which means untruthful in one part, untruthful in all.

    But when it comes to the media, we believe against evidence that it is probably worth our time to read other parts of the paper. When, in fact, it almost certainly isn’t. The only possible explanation for our behavior is amnesia.

  7. Tim says:

    I would guess that the phrase was the result of the editorial process – that a top editor, line editor, fact checker along the way said they needed to attribute A/B testing to someone, that the writer fought back, and the phrase is a compromise. Or it could be the writer. But I doubt it.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>