The Lost Art of Fact Checking

If you were watching ABC’s Nightline this past weekend, you would have seen this segment on the new campus sexual assault documentary The Hunting Ground.  About 4 minutes into the segment, you would have gotten to the supposed FBI statistic I discussed in How To Lie And Mislead with Rape Statistics: Part 1.  Here is a transcriptions of the relevent part:

Annie Clark: It’s hard because there is such a stigma attached.  It is the only crime where we blame the victim.

Amy Robach (ABC News): And, there’s a notion out there, that there’s a high level of false reporting when it comes to rape. 

Annie Clark: The false reporting rate on sexual assault is 2-8% according to the FBI, which is the same rate, if not lower, than any other crime.

If you’ve read my linked blog post on the subject, you know that there is simply no FBI statistic that finds a 2-8% false reporting rate for rape.  In 1995, 1996, and 1997 the FBI did publish an 8% false reporting rate, but rather than being “the same rate, if not lower, of any other crime” here is what they actually found1:

1995

The “unfounded” rate, or percentage of complaints determined through investigation to be false, is higher for forcible rape than for any other Index crime. In 1995, 8 percent of forcible rape complaints were “unfounded,” while the average for all Index crimes was 2 percent.

1996

The “unfounded” rate, or percentage of complaints determined through investigation to be false, is higher for forcible rape than for any other Index crime. Eight percent of forcible rape complaints in 1996 were “unfounded,” while the average for all Index crimes was 2 percent.

1997

A higher percentage of complaints of forcible rape are determined “unfounded,” or found by investigation to be false, than for any other Index crime. While the average of “unfounded” rates for all Crime Index offenses was 2 percent in 1997, 8 percent of forcible rape complaints were “unfounded” for the same timeframe.

While this isn’t quite as bad as if Robach had used this statistic, it still represents pretty poor journalism in my eyes.  This was a pre-taped interview and Robach clearly knew ahead of time that they were going to bring up this stat as sets up Clark to deliver her line .  How long would it take a professional fact checker to find out that not only did the FBI statistic in question not exist, but the real statistic made rather the opposite point? 5 minutes? 10? Also of note is the fact that approximately 15 seconds before the above exchange occurred, Robach herself used a different sexual assault statistic2, so we know that ABC staff was already researching sexual assault statistics. Particularly in an era where it seems the media feels the needs to present even the most clear cut issues as though there are two equal sides, is it too much to ask a major news organization to simply point out when one their guests makes an argument based on a nonexistent statistic?

I have reached out to ABC News in order to obtain their policy on fact checking statistics used by guests during pre-taped interviews, but have yet to receive a response3.

  1. From Section II of the 1995, 1996, and 1997 Crime in the US reports
  2. The 20% sexual assault reporting rate for students that is found in this report
  3. Both of the email addresses I was able to find were inoperable, so I had to use the contact form available on their website.  Additionally I reached out to their two listed social medial contacts on Twitter, but did not receive a response there either

A Cut Too Deep: A case study in bias

If I were to create Francis Walker’s Rules of Using Statistics, Rule #1 would probably be this:

1. You are not allowed to use a statistic in an argument unless you have read the underlying research

This rule would also have a sub-part:

1a. Do not skip from the “Introduction” section to the “Results” section, you must also read the “Methods” section

In each of the posts I’ve written so far, bias has played a role, and by reading the “Methods” section of research you can get an understanding of what type of bias might be influencing a particular research study.  The other week I read an article on Ars Technica that reminded me of the first time I realized just how critical the “Methods” section really was.  What was said Ars article about? Digital Security? Space Travel? General Technology, perhaps?  Nope – it was about circumcision.

Read more…