Statistically Sneaking

Should we believe crime statistics?

The U.S. crime rate is still dropping, FBI data shows (msn.com)

I am skeptical of the claim of the attached headline. It’s not because I don’t trust data. It’s because I understand data, and I don’t trust reporters to be skeptical enough. Statistical calculations can be burdensome, and interpretations can be misleading. Increasingly, the most useful statistical tools have become accessible to anyone using Excel. This is a good thing, unless the users don’t understand the limitations of the underlying methods.

In my most typical experiences in manufacturing, the results of my earliest statistical analyses of anything would make no sense. After performing rigorous and time-consuming computations, things that should be related wouldn’t be. Experimental adjustments would not have the expected outcome. This was because the things that were being measured were complex. There was a compounding of interactions and interferences that defied prediction until they were sussed out and controlled or eliminated. This isn’t unusual. Much of manufacturing work is in this arena – working the physical mechanisms and measurement systems until the data is actually useful and outliers can be attributed.

Among the worst scientific hazards is to rely on statistical results before the underlying data is meaningful. This, I fear, is where the public may currently be misled on crime.

Whenever I read a claim such as the captioned headline, I test it against my personal observations. By this, I do not mean media claims or media-covered events. I mean things that I personally witness. I am aware of an increasing number of property crimes in my community that never result in convictions. I also know more people who have changed their behavior based on the fear of crime than at any time in recent history. These observations cover a period of years, and you can’t draw a trend line through them. It could still be a media-led perception, as opposed to reality. Yet it still gives me pause because it doesn’t match the headline.

The article acknowledges the data is preliminary. The FBI compiles and publishes nationwide crime statistics in September-October of the following year.

“It suggests that when we get the final data in October, we will have seen likely the largest one-year decline in murder that has ever been recorded,” said Jeff Asher, a former CIA analyst who now studies crime trends.

That may be the case, and I hope he’s right. He is much more focused and informed than I can hope to be. Nevertheless, similar claims were made comparing 2022 to 2021. Here, I refer to the FBI release of crime statistics for 2022, reported in October, 2023 (FBI Releases 2022 Crime in the Nation Statistics — FBI). That report includes the following summary:

The FBI’s crime statistics estimates for 2022 show that national violent crime decreased an estimated 1.7% in 2022 compared to 2021 estimates:

Compare that to the conclusions offered in the Department of Justice’ National Crime Victimization Report (“NCVS”) (Criminal Victimization, 2022 (ojp.gov)), published in September, 2023, which reports on 2022. Look at it. Read it. Compare categories. If you don’t care to, suffice it to say it includes this unambiguous summary statement:

The rate was higher for all types of violent crime in 2022 than in 2020 and 2021.

So, which is it? Both reports are DOJ products. One matches my personal observations, and one does not. Assuming the reports are based on the same data, shouldn’t they say the same thing? What should we expect from this coming September’s NCVS report, and why is nobody reporting on that?

But they’re not based on the same data (emphasis mine): “The NCVS collects information from a nationally representative sample of U.S. households on nonfatal crimes, reported and not reported to the police, against persons age 12 or older.” (NCVS Dashboard: Home (ojp.gov)). The survey has been consistently performed since 1993. Conversely, the FBI report is extrapolated from the Crime Data Explorer (CDE (cjis.gov)), based on reports submitted from law enforcement agencies. The data for these reports are in transition. The traditional Summary Reporting System (SRS) is being replaced with a more comprehensive National Incident-Based Reporting System (NIBRS), as of 2021, and the transition is not yet complete (NIBRS — FBI).

This situation is reminiscent of my experiences with any early statistical interpretation in manufacturing process. It doesn’t make sense, yet. The data are not consistently representative of what we are attempting to measure. So the first work is to ensure the data has integrity. Crime measurement is very complex. The use of transitional data and the evolving definitions of crime make it more so.

And yet, the captioned article reports the most suspect data as fact, including snarky points about how the public misperceives its own risk. This highlights that regarding public perception, there is inherent conflict relying on reports from enforcement agencies. As an example, last year, I was in a car accident. Another car made illegal use of a turn lane and ran into my car. We both pulled to the side and once we determined that there were no injuries, we contacted the police to initiate the reports for insurance. We were informed that the local police were no longer dispatched for non-injury accidents. No citations were issued. So, if the local police are asked if there was a crime, their answer would have to be “no.” But if you ask me if I was the victim of a crime, my answer would be “yes.” Which of these answers best compares to history? How much of this ambiguity permeates other crime data? Which is more relevant to the public? In my opinion, it is clearly the victims’ experience.

Personally, I hope the claims of the article are true. Given the uncertainty, I will believe it when my observations and the victims survey corroborate it.