Understanding the NICJR report

Given the frequency in which the NICJR report is citing in City Hall and Seattle political circles, you would be forgiven if you thought that copies of it were flying off the shelf. But reading the report (affectionately referred to as “Nick Junior”) is like teenage sex: a lot of people are talking about it, but almost no one is doing it. And yet it has taken a central role in the debate over the nature and pace of changes to 911 response in our city, so it’s worth knowing what it says — and how skeptical we should be of its conclusions.

As a component of Mayor Durkan’s 2020 “Executive Order to Reimagine Policing and Community Safety in Seattle“, SPD contracted with the National Institute for Criminal Justice Reform (NICJR) to “conduct an analysis of SPD calls for service for the period 2017-2019” to answer four questions:

  1. What are the characteristics of calls for service to which Seattle Police respond?
  2. What are the primary initiation sources for calls for service to which Seattle Police respond?
  3. How much time do officers spend responding to called for service?
  4. Which types of calls for service should be responded to by a non-SPD alternative?

The report has pages of tables and charts in answer to the first three questions, but let’s be honest here, the whole point of the report is question #4: an attempt to quantify how much of what SPD does in response to 911 calls could be handed off to civilian-based organizations, some combination of government agencies and community-based organizations.

Click to access attachment-3-seattle-calls-for-service-analysis-report-with-appendices-nicjr-june-2021-final.pdf

There is a short list of talking points from the report’s findings that are on auto-repeat in City Hall:

  • Nearly 80% of SPD’s 911 calls are non-criminal in nature, while only about 6% relate to felonies;
  • 49% of 911 calls to SPD should be handled by organizations other than SPD;
  • SPD has admitted that 12% of its 911 calls could immediately be offloaded to alternative responses.

The truth, as you may imagine, is more complicated and nuanced. So let’s dig in.

SPD gave NICJR three years of police dispatch call data (2017-2019), which seems to be the same dataset the department has posted on its website. NICJR presented a high-level breakdown of calls for service by their source of origin: 911, alarms, police officers noticing something and calling it in (called “on-view”), and people calling SPD’s non-emergency number. They also presented several cuts on the most common call types, looking at overall statistics as well as from each of the sources of origin, seasonal shifts, day of week, time of day, precinct, and a few other categories. It’s all in the report, and on its own none of it is particularly enlightening. Then they start to get to the heart of the matter: they break out the calls by “crime category”: misdemeanors, nonviolent felonies, violent felonies, and non-criminal matters. They found that almost 80% of the calls were non-criminal in nature.

They also looked at which call types led to the most arrests, and what percentage of those call types ultimately led to arrests.

NICJR has a standard four-tier model that it uses to decide whether a particular call for service should be reponded to by SPD, a “community emergency response network” (CERN), or both:

Tier 1: CERN dispatched only
· Event type: Non-Criminal
Tier 2: CERN lead, with officers present
· Event type: Misdemeanor with low potential of violence
· If CERN arrives on scene and determines there is low potential for violence and an arrest is unnecessary or unlikely, officers leave.
Tier 3: Officers lead, with CERN present
· Event type: Non-Violent Felony or an arrest is likely
· If officers arrive on scene and determine there is no need for an arrest or an arrest is unlikely and violence is unlikely, officers step back and CERN takes the lead.
Type 4: Officers only
· Event type: Serious Violent Felony or high likelihood of arrest

Based on the crime category and the likelihood of an arrest, they assigned every call type (SPD uses about 350 different call types to categorize its calls for service) to one of the four levels. This led to their conclusion that up to 49% of SPD’s calls could be handled without SPD sworn officers’ involvement, and another 24% should be led by a CERN with SPD officers in secondary, supporting roles.

Looking deeper at the tier assignments, they decided that 100% of the top ten officer “on-view” call types and 80% of the 911 and non-emergency call types were in “Tier 1”, which does not require an SPD officer to respond.

The NICJR report concludes that the city should move forward to design alternative response systems for the 73% of calls where a CERN can lead the response.

To begin, there are several issues with the data that NICJR used for this analysis, as detailed in the report:

  • the call data doesn’t differentiate between sworn and non-sworn response;
  • not all units are included, such as the civilian-based Crisis Response Unit;
  • there is a bit of noise as changes in SPD’s call-type taxonomy happen over time;
  • if an arrest happens after the end of a service call (perhaps even days later), that is not linked back to the originating call;
  • this gives us a breakdown of the calls, but not a breakdown of where SPD officers are spending their time, since different call types require a greater or fewer number of officers and a greater or fewer number of officer hours to resolve.

These are important issues not to be ignored, but on their own they don’t undermine the study. However, there is one more, much larger, issue that does call into question the validity of the whole analysis. Each call has two call types associated with it: the one assigned by the 911 dispatcher at origination, and the “final disposition” assigned at the resolution of the call for service. The NICJR team used the final disposition call type for its analysis, because it more accurately describes what actually happened during the call. But this divorces the analysis from reality, because 911 dispatchers don’t have the benefit of hindsight: they don’t make decisions about whether to send out sworn officers or civilian alternatives based upon the ultimate disposition of the call; they have to make it based on the limited information they have when the call comes in. This highlights a critical assumption being made by NICJR: that 911 dispatchers can accurately predict the final disposition of a call at the time it comes in. There are two levels where this is problematic: first, a quick look at the call data shows that a large percentage of the calls change call type between the initial assignment and the final disposition; and second, for call types such as “prowler- trespass” where 1,866 of 40,661 calls resulted in an arrest, dispatchers may not be able to distinguish between when an officer is and is not needed.

To illustrate the first problem, here is a snapshot of all of the calls in the first quarter of 2017 with a final disposition of “Disturbance – other”, which is in the top 10 for On-view, 911, and non-emergency calls and listed as “Tier 1” for all three. You can see the wide variability in originating call type:

Click to access disturbance-calls-Q1-2017.pdf

And here are all of the “Prowler – Trespass” calls; not only do they also have significant variation between initial call type and final disposition, but (in reference to the second problem above) there’s nothing apparent that clearly distinguishes the calls that result in arrests from the rest (though there may be other information that dispatchers have that might help).

Click to access prowler-trespass-q1-2017.pdf

Another example: NICJR categorized alarm calls (including banks, ATMS, schools, and businesses) as “tier 1,” meaning that they would only receive a civilian response at first. While we can grant that many of those calls are false alarms, 911 dispatchers have little ability to predict which ones are false and which are real.

This leads us to the two key questions we need to be able to answer in order to create a real alternative response system:

1. Can dispatchers accurately predict, given the information they have at the time of the call, whether an armed response will be required?

2. What happens when they are wrong?

Answering the first question requires a type of analysis well-known to mathematicians and computer scientists, called “Bayesian inference.” It’s used to build models for a wide variety of diagnostic systems, from healthcare to car repairs. Given information about a set of outcomes (e.g. diseases) and the characteristics we are likely to observe (e.g. symptoms), Bayesian analysis lets us flip that around and predict the outcomes based on the observations. In this case, with a collection of data on the criminal-type calls and the ones that led to arrests (or otherwise required an armed officer), can we accurately predict them from an arbitrary set of information given to a dispatcher?  There is no doubt that a Bayesian model could be built; however, until it’s tried, there’s no way to know whether it will be accurate enough to be useful in making dispatch decisions. It may turn out to be accurate enough for only a subset of call types; that is still useful enough to become a component of a solution, even if it means in the short term only some calls can be diverted to alternate responses.

The second question — what happens when the dispatch decision is wrong — is tougher. SPD was worried enough about that question that it asked a noted academic criminologist, Professor Geoffrey Alpert from the University of South Carolina, to review the NICJR report and provide his thoughts. Alpert gave two pieces of feedback: first, he agreed with NICJR that the tiered-response model is the right approach. Second, he said the analysis and assignment of call types to the four tiers is overly simplistic because it leaves out the notion of risk: what happens when the wrong response is sent to a call.

Alpert recommended that SPD look at parallel work that went into the creation of the Cambridge Crime Harm Index, an effort to improve crime reporting. It recognizes that simple tallies of the number of arrests for crimes doesn’t give a useful picture of crime trends, especially when a petty crime and a more serious, violent crime are counted equally. The Cambridge Harm Index creates a weighted count, assigning to each type of crime a weight corresponding to its seriousness; it uses as a proxy for seriousness the minimum sentencing length for someone convicted of the crime. So in a city where violent felonies are decreasing and misdemeanors are increasing, the Harm Index will likely go down recognizing that the overall burden of crime on the community has decreased.

Alpert is suggesting that a similar index could be used to quantify the risk of harm associated with a call, and also potentially the risk of harm if the wrong response is sent. It would conceivably capture the harm caused by a crime in progress where armed officers were not sent in response and couldn’t intervene to stop it; the potential danger to a civilian responder caught up in a violent incident; and the increased risk of a violent conflict that harms suspects or bystanders if an armed police officer is sent into a situation where his or her presence will escalate tensions.

In its written response to the NICJR report, the executive branch’s inter-departmental team looking at alternatives to policing claim that they are following Alpert’s recommendations and starting to research an appropriate “harm model” that could be incorporated into a tiered dispatch process. It’s important to emphasize, however, that this is breaking new ground: there are no models to copy on how to build such a harm index for this context.

In the meantime, the interdepartmental team has acknowledged that about 12% of SPD’s 911 calls, related to “person-down” calls and lower-priority wellness checks, “can and should be explored for alternative responses starting now.”

Based on an initial cooperative analysis with SFD and CSCC, SPD agrees that 12 percent of calls, can and should be explored for alternative responses starting now. These are calls where we know the risk of harm is very low. This reduction does not equate to – and is not reflective of – the percent of service hours SPD currently spends on calls that could ostensibly be offloaded. These calls include person-down calls and priority three welfare checks which accounted for over 23,000 service hours between 2017 and 2019.

This led to the proposal for a “Triage One” pilot organization that Mayor Durkan included in her budget. The team would be stood up and plugged into the 911 dispatch process over the course of 2022, though it would remain a limited-scale pilot and would not be able to offload the full 12% of calls envisioned until it is proven out and city-wide expansion is funded.

To meet the immediate need for alternative responses, SPD, SFD, and CSCC are proposing a new pilot aimed at helping 9-1-1 triage non-criminal calls with no imminent health concerns. These triage teams will be a civilian forward alternative to fire or police, housed within the Mobile Integrated Health (MIH) program. Triage teams will respond directly to wellness check calls as identified by 9-1-1 dispatchers through collaboration with SFD and SPD. We will work with community organizations to hire people who bring not only expertise in outreach and behavioral health, but also lived experience and a tangible connection to the communities they will serve. They will be equipped with radios to request a police or EMS response as needed. It is expected that SPD will be requested only for criminal situations or to assist with potential violence or active suicidality. On the back end, the teams will be provided with a case manager able to follow up on client referrals and service connections and reduce the chance that they are called in the future.
This triage response team pilot is the first step of many to truly reimagine public safety in Seattle. SPD and The Mayor’s Office are committed to continuing this work to increase the number of available alternative responses and reduce the number and types of calls that are unnecessarily responded to by SPD patrol officers- those with no immediate safety or health risk.

This complicated picture looks very different from the simple talking points on the NICJR report we hear. If 911 dispatchers are clairvoyant, they could send civilian alternative responses to the appropriate set of 911 calls in place of SPD officers; unfortunately they aren’t, and they can’t — yet. And as of now the city has no model for the risk of harm it may be creating for responders and Seattle residents by trying to offload some calls. There is well-founded optimism for what lies ahead, but we’re not there yet. In the meantime, however, there is also great impatience to start offloading SPD calls now, both for the optics of showing meaningful progress on elected officials’ promises to downsize SPD’s role, and also to help address the staffing shortage in the police department.

The good news is that it looks like the budget for the Triage One team is surviving the Council’s budget process, as is SPD’s technology investments to start to build a harm index. So in the next year, we may see the first step in offloading that 12% of calls from SPD, and some groundwork laid for a much larger restructuring of 911 response in Seattle.

I hope you found this article valuable. If you did, please take a moment to make a contribution to support my ongoing work. Thanks!

Article Source: https://sccinsight.com/2021/11/11/understanding-the-nicjr-report/