Soon after the Charleston church shooting, FBI Director James Comey argued that Dylann Roof’s actions probably weren’t terrorism, eliciting criticism that law enforcement employs a double-standard: It calls violence attributed to Muslims terrorism, but refuses to attach that label to attacks in the name of right-wing ideology. On the other hand, Roof, the young white man alleged to have carried out the attack, has been charged with nine counts of murder and if proven guilty will face severe punishment. So what does it matter whether we call the attack terrorism?

How a society describes violence is important. Calling an act of violence terrorism signifies that it is part of a broader pattern that requires attention beyond ordinary crime fighting. Calling Roof’s actions murder rather than terrorism underplays the role of his evident racist motives and avoids questions about the prevalence of racism in our society. Similarly, the killing of three American-Muslim students in North Carolina earlier this year, was quickly dismissed as a random act of violence with little attention paid to whether it was part of a broader pattern of violence directed at minority populations. In contrast, a hatchet attack on New York cops by a Muslim convert whose social media presence was described by police as “anti-Western, anti-government and in some cases anti-white” and who had watched videos of beheadings by the Islamic State, was quickly and firmly laid at the door of extremist Islamic ideology.

Under federal law, you don’t need an ideology to be a terrorist. Violent acts are considered terrorism if they appear “intended to intimidate or coerce a civilian population” or to influence government policy or conduct. Even if, as the FBI Director suggested, a political motive is needed, there is sufficient evidence to call Roof’s actions terrorism. In addition to his Facebook posts and pictures, he is reported to have told witnesses that he went to the church to “shoot black people” because they “rape our women” and are “taking over our country.”

Upgrading the Charleston attack to terrorism may encourage law enforcement to pay greater attention to the risks posed by white supremacist groups and their ilk. As the media seems to have only recently discovered, terrorism doesn’t necessarily come with the prefix “Islamic.” According to a study by West Point’s Countering Terrorism Center, in the decade following the 9/11 attacks, right-wing violence killed 254 people in the United States. During roughly the same time period, terrorism plots by American Muslims caused a total of 50 fatalities. A recent survey by the Police Executive Research Forum and Triangle Center on Terrorism and Homeland Security found that state and local law enforcement personnel are almost twice as worried about right-wing and anti-government terrorism as they are about the threat from al-Qaeda types.

But diversifying the threat portfolio won’t address the problems that American-Muslims face. Too often, calling something terrorism leads to a politicization of how we treat violence, as well as actions that are only loosely connected with violent ideologies. Violence by people from minority groups is hastily labeled terrorism and often the broader group pays the price in surveillance and suspicion.

American-Muslims know this only too well. They are regarded as inherently suspicious by their own government and their fellow-citizens. The New York City Police Department has for years conducted a surveillance program aimed solely at the city’s Muslims residents, creating maps of where they live, eat, and pray, and infiltrating mosques and community groups to report on their everyday conversations about their families and their faith. The FBI employs similar tactics, using the guise of community outreach to create dossiers on Muslim religious and civic leaders. Growing American hostility to Muslims manifests itself in opposition to the building of mosques, the trumped-up anti-sharia movement, increasing levels of hate crimes against Muslims, and employment discrimination. Ritual burnings of the Muslim holy book have been replaced by competitions to draw the prophet Muhammed, acts that some Muslims believe are forbidden by their religion and nearly all consider insulting.

Against this backdrop, it is important to address the glaring disparity between the characterization of attacks supposedly inspired by different ideologies. But that’s not enough. As a society, we need to get away from the politics of violence and take a clear-eyed look at the range of threats that we face. We should evaluate dangers based not biases and perceptions about particular racial, religious, or ethnic groups, but on facts about actual attacks and the seriousness of purpose and operational capacity of those who seek to harm Americans. Our response needs to be calibrated to these realities and not sweep up innocent people who have no connection to violence but who we find vaguely threatening. Only then will we have a national security policy that works to protect the safety of all Americans.