Violence is often affirmed as a necessity when considering the expropriation of the means to production (and such) from those who through various methods, have privatized them and maintain such privatization through the forces of various institutions many anarchists refer to as the state. The so called middle class is defined differently depending on is using the term. Some definitions of the middle class are based on full-time, contracted, unionized, or otherwise fairly secure employment with an income large enough privately own ones home, stocks and bonds, are part of a nuclear family with two "primary income earners", etc. Other definitions of the middle class are based on capital: the small business owners, professionals, and others who have accumulated enough capital to not only own their means to shelter, transportation, and such but to also make a profit. This latter definition is middle class as petty capitalist who makes a relatively small amount of profit through the exploitation of the working class (wage labor). There are other definitions as well...
This is a very specific sort of violence that is considered in a very specific context of abolishing the private ownership of what everyone depends on for survival and certain standards of living. Violence towards the middle class (despite what definition is used) doesn't fit into this context. Working for petty capitalists (small businesses) and such is much different than working for the ruling class and expropriating those petty capitalists of their wealth would amount to... jack shit. The basic goal isn't to simply equalize the distribution of remuneration or income, but to end for good the very system of private ownership and remuneration that makes it possible for individuals (the ruling class) to accumulate so much of society's wealth that the majority are always indebted to them, subjugated by them, in other words ruled. The wealth that we produce in our societies depends upon what the ruling class owns and not what the middle class owns: the means of production (patents, copyrights, land, technology for mass production, architectural facilities, communications networks, and increasingly water, air, universities, etc.). The middle class tend to own social capital, knowledge capital, and other shit so they can demand a higher remuneration for their service labor (professionals) or they own store fronts, .com addresses and such as distributors of commodities (and the small forces of working class labor that do the actual packing, shipping, clerical work, etc.).
So it is basically... if those working for Mom and Pop Distributions expropriate that business or take it over or whatever the fuck, they would be merely looting the commodities produced by those working for the ruling class if they even could get away with it (and by that I mean simply taking all the shit since obviously the supply of products Mom and Pop Distributions sell would be cut off by ...you know who). If those workers somehow managed to increase market demand or otherwise keep Mom and Pop Distributions (now collectivized) running, that would be about the same as just starting up a co-operative business anyway. Since we're not living in an economy just chock full of small businesses, but one in which the total wealth of society has been accumulated by such a small class of people... that strategy would be a bit absurd.
As for violence related to racists, sexists, and other biggots ...I haven't ever read or heard an anarchist giving a shit what class that person is from or is currently still a member of. Violence against the police and those other professional enforcers of Capital is also a completely different context.