All political factions are our enemies. However fascists, with their desire towards total control of both the state and economic spheres, through both authoritarian measures and hyper-nationalism represent, arguably, the most virulent contrast to anarchy. they are, without any question, our enemy.
Which is not what you asked. You asked about the relationship to anti-fascism. there is an historic relationship: anarchists were active in struggles against the Fascists, Nazis, Franco, the Greek Junta, and many other fascist powers of the mid-20th century. In all of these places anarchists faced severe and widespread repression at the hands of the fascists, so yeah, we have some beef.
Additionally, the anti-fa movement has direct connections with the same factions of the left that anarchists do (NO!- I am not saying that the anti-fa's or the anarchists are the same as- or a part of- the left, just that there are connections, Jeeez!)- in particular the autonomen - which is where black bloc tactics and the related (if not the same) tactics of the anti-fa's developed from.
All the history and lineage doesn't matter though. Fuck the fascists. They need to be confronted and fought at every chance. Concerns about restricting their freedoms are hollow, in my opinion. I expect that those who would not respect my right to live free from their authority would attack me, and that their way of understanding the world is in perpetual conflict with mine. There is no reason I need to not act on the information I have (that being that they profess to be fascists - if they don't believe in fascism, they probably ought not claim it). certainly this is touchy- at what point am I just deciding to liquidate my enemies the same as they might? I can't draw you a line, but there is one.
One last thought- I would posit that confronting fascists (who, at least in the US, are often protected by the powers of the capitalist state, aka: the cops) with whatever tactics are appropriate provides a poignant piece of propaganda by (if not deed) action: Not only are fascists being confronted militantly and directly (confronting oppressive, domineering forces or individuals), but they are being confronted outside of the confines of legally prescribed and sanctioned counter-protesting. It doesn't ask the city to not issue a permit or for permission to confront what it hates. It expects that the cops will work to protect their kindred authoritarians. It is individuals coming together and acting for themselves, and each other to protect their freedom and autonomy.