I guess it would depend on the kind of anarchism one is speaking of. If your idea of anarchism relies on the conversion of society – then it matters. If it doesn't give a fuck about society – then, no, its doesn't matter.
If you are focused on being the 'greatest hope for humanity', then the misrepresentation of anarchism in society would be problematic for you and your agenda. That line of thinking relies on knowing what is best for a large group (presumably /all/) of people, and therefore, in order to fulfill itself would need to work to ensure that said 'best' is applied to everyone's life. So, if everyone misunderstood your grand intentions of bettering the world for them, that may be problematic if they don't clearly understand your ideas and reasoning.
However, this is not the only line of anarchist thought out there (thank goodness).
For some anarchism is, in fact, about destruction. It is about destroying and negating what now exists, including the society you say may be misrepresenting them. In which case, their enemy's opinion of them may not mean much, without regard to whether it is right or not.
Of course, there are millions of variations and different ways of viewing anarchy. So, there will be appropriately varying levels of reliance upon society's opinion of them.
And, utopic visions aside, the widely accepted view of anarchism can be annoying, at best, when it comes to court cases. Society is generally less than sympathetic, which often lands anarchists with absurd sentences. In that case, it may matter, but I don't think that is really what you are asking about.