I downvoted this because I don't care that you don't like the responses. I also feel like you do deserve more of a response than that.
What do you mean by humane? Why should I care about being humane to someone who has wronged me? Even moreso, why is a humane response required to "mass murder, rape, serial killing?"
If the objection is that responses seem overly violent or cruel, I'm sorry but I just don't feel much sympathy towards that. If it is someone close to me that is raped or murdered, I very well might respond barbarically, and I would probably feel just fine about that. You are fixated on the worst of the worst. Certainly they are bad, but if we are looking at actual psychopaths, why not just kill them when they become a threat to us? In the absence of laws and judges and cops and prisons, that seems to me the most rational, if not humane response
If it is that they are arbitrary, well, yes. As our punishments for many crimes under the law. The fear of things devolving to "an eye for an eye" is based in a really different perspective on human behavior than the one I hold.
I get the sense you want a blueprint for how we anarchists would react more civilly in particular situations and I just can't and won't give you that, none of us can offer that up. We would (and do) deal with the messy shit as we need to, and it is different given lots of different factors.
Here are my perspectives on the three responses you touch on, and I am writing about them here not in some sort of utopian future, but how I see them playing out in the here and now. I am mostly writing in the context of discussing rape, of the three things you brought up, becasue I think there is a big leap between rape (a relatively common, and thus, to my mind, more important to address) phenomena, and mass murders or serial killings (which are both relatively rare):
Let the victim decide: First, victim is a role that could be problematized, as there is also an ideology of victimization (and of the absolute infallibility of the victim) that can develop around this. Most often, anarchists use words like survivor, which is borrowed from the world of DV support (to the best of my knowledge) and is also problematic, but is at least allowing the individual to move beyond merely being a victim.
If the survivor of an assault (rape, other physical assault, etc) needs to act in retribution, they probably will do so, often with the support of people close to them. It isn't my role to say what they can or can't do. Equally, it is by no means required of me that I actually play a role in whatever retribution this person seeks to enact (for example if I think that they are unbalanced, making it up, or I just don't like them). Given the reality of the rape culture we live in, at least in terms of sexual assault, I tend to err on the side of believing the survivor, but I can think of at least a couple times where it wasn't clear cut enough to feel like I could comfortably decide that.
Also, yes, this can lead to factionalism, and it can lead to all sorts of ugly shit, but anarchy is a beautiful idea, not necessarily beautiful in practice.
Exile: If an individual is a threat to a group's safety, exile or exclusion is a pretty sensible reaction. there is no guarantee that they won't come back and try to enact some sort of vengeance on the group, but that would then simply mean responding in an appropriate manner. Similarly, if someone has shown patterns of manipulative, violent or abusive behavior, they are people I don't necessarily want to be involved with.
Once upon a time exile was essentially the equivalent of a death sentence, but now people are able to pick up and leave for another place. The problem with this is that they might (and, in my experience often do) not use this as an opportunity to reflect on what they did to make a group that will accept all sorts of wingnutty people choose to exclude them, but instead use it as an opportunity to wipe the slate clean, and perpetuate their old patterns of fucked up behavior. To an extent the internet can serve as a means to communicate between disparate locations about predatory or sociopathic individuals, but it isn't perfect, and lends itself to badjacketing, manipulation by our enemies, and so forth.
Rehabilitation: I don't view people working on their shit in the context of rehabilitation the same way that the state would view rehabilitating a criminal. This is based in a very christian and very particularly progressive perspective that most contributors (at least to this site) would probably reject. The idea of rehabilitation normally includes a strange combination of improving the quality of the person in question and punitive punishment for being bad. What it ends up looking like is modern prisons.
I am not opposed to counselling. I've done it, I have friends and loved ones who have done it, and I have friends that are counselors or therapists. Sometimes it helps, sometimes it is a waste of time. If a person enters into such things sincerely, it *might* help them work through their internal stuff. The same with support groups, and so forth. Since we can't force someone to willingly participate in these things, you are right, the worst of the worst probably won't, only people who value their place in particular milieus or groups will find value in doing so. We can't change that.
Lastly, there is no anarchist-friendly word for crimes. A crime is a violation of a law, and if we are seeking to live outside of laws, those distinctions don't matter, and, alternatively, we are all potentially criminal for seeking to live so.