Great question, but also a huge one. I will try to keep my response as succinct as I can.
As funkyanarchy commented, agriculture seems inherently tied up in civilization. Certainly, the delayed return inherent in agriculture is tied up with the sublimated desires necessary for civilized work (slave) life – and recent anthropological findings at Goebekli Tepe do indeed suggest that religion (alienated value and meaning) occurred just a few hundred years before the emergence of plant and animal domestication.
Because agriculture involves so much drudge work, it is very much tied up in slavery – as one example, the Sumerian (first agricultural empire) word for foreigner is the same as the word for slave. Avoiding unpleasant work is a powerful incentive for slaving, after all. And the depletion of soil from agricultural monocropping means that expansion in some form (whether literally taking more land or taking synthetic fertility from elsewhere in the form of fertilizers) is necessary for agriculturists – hence, you have war, which in turn ensures a steady supply of slave foreigners, and so on. James C. Scott points out that, on the macro-scales of time and space, foragers are really the sedentary ones whereas agriculturists always need to be moving into new spaces.
Permaculturist Toby Hemenway points out that once you have agriculture, you have a stored surplus. And once you have a stored surplus, you need a lord to dole it out, bureaucrats to measure it, soldiers to defend it, slaves to create and maintain it, an ideology to rationalize the whole process, and so on – in other words, you have civilization.
More basically, in whatever greenwashed form, agriculture is fundamentally ecocidal – it requires the destruction of nonhuman habitat, the loss of topsoil, a reduction in biodiversity, and increasingly absurd and destructive means of combating non-domesticated organisms (pests) who are trying to rebalance their local ecology by attacking domesticates. Organic agriculture does nothing whatsoever to change this fundamental problem – hilariously, in fact, organic farmers often have even worse soil health on their farms than do the “conventional” ones.
So, for anti-civilization anarchists (like me), a critique of and hostility toward agriculture is essential.
In response to dot and funkyanarchy's comments asking whether all gardening is agriculture, I would reply with a hearty no. The image of a pure forager or hunter-gatherer is an idealized, anti-ecological fiction – every organism (including non-animals) does a kind of /gardening/, that is, they all attune their environments to better support them, whether by spreading seed or altering soil flora with their shit, broadcasting secretions that poison or support creatures around them, and so on. So, the question is not whether to manipulate one's environment, but how it is done. Forest gardening – an ancient and diverse set of practices employed by peoples the world over (off the top of my head, I know of cases in what are now called North America, Central America, South America, West Africa, and Central Asia) – is a way of manipulating one's environment that allows the human organism to behave like most organisms behave: in dynamic equilibrium with one's local ecology, minimizing self-alienated activity (work), and increasing biodiversity by creating localized, moving disturbances in so-called climax forest conditions. Correspondingly, it tends to create and reproduce anarchic, rather than hierarchical, social conditions.
Being an anti-civilization anarchist, for me, means making as a core part of my theory the recognition that our subsistence is always in a reciprocal relationship with our social conditions and belief structures.
