ah yes, i believe i have a thought or two about this! (lol)
i've been a part of various study groups over the decades, most notably one that has been on-going for over ten years. and i've thought a lot about what enables it to continue.
a. a critical mass of people have to do the reading, but it can't be obligatory that everyone do it. this means having a core of (three? four?) people who come regularly, do the reading, and will talk about the reading. this tends to mean that shorter length pieces work better. books can be great, but unless someone who's really dedicated (ie will always do the reading, and speaks about the book knowledgeably even when no one else has read it) is spearheading a title, people's attention will wane.
b. a mix of readings. i don't think it matters that much what the readings are exactly, but some people like history, some people like theory, some people like fiction. every reading will not appeal to everyone, but a mix over time is important.
c. a critical mass of people have to be interesting to each other in a variety of ways (some people are politically interesting, some people are from different parts of the world, some people want to get laid, etc). people don't have to like each other, but they need to be curious about each other.
d. structure and culture -- the reading group i'm in now started out as a free school class, taught by someone of long standing in anarchist circles who's very opinionated and serious. when the class ended officially, people didn't want to stop meeting, so they continued. the original teacher is long gone now (although still stops in to talk about specific topics), and we continue. but this raises the point of creating a culture, which the class did. the class succeeded in bringing people together who were interested, the structure was clear (same day, same time, same place every week), and there was a teacher who was responsible for presenting ideas. all of that is a very strong, stable way to get people to engage and commit.
e. no requirement for attendance. again, since stability is also important, this means that there is a core group that wants to come every time (wants to = does, not talks about wanting to and doesn't). people have to be able to not come when they don't want to. although it's possible that required attendance for some months would be a way to help cement the date and time in people's heads/schedules.
other reading groups i've been in that i could call successful were mostly around groups recruiting people. so there was a clear agenda, they had things to say about the readings, and there were regular meetings. obviously the politics of that are fucked up, and i wouldn't have gone to them for long, but people had interesting conversations about the readings, and had things to think about when they left, which is ultimately what reading groups should be about, i think?
there's a tension about doing the reading. there are people who have enough life experience/reading/information/confidence that they can find interesting things to say without having done a specific reading. but most people have a much richer experience when they have actually read the reading (and are much more likely to be interesting to both themselves and others when talking about it, instead of things just off the top of their head). on the other hand, pushing hard for people to do the reading mostly just means that people won't show up if they haven't.
there. some thoughts off the top of my head! :D
edit to add:
people should disagree with each other! i'm not just saying this as a chronically conflict-driven person (ha!), but without disagreement there is no engaging conversation. obviously the level and type of disagreement and the emotional punch behind it will vary.
practical tip for readings: either pick things that have some emotional resonance (thoughtful pieces on topical issues) or that a few people have emotional affinity with (some people feel passionately about the kronstadt rebellion, for example) from which lessons can be derived for today (like, why we don't work with commies, ;) ). philosophical or historical pieces have to be made relevant to people's lives and actions.
edit again (omg this answer may never stop):
the location is also a thing. meeting in a place that is neutral (not someone's house, and not overly affiliated with any one person or even maybe one political tendency) is probably really helpful for getting a mix of people. and if it can be free, that takes a significant if low key stressor out of the mix.
if the group lasts for a while, don't be afraid of having an inner circle. we get people who come in and complain about things like, "only some people get a certain kind of seating". (my thought has been that the different kinds of seating is not a bug but a feature. it's better to have different options for seating for those who want to participate less.) or they complain about in jokes, or whatever. whatever reading group lasts will have its own flavor. for its own purposes it might want to remain open to new people (i get a lot out of new people coming), but not seek to recreate itself for every person who's come for a month or two.
edit #1483
logistics about readings - afaict it's important to have access to the reading during the group (if people have different interpretations, or any interpretation, going back to the text helps avoid a circular empty conversation). that means either a print out or some gadget that holds the text. free seems to be more important than hard copy, but one way to be free(ish) is for someone to have access to a copy machine of some sort. preferably more than one person, actually.
