This post is part of a series:

• So you wanna conference
• Money
• Venue

Proposals

Selecting proposals from the CFP process is very difficult, and you need a very diverse group of talk selection committee members. It is also one of the first parts where PR can get out of hand and negativity brews around the event. We received over 100 proposals, and we had a total of 36 slots… UGH.

PyCon has a great selection process in my opinion. It shifts through HUNDERDS of talks. You can see the details here. We adapted that slightly to fit the smaller scale of PyTennessee.

Overview

• By Nov 3rd: Initial review took place on the website. The committee reviewed each proposal, left feedback for the author, and voted on the proposal using an “identify the champion” system. We also meet occasionally on IRC to discuss talks and coordinate feedback to presenters.
• Nov 3rd: IRC review begins. The first round, nicknamed “kittendome”, reviewed each talk in isolation, determining if the proposal would be a “good talk” for PyTN. Some talks were rejected at this point; the rest will passed on to…
• Nov 3rd: The second round, nicknamed “Thunderdome”. The remaining talks (typically about half of the proposals) were grouped by topic into groups of 3-5. During Thunderdome, we’ll discussed one group at a time, and select the best talk(s) from each group.
• Nov 4th: Notify Speakers
• Nov 15th: Publish Talk listing

Initial Review

The committee had two goals here:

• Give feedback to authors and help them improve their proposals (authors could edit proposals up until the CfP closes).
• Get an initial set of votes from as many committee members as possible. This will inform the later meetings.

We made sure that each talk got reviewed by at least 3 committee members (and ideally more).

Voting

The process we used to assess proposals was based on the Identify the Champion process. The idea is to to assess each talk and identify “champions” and/or “anti-champions” – people who will argue strongly for or strongly against a talk in initial review.

The document linked above uses an “A/B/C/D” scale for reviewing; we use the same idea, but use “+1/+0/-0/-1” labels (after the Apache voting system). These votes mean:

+1 – I will champion the talk at the program committee meeting. The topic sounds good and the proposal is solid. This vote means you’re willing to put your reputation on the line in favor of the proposal and its author, and that you believe PyTN would be worse off for not having the talk.

+0 – This topic is good and the proposal is solid. The speaker is capable of correcting any deficiencies, and I think a significant number of PyTN attendees would want it available. I don’t feel strongly enough about it to serve as its champion at the program committee meeting.

-0 – I am not in favor of this talk as proposed. The talk, the topic, or the proposal has problems that concern me. I’d rather see it not accepted.

-1 – This talk or its proposal has significant problems. I believe that PyTN would be worse off for having it, so I will argue strongly to reject this talk.

If you don’t know enough about a topic to judge a proposal’s coverage of it, or it’s a topic you tend to actively avoid, you should not recuse yourself from voting on that basis alone. You can still judge the structure of the proposal, the likelihood that the speaker can cover the material proposed in the time allotted, whether the topic is likely to appeal to others, etc.

“Obvious” accepted/rejected talks

After the initial review, there will be a small set of talks that are “obviously good” or “obviously bad”. We’ll set a threshold, by committee consensus, for talks that are automatically rejected or get to automatically skip ahead to Thunderdome. This is usually something like “reject all talks with 3 or more -1s and no +0s or better” or “accept all talks with at least 4 +1s and no -1s”. Let me know your thoughts

These accepted talks aren’t a free pass to PyTN - it just means that the talk goes directly to Thunderdome. A reject is final, however; it weeds out the few obviously bad proposals that nobody on the PC wants to support. After the initial review we had nine 45 minute talks, and six 30 minute talks that all had roughly equal ratings, or a program committee member who really wanted to raise a talk up the rankings.

IRC Meetings

Next, we held meetings in IRC to discuss the talks. We allowed a champion to speak for the talk being discussed, and allowed the rest of the program committee to respond. We then took another vote. This process continued until all of those talks were refined down to the list you see on the schedule, and a set of backup talks.
There we three attempts to add a 5th track to the conference to accommodate more of the talks. We wanted to accept many more talks than we did; however, we have a big enough event on our hands now for a first one. Then we wrapped up with a vote on the conference as a whole. The question was, “Did anyone feel that any one talk would detract from PyTennessee being a great conference?” After that passed, we have the list you see today

Kittendome

In the first round of meetings, we went through each proposal individually. The goal here was simple to determine if a given talk – reviewed in isolation – is potentially good enough for PyTN. That is, in the first round we reviewed talks strictly on their merits and avoid holding them up against similar talks (we do that next, in Thunderdome).

We reviewed talks, one at a time. We gave a couple minutes for the champions (identified by the website votes) to make an argument for the talk. Then, a few minutes of general debate. Finally, we voted yay/nay on each talk. Majority rules - a majority of “yay” votes will indicated that the talk was accepted – it moves onto Thunderdome. A majority of “nay” votes indicated that the talk was rejected – it’s out. The chair will break ties.

Thunderdome

After round one has ended, we’re left with a set of talks that are all “good enough” for PyTN, and so we pit similar talks head-to-head and try to select the best.

In each “round” of Thunderdome we reviewed 3-5 talks, roughly grouped by theme/content/topic/etc. Out of each group we’ll got one or more winners, possibly some damaged talks, and some talks that are out. We did this by discussing, and then the committee voted for the talks they liked. Members voted for any number of talks (including none or all of them).

The winners were the talks from the group that absolutely should be at PyTN. It advanced and was safe from future cuts. It’s on the program! The winners were talks that had a 75% supermajority.

Damaged talks were ones that had been voted down by Thunderdome but aren’t quite out of the running yet. There may not be any damaged talks in a given group, or all talks may end up damaged. Damaged talks are any talks that didn’t win, but did receive votes from at least 50% of the committee. Talks that were selected by fewer than 50% of the committee in attendance are out – they will not appear at PyTN.

We then allowed a champion to speak for each damaged talk, and allowed the rest of the program committee to respond. We then took another vote. This process continued until all of those talks were refined down to the list you see on the schedule, and a set of backup talks.

There we three attempts to add a 5th track to the conference to accommodate more of the talks. We wanted to accept many more talks than we did; however, we have a big enough event on our hands now for a first one.

Then we wrapped up with a vote on the conference as a whole. The question was, “Did anyone feel that any one talk would detract from PyTennessee being a great conference?” After that passed, we had the list that appeared at the conference.

UGH… DRAMAS

We attempted to do a fair and honest review of the talks, did we do a perfect job… No. Program Committee members were not allowed to submit talks, share talk reviews, or beseech outside opinions. During Kittendome and Thunderdome people abstained from voting for coworkers, bosses, close friends, etc. In my case, that meant I cast less than 5 votes the entire dome process for individual talks. :( I’m certain people could find flaws in our process, or point out some sort of bias.

###I can be bought with Cookies, and that’s really no secret.

So what kind of drama did we have?

Well first, we were clear from the onset that we wanted to be a different conference. We picked a diverse group of talks, and didn’t limit ourselves to just python or even code. In fact, two of the highest vote getting talks were on Systematic Bias and Mental Health. Those were contentious in the organizers group, but the amount of “not python” was a shock to people. I got emails calling me stupid, saying I destroy the conference, and that I was doing Python a disservice.

Secondly, we choose to allow presenters to have multiple talks. During the domes, we didn’t pay attention to how many times a speaker names appeared in our selected talks other than to decide if we could vote or not due to some potential bias. After the committee members were done selecting talks, they noticed that a few speakers had multiple talks chosen. A discussion then occurred, and the decision was made to say these were the talks chosen, they are the talks that should be presented. Unfortunately, many of these people were friends or close to a few of the organizers. This also caused several emails accusing us of playing favorite or placating big name presenters, etc.

Finally, A few people we turned down sent nasty emails about how well known they were, or how great a presenter they are and how stupid I was for turning them down. In a few cases the submitted proposal was one line about the content and several lines about past presentations. As you can imagine the committee voted that talk down over one with a nice full proposal. I still got told I had no business running a conference, I obviously knew nothing about development, and that “my talk would have been selected at” conference X.

These three days lead to a massive cry!!! Thanks to strong support from the committee and a few members of our local community I made it through. I knew we had a great set of talks selected, and I knew that it was done as fair as possible. And a week later the drama had died down and people began to get excited about the talk selection and were tweeting about it.

And now looking back after the conference. It was totally worth all the drama. We had an overwhelmingly positive response at the event. I and the rest of the committee were ecstatic with the results.