On the blindness of blind reviews

Over the last couple of years, blind reviews have been popularized as the ultimate method for fair talk selection in industry conferences. While I don’t really submit proposals myself, I have served several times on the other side of the process, doing speaker selection in conference committees, and the more data points I collect, the more convinced I become that the blind selection process is fundamentally flawed.

Blind reviews come from the world of academia. However, in academic conferences, you do not judge a talk by a 1-2 paragraph abstract, but by a 10+ page paper, so there’s way more to judge by. In addition, in academia the content of the research matters infinitely more than the quality of a talk. In industry conferences, selection committees in blind reviews have both way less data to use, and a much harder task, as they need to balance several factors (content, speaker skill, talk quality etc). It’s no surprise that the results end up being even more of a gamble.

Blind reviews result in conservative talk selection. More often than not, I remember me and my fellow committee members saying “Damn, this talk could be great with the right presenter, but that’s rare” and giving it a poor or average score. Few topics can make good talks regardless of the presenters. Therefore, when there is little information on the speaker in the initial selection round, talk selection ends up being conservative, rejecting more challenging topics that need a skilled speaker to shine and sticking to safer choices.

One of my most successful talks ever was “The humble border-radius” which was shortlisted for a .net award for Conference Talk of The Year 2014. It would never have passed any blind review. There is no committee in their right mind that would have accepted a 45 minute talk about …border-radius. The conferences I presented it at invited me as a speaker, carte blanche, and trusted me to present on whatever I felt like. Judging by the reviews, they were not disappointed.

In addition, all too many times I’ve seen great speakers get poor scores in blind reviews, not because their talks were not good, but because writing good abstracts is an entirely separate skill. Blind reviews remove anything that could cause bias, but they do so by striping all personality away from a proposal. In addition, a good abstract for a blind review is not necessarily a good abstract in general. For example, blind reviews penalize more mysterious/teasy abstracts and tend to be skewed towards overly detailed ones, since it’s the only data the committee gets for these talks (bonus points here for CfS that have a separate field for more details to conf organizers).

“But what about newcomers to the conference circuit? What about bias elimination?” one might ask. Both very valid concerns. I’m not saying any kind of anonymization is a bad idea. I’m saying that in their present form in industry conferences, blind reviews are flawed. For example, an initial round of blind reviews to pick good talks, without rejecting any at that stage, would probably solve these issues, without suffering from the flaws mentioned above.

Disclaimer: I do recognize that most people in these committees are doing their best to select fairly, and putting many hours of (usually volunteer) work in it. I’m not criticizing them, I’m criticizing the process. And yes, I recognize that it’s a process that has come out of very good intentions (eliminating bias). However, good intentions are not a guarantee for infallibility.

  • remotesynth

    I totally agree with this. When I ran a conference for 5 years, I was of the mind that who gave the talk was generally more important than what they were talking about. There are people whose talks I want to see regardless of what the topic is – they are engaging, thought provoking and I always come away learning something. Other people could pick the best topic and even have the best slide deck and bomb.

    I chose to have invite-only speakers list. That being said, I always set aside a certain amount of slots for speakers I’d never seen or who were new. The trouble with invite only events is the tendency to invite from within the same group every time.

    To me the best option is to have a committee that you trust and who represent a diverse set of experiences, backgrounds and views. Have this committee be conscious of efforts to be inclusive and make sure there is room for some fresh faces (even acknowledging that some of these will inevitably bomb).

    As you say, each method has its flaws and potential for bias, but even the blind review (as you point out) has bias, just of a different kind.

  • Blind reviews have their place. Don’t use them for all speaker spots, but use them specifically for a “newbie” stage. Perhaps, a conference could be done like a music festival; get some established rock stars (to draw audience and to ensure quality), but in addition to that, also get some newbies which selected by blind reviews. You may end up with some stinkers, but you may end up with some undiscovered gems.

  • Madeleine Neumann

    Ill enjoy reading this article. Whats missing for me, is the point, how to do it better. In the next year, I have to face this problem and I would like to do it in the best way I can. But how? Should I make a community voting from potential attendees? Do I have to make a blind voting? Do I have to choose a few people, who decide, what kind of talks will be the best for the conference? How could I manage my diversity? I think its more effective to talk about, how to do it in a better way, not what’s the worst way (sorry for my grammar, Iam not a native speaker) –

    • I did include a suggestion for using blind reviews without these issues. Read more carefully?

      Community voting is a terrible way to do it. Not only it’s subject to numerous biases, but it turns speaker selection into a popularity contest. See what happens with SXSW for evidence on this.

  • You make some good points but I can’t agree with this:
    “this talk could be great with the right presenter, but that’s rare”

    If you do or say that, you’re adding bias to the selection process. If the process is anonymized how can you know the speaker is not right? You’re assuming the speaker can’t deliver and not giving them a fair opportunity. If, after you select the talk and you see who the speaker is, tou think the speaker is not good enough you can provide some guidance on how they can present it better. If conferences want to ensure quality they should also do some follow up with speakers to ensure they are preparing themselves timely and accordingly. Specially for first time speakers.

    As for writing abstracts being a different skill, that might as well be true, but if someone isn’t capable of resuming their thoughts in an abstract, they probably wouldn’t do such a good job in a talk anyway. Talks are all about presenting a certain amount of knowledge in a limited amount of time in a way people can understand it, which means choosing what to put in and how to make people understand it.

    And as Karolina said on twitter, it might be that someone had some very important or unique knowledge to share that does not have good experience speaking or writing abstracts. And that’s OK. Select them, if you realize after they are inexperienced, provide them with some guidance.

    Anyhow this whole post doesn’t address that which I think is a core problem with your criticism of this particular situation: you know about these two people that didn’t get selected and their talks, but you don’t know about the other talks that were proposed or got selected. If you were doing this post conference I could understand it, but essentially you’re assuming the process failed without having the full story, and that is not fair both to the conference and to the process in itself.

    • Whether you agree with it or not, it happens with every blind review. When the topic is so difficult that there are very few speakers that could give a good talk on it, the assumption tends to be that the submitter is not among them. It’s just statistics. And since almost every topic could be a great talk with the right speaker, and a terrible one with the wrong speaker, judging by the description alone is guaranteed to be unfair.

      Re:abstracts, it’s not about capability to resume one’s thoughts, it’s about capability to convince the organizer that the talk would be good, which is different than provoking interest in the attendee.

      And duh, of course I wouldn’t post this after the selected talks have been made public. Are you completely clueless? If I posted this after that, it would sound like a public criticism against the selected speakers, which I would never do. But about the two proposals I mentioned, I had seen the abstracts, and in one of the two I had seen the speaker speak as well, and I have trouble imagining better proposals in all aspects, let alone multiple of them. In one of the two, I had told the speaker “Of course you’ll get in, I’d kill to have a proposal like this when I was reviewing abstracts for CSSConf US”, that’s how awesome I thought it was. But please don’t make this specific. I held the same views about blind reviews long before, this was just the extra push I needed to blog about said views, not some way to take the piss on CSSConf EU.

      • “Whether you agree with it or not, it happens with every blind review.”

        Again, you’re operating on the worst assumptions of people reviewing the talks being that biased. The fact you think like that doesn’t mean everyone else does, and it doesn’t mean that people might not codify that into the actual process. Yeah, bias will always still be there and it’s hard to beat, but if people are pushing for an anonymized process they might have an extra motivation to try and remove other biases as well.

        “When the topic is so difficult that there are very few speakers that could give a good talk on it, the assumption tends to be that the submitter is not among them.”

        Are you seriously implying that for a well known industry leading event the main persons involved in the field wouldn’t submit a talk about it? I would love to see actual data on this but I honestly don’t believe it. It might be so for a new, smaller, lesser known event, but for well known events I think there’s a real possibility the more qualified people will be submitting talks.

        And even then, you’re still ignoring the point that Karolina made and I reiterated on the previous point: having a not so good speaker on a unique topic is probably better than having a really good speaker on a topic people have already heard lots about. Or even then, a less experienced speaker might have a unique point of view that the leading expert in the field doesn’t.

        If the purpose is to only have top notch speakers then why do an anonymized process anyway?

        “it would sound like a public criticism against the selected speakers, which I would never do”

        So instead you chose to essentially criticise a process you don’t have the entire details of and without knowing about the other submitted talks.

        Can’t see how that’s any better.

        • Your reply shows such utter lack of understanding of both reviewing talks and of what I said, that replying to you is getting very tiring and feels futile. Not sure for how long I can keep doing it. This is probably my last reply to you, unless you somehow have some sort of epiphany and get a clue (yeah, I’m an optimist).

          No, I don’t assume everyone does because I do. I assume everyone does because everyone I know in such committees does. Or do you assume everyone judges in a vacuum without ever sharing their thought process with the other reviewers? “Could have been a great talk, but only with the right speaker dammit” is so common it has almost become an inside joke.

          Are you seriously implying that for a well known industry leading event the main persons involved in the field wouldn’t submit a talk about it?

          LOLWUT? First off, no, the main people in the field don’t tend to submit proposals. They get too many conference invitations as it is to submit proposals for even more events. I haven’t submitted a proposal to a Call for Speakers since 2011. But that’s not relevant to this discussion anyway, because who is a “main person in the field” doesn’t matter one iota to what I said.

          If the purpose is to only have top notch speakers then why do an anonymized process anyway?

          The process is not anonymized to get new speakers. Getting new speakers is achieved by having a Call for Speakers in general, anonymized or not. The process is anonymized to eliminate bias.

          So instead you chose to essentially criticise a process you don’t have the entire details of and without knowing about the other submitted talks.

          I have been in the committee for CSSConf US, not sure how the process of CSSConf EU would differ that much more. And it’s you that’s making it about a specific conference. My blog post was about blind reviews. What prompted it is irrelevant. You’re seeing something that is not here, then arguing against it, and you expect me to take you seriously?

  • Pingback: xmt85c4wx5ctwxw3tcerthve56()

  • Pingback: binge drinking definition()

  • Pingback: دانلود قسمت هشتم شهرزاد دوم()

  • Pingback: دانلود شهرزاد قسمت نهم فصل دوم()

  • Pingback: دانلود شهرزاد قسمت دوازدهم فصل دوم()

  • Pingback: دانلود شهرزاد قسمت هشتم فصل دوم()

  • Pingback: دانلود شهرزاد قسمت هفتم فصل دوم()

  • Pingback: دانلود قسمت هفدهم عاشقانه()

  • Pingback: دانلود قسمت 10 شهرزاد دوم()

  • Pingback: دانلود قسمت 12 عاشقانه()

  • Pingback: دانلود قسمت ششم فصل دوم شهرزاد()

  • Pingback: Anna Bell Peaks()

  • Pingback: Rachel Starr()

  • Pingback: Jada Stevens()

  • Pingback: Abigail Mac()

  • Pingback: Peta Jensen()

  • Pingback: Karter Foxx()

  • Pingback: Jessie Volt()

  • Pingback: Dillion Harper()

  • Pingback: dolandirici()

  • Pingback: Free Stuff Worldwide()

  • Pingback: Paleo Diet()

  • Pingback: real money casino games()

  • Pingback: free online movies()

  • Pingback: پنجره()

  • Pingback: دانلود قسمت اول شهرزاد دوم()

  • Pingback: دانلود قسمت دوازه عاشقانه()

  • Pingback: Loyola College for Sale by CollegeDunia()

  • Pingback: دانلود قسمت یازدهم شهرزاد دوم()

  • Pingback: how to bypass windows 7 password()

  • Pingback: stephen curry shoes australia()

  • Pingback: دانلود قسمت سیزده عاشقانه()

  • Pingback: titleist ap2 australia()

  • Pingback: videos porno()

  • Pingback: Roberto()

  • Pingback: porno gratis()

  • Pingback: videos porno()

  • Pingback: videos porno()

  • Pingback: porno gratis()

  • Pingback: porno gratis()

  • Pingback: porno gratis()

  • Pingback: porno gratis()

  • Pingback: porno gratis()

  • Pingback: porno gratis()

  • Pingback: videos porno()

  • Pingback: porno()

  • Pingback: دانلود قسمت 17 عاشقانه()

  • Pingback: قیمت پنجره دوجداره()

  • Pingback: love compatibility test meter()

  • Pingback: Weather Station()

  • Pingback: پنجره ضد سرقت ایرانی ترک المانی()

  • Pingback: دانلود سریال عاشقانه()

  • Pingback: Facial Spa Services Deerfield Beach()

  • Pingback: Acne Facial Treatments Dania Beach()

  • Pingback: Facial Spa Services Lauderhill()

  • Pingback: Acne Facial Treatments Davie()

  • Pingback: Facial Spa Services Sunrise()

  • Pingback: دانلود موزیک()

  • Pingback: groin brace support()

  • Pingback: فروش توری()