A fair evaluation process

Came across a post on results of Android developer challenge on TechCrunch site. They had announced a challenge for the developers to submit the application ideas for the android platform. While announcing the top 50 ideas/applications, the authors have also documented the process they used for the evaluation.

I must say this is one of the most elaborate evaluation process I have ever seen documented. The author goes to the details of how extreme/exception cases were handled and what care they had to take to minimize the impact of exclusions/exceptions. To me the biggest task in any evaluation, when it is by multiple persons, is how to normalize the scores. It is likely that a particular judge likes the application because he/she has some personal interest in that application, while some other judge simply dislikes it for some other reasons. Even if it is expected that judges do a fair job during scoring, one can't forget the human element. And the process explicitly tries to take care of this aspect well.

One thing that is missing in this post is how the scores were given and whether there were any criteria/guidelines provided to the judges. Except that, a fantastic post...

References:
  1. Android - It is a first complete, open and free mobile platform by "The Open Handset Alliance"
  2. Android Developer Challenge - To build great mobile application on the Android platform. The prize money, in total, is $10 million.
  3. TechCrunch - A weblog dedicated to profile and review the new internet products and companies. A cool site to track latest things in the internet world

Comments

Post a Comment

Popular posts from this blog

What is "B A Y"?

Weekend time. I was playing with Mrunmay, one of his games - 3 letter words. Basically there are some paper strips having one letter printed on it and then there's a letter dispenser, which dispenses 2 letters at a time. The game is to make up a word with those two letters and one of the paper strips. It's too early for Mrunmay to know the spellings and obviously the job of finding out matching spellings was with me. After every round, I was telling him the meaning of the word formed, in the form of question / answers. Like "Do you know this word?", "Do you know it's meaning?" and wherever required giving him examples and/or translations. After few rounds, the dispenser gave us two letters: "B" and "A". The only possible combination left was the word "BAY". I started asking him the meaning... Me: "Do you know what is BAY?" Mrunmay: "Bay??" Me: "Yeah.. do you know what is the meaning of Bay?&q
Read more

Subscription Services

Image
As the internet penetration increased over the last 2 decades, it also gave rise to digital subscriptions. The internet users slowly started developing trust in internet-based payments and that too on recurring payments. It probably started with the tech community using the recurring payment systems and slowly spreading that faith to non-tech-savvy users. There was also a phase when I started using the credit card for online transactions but wasn't comfortable storing the card details with the merchant. I used to enter the details every single time. (That is how I still remember the CC number that was in use at that time) In the late 2000's I was introduced to Ecom website development and since then it has been part of my professional as well as personal life. Over the years, I learned about the  PCI-DSS standard , and the extent to which the Ecom companies make an effort to protect consumer data. There have been instances of a data breach across the globe, but alon
Read more

Music Musings

Image
Songs have a huge role in our life. In childhood, the primary source of songs was Radio. It slowly expanded to TV, then dedicated music channels. Somewhere in between or in parallel to Radio and Television, there existed the parallel world of Cassettes (and then later CDs) which served the audience. Since then, we have evolved to on-demand, yet unlimited songs. Each of these phases has/had some peculiarities of its own.  When I was a regular listener of the radio, there was only a handful of channels - probably two or three. One channel was primarily for music where Hindi songs were dominant and the other was for general. I think they were named “ विविधभारती ”, “आकाशवाणी अ” and  “आकाशवाणी ब” etc. The beauty of the songs on the radio channel was the surprise element - you could get to listen to a song of your liking which would lift your mood automatically.  The television channels, specifically those dedicated to music stole the limelight that radio was enjoying. However, in the initia
Read more