Brian Burke speaks about Feedback 2.0 – Part II
Posted by Ninad on March 09, 2007 at 04:14 PM in General , Trust & Safety Corner | Permalink
Okay, folks, I'm back to share more nuggets from my really insightful conversation about Feedback 2.0 with Brian Burke, Director, Global Feedback Policy. Today I'm going to share what Brian told me about the impressive journey, analysis, and the rigorous research that went into Feedback 2.0. (For more background on Feedback 2.0, read my earlier blog post).
Brian revealed that the research that culminated in Feedback 2.0 actually started as long ago as 2004. Back then, Brian and his team had just rolled out one of the relatively few big changes to Feedback in the last 11 years – "de-scoring" Feedback from members who failed to respond to the Unpaid Item Process. The team decided to follow this up with an extensive survey about Feedback on a huge sample of 20,000 eBay members. It was through this research that the four dimensions of seller performance eventually enshrined in Detailed Seller Ratings (DSR) emerged. The research showed that buyers placed a great deal of importance on sellers' ability to deliver on the dimensions of item description, communication, shipping time, and shipping & handling charges.
Much to the team's satisfaction, this research was also borne out by an independent study (on 8000 eBay users), carried out by Professor Axel Ockenfels of the University of Cologne, Germany, in the spring of 2005. The professor's study corroborated the four dimensions that the eBay team had found were important for members. The professor also conducted lab exercises to show users' preferences for various rating scales, such as a 3-point scale and a 5-point scale. The research showed that a 5-point scale provides more differentiation (hence the 5 stars for each DSR dimension). These findings were also turned over to eBay's design team, and served as valuable input when they were designing the look and feel of Feedback 2.0.
Meanwhile, on a different research track, eBay's User Experience Research team was investigating what was the most important information that members wanted displayed on a seller's feedback profile. Their research was based on responses from 40,000 users!
Based on this preliminary research, by the summer of 2005 the design team created some initial visual concepts that were tested in eBay's usability labs. Other Community groups like Voices were also consulted. The team also consulted with the man who had designed the original feedback system –- Pierre Omidyar, eBay's Founder and Chairman.
While the research effort was headquartered in San Jose, the team also gathered input and data from other eBay markets, such as Australia, Germany, France, and Italy.
The designs went through several modifications, and tweaks were made to incorporate additional research findings or insights gathered from other channels.
By November 2006, Brian and the team were ready for a live test. For a brief period (2 days in the US, 4 days in Germany and 7 days in France and Italy), members in these countries had the option to actually use Feedback 2.0 to leave detailed seller ratings on their transactions.
The results of the live test couldn't have been better. Brian said that at more than 80% of all members who encountered the Feedback 2.0 concepts chose to leave Detailed Seller Ratings – a ringing endorsement of the concept!
The comprehensive research effort, one of the biggest undertaken at eBay, to create and refine Feedback 2.0, underscores the importance of "getting it right" that the team placed on such an important change.
A few days ago, eBay launched Feedback 2.0 in eight pilot markets around the world. As Brian says in this video, he and his teams are keeping a close eye on how Feedback 2.0 performs in these pilot markets. For something as critical to eBay as Feedback, you can never get enough data!
Comment on this post in the Discussion Board
