On the day of Google's 15th birthday, Google released its new "Hummingbird" search algorithm, this is the first time that Google changes its search algorithm since 2010 after the Caffeine algorithm. It will affect 90% of search results, according to Amit Singhal, Lead of Google Search.
Some may be curious about how Google measures the improvement of a new search algorithm. There are many factors need to be considered, for example, accuracy, relevance, search depth etc. According to Amit Singhal, they have the entire web in a sandbox that only their engineers can see, and their engineers can take their new algorithm and see it change millions of queries. Engineers typically start by running a series of experiments, tweaking small variables and getting feedback from colleagues until they are satisfied. Then they send it to testers, whom they pay, but they don’t tell them what they are testing.In this phase, the search precision is evaluated, each one of the tester will get a Search Quality Rating Guideline, this guideline will tell testers how to measure precision.
If the evaluators’ feedback looks good, they move forward with a “live traffic experiment.” In these experiments, they will change search for a small percentage(one percent) of real Google users and see how it changes the way they interact with the results. Later they carefully analyze the results to understand whether the change is an improvement to the search results. For example, do searchers click the new first result more often? If so, that’s generally a good sign.
Finally, Google's most experienced search engineers carefully review the data from all the different experiments and decide if the change is approved to launch or needs to be refined and retested.
For details about measurement of search engine algorithms, you can refer a more technical version : Evaluating Search Engines
That is very nice!