Blog: Yahoo's Hadoop Software Transforming the Way Data Is Analyzed

Yahoo's Hadoop Software Transforming the Way Data Is Analyzed
SiliconValley.com (11/05/08) Ackerman, Elise

Yahoo!'s Hadoop open source data-mining program is capable of searching through the entire Library of Congress in less than 30 seconds. Universities also are using Hadoop, which is part of Yahoo!'s huge computing grid. "It makes it possible to actually take advantage of all the computers we have hooked up together," says Yahoo!'s Larry Heck. Hadoop improves the relevance of ads Yahoo! displays on the Internet by analyzing the company's endless flow of data, which is now more than 10 terabytes a day, in real time. As users navigate through Yahoo!, Hadoop determines which ads are likely to catch their attention. Yahoo! also will be using Hadoop on the sites owned by the 796 members of a newspaper consortium that is working with Yahoo! to sell more advertising at better prices. Hadoop was first used to build Yahoo!'s Web index. Since then, the software has been adjusted by engineers and researchers both inside and outside of the company for use in experiments with giant data sets. Amazon, Facebook, and Intel developers are using Hadoop for tasks such as log analysis to modeling earthquakes. "We are leveraging not only the contribution that we are giving to the software, but the contributions from the larger community as well, everybody wins from it," Heck says.

View Full Article