NAL Contributes to Searching the Deep Web of Science

The National Agricultural Library (NAL) is helping people around the world get better access to scientific agricultural information.

Conventional search engines cast a wide net without weighing the credibility of individual sites, often leaving people to sort through thousands of websites to reach only factual science-based content. NAL is helping answer that problem by contributing to WorldWideScience.org, an Internet portal that searches only nationally sponsored science websites from more than 70 countries.

NAL is part of the Agricultural Research Service (ARS), the principal intramural scientific research agency in the U.S. Department of Agriculture (USDA).

NAL selects USDA websites (with the exception of the Forest Service) for Science.gov, the Internet gateway to U.S. federal science websites. These websites then go to WorldWideScience.org, where they help provide a global picture of scientific research.

This access can only enhance research into agricultural issues, given that such problems don't often stop at international borders. For example, U.S. scientists looking for biocontrols for the brown marmorated stink bug will likely find scientific information from China helpful, since the insect came from Asia, explains NAL Deputy Director Eleanor Frierson, who serves as the principal U.S. representative to WorldWideScience.org.

In addition, much of the content retrievable through WorldWideScience.org simply does not show up through conventional search engines such as Yahoo! and Google. Such information resides in what is termed the "deep web," because it is in databases that are not directly accessible by search engine "crawlers," which create indexes of website contents. To respond to a query, a search engine consults its index, leaving unindexed websites and databases invisible.

In contrast, WorldWideScience.org accesses websites and databases directly, converting a search request into the query style each needs and even translating the query into a variety of languages. Each site then returns results to the WorldWideScience.org search engine, which compiles them into one search response.

A recent test of typical scientist search queries showed that results from WorldWideScience.org were uniquely different from Google and Google Scholar results 92.7 percent of the time, a number that jumped to 97.6 percent when looking at only the top 50 results.

Read more about NAL's participation in WorldWideScience.org in the October 2011 issue of Agricultural Research magazine.