Google and other search engines in VS .NET

Generate QR Code in VS .NET Google and other search engines

Web search engines are complex pieces of software Almost every search engine, including Google, consists of three primary parts: A Web crawler, sometimes called a spider: The spider finds and retrieves meaning, gets and saves a copy of Web pages An indexer: The indexer indexes all the words on a page, and stores the words in a huge database that cross-references the keywords with the pages in which they appear As I explain in 4, short words such as and and how, also called stop words, are indexed, but do not appear in search results unless you override Google s defaults by adding quotes

TEAM LinG - Live, Informative, Non-cost and Genuine !

A query processor: The query processor makes sense of queries entered by users of the search engine, compares the words in your search to the keywords in the index, and returns the resulting matches in some kind of order (To find out more about queries and how they are processed, see Part I of this book) Google s primary innovation when it first became available was a relatively sophisticated way of determining relevancy in the results returned by its query processor This technique, called the PageRank algorithm, depends on the number of Web pages that link to a particular Web page I explain the PageRank algorithm in more detail in 11 Essentially, pages that have a higher rank appear closer to the top of the Google search result listings By and large, there s not much difference in the way you can implement a spider or an indexer However, not all search engines have indexed the same parts of the Web In addition, each search engine s query processor is built differently, using somewhat different principles That s why different search engines give different results to the same query, and why there are queries that can be answered by a particular search engine, but not by others Some other search engines license their technology from Google For example, Yahoo! licensed its search technology from Google for a while until it developed its own search technology Currently, A9, wwwa9com, licenses much of its technology from Google So if you get almost identical results from several search engines, they are probably using the same underlying technology Admittedly, Google is one of the best search engines, run by a world-class team of technologists But if you can t get relevant results using Google, you should try some other search engines before giving up on finding the information on the Web (see 2 for a comparison of different search engines)

There are large parts of the World Wide Web that no search engine including Google can see Providing a good definition of the invisible Web (which is also sometimes called the deep Web and dark matter) isn t so easy to do The best way to think of it is simply as material that is on the Web that has been excluded from search engines, specifically from Google, either on purpose or due to technological limitations Material on the Web that is invisible to Google will almost certainly be invisible to other search engines as well

TEAM LinG - Live, Informative, Non-cost and Genuine !

You can easily see why some Web sites such as those with adult content might be excluded from Web search engines and thus rendered invisible But it may be a little harder for you to understand why sites that contain information of value to researchers are also invisible There are a number of possible reasons that Web pages might be excluded from Google (and other search engines) These include Dynamic results aren t easy to read: If a page is dynamically generated and assembled from a database, Google might not return it as a result to your query Although spiders can access dynamically generated pages, particularly if a page is pulled intact out of a database and even the returns page from a Google search can be considered a dynamically generated page spiders can have trouble with any dynamic generation that involves setting multiple fields to return the results Pages that require logons may not be accessible: If you re required to log on to access a page and/or a subscription or fee is required to access the page (see Other online research services ), the results may not come up (because, obviously, Google s spider cannot log on) The page is not connected: If the Web page is disconnected, with no other pages linking to it (so a spider cannot find it by following links), it won t show up in a list of search results The page doesn t have words: If a Web page contains mostly visual matter, indexing may be limited to ancillary text such as that in alt parameters of the img tags Material can be excluded by the Webmaster: Depending on the site and the information it contains, a Webmaster may mark specific pages as off-limits to crawlers A file on a Web server named robottxt tells the spider which pages are verboten For more information about how this works, see the information about Robots Exclusion at wwwrobotstxt org/wc/exclusionhtml The information is in a format that can t be easily read: If a file is in a format that is hard for the spider to read, such as an executable file, or a compressed file (such as a zip file), Google may not be able to find it Google (unlike most other search engines) does have an impressive ability to index Acrobat (pdf) and Postscript files, and Microsoft Word (doc) documents