Intro to Web search - Web, image, video, news

The internet, simply put, is a big junkyard. People and companies dump anything and everything they can onto a page and give it an address. As of February 2007, there were 433,193,199 hosts on the internet, according to the ISC Internet Domain Survey. If you math is a bit weak, thats over 433 million hosts on the internet.  Multiply that by sub-domains and then by pages on each domain and sub-domain, and the page count for the internet goes into  billions. Point is, it is not possible to find anything, let alone something of interest to you, unless there is a method to this madness. This is where search engines come in. Today we have search engines which spit out keyword based results for simple text search, image and video search, news, books, local search, product search and FTP search. That's a lot of search. And there's whole new industry based around finding new things to search and making existing search better.

What exactly is search, how does it work and what's the best way to make use of it? And where did this start? Who invented search? All these questions, and more information about the history of search engines, is answered here. For the record, the first search engine was called web wanderer, and it was created by Matthew Gray in 1993. The simple way to describe a text search engine in action is that it stores copies of web pages, compares the requested keyword with the stored data, and spits out pages which come closest to matching the keywords. Each search engine uses its own formula, or algorithm, to determine the match. If you want to know more about search engine algorithms, click here. All other modes of search engines basically work in the same way, using text associated with images and videos to make the decision.

Major search engines, as of date :

Text : Google, Yahoo, MSN Live 

Image :  Google, Yahoo, Picsearch, FaganFinder, AltaVista

Video : Google, Blinkx, Vidsea, AOL Video

Book Search :  Amazon, Google

Product Search : Nextag, Froogle 

Local Search :  Superpages, Google, Yahoo

FTP Search :  Filewatcher

Click here to see the full list of all existing search engines. An important sidenote - There are search engines which aggregate the results of search engines. The best example is Dogpile.com, where you get results from all the major search engines, for any given keyword. Personally speaking, I prefer to simply use Google. However, if you are looking for multiple links to the same subject, Dogpile can be useful, because Google, Yahoo, MSN and other search engines each give varying degrees of importance to different sites.

Let's down to how best to use these search engines. Due to abuse and search marketing techniques known as black hat SEO, more than 50% of the results for any search are trash and uninformative. I use Google for text, image and video and news search. I rarely need any other type of search engine, so i can't vouch for book search or product search or local search. I'll share with you my simple rules for text search, which generally speaking, should apply to all other kinds of search also.

1. Always give preference to single word domains within search results. Example : jobs.com - Check. great-jobs.com ( not so great )
2. Prefer domains which do not have the keyword in the URL. Example : If you're searching for UK  jobs, then prefer a site which says UKjobs.com, rather than  everythinguk.com/jobs/
3. Install the Google toolbar and enable pagerank. Once you've clicked on a site within the search results, first thing you check is the pagerank of the opened page. If it's PR3 or above, then its probably worth reading. If not, consider all contents within te page to be unverified.
4. Anything beyond the first 100 results probably does not deserve to be read.
5. Make sure you're using the correct keywords. Read the search results, and see if keywords similar to the one you typed in are coming up again and again. Maybe that's the one you really want.

If you follow these rules, you should be able to make effective use of the search engines.

Search logic : I'm an internet researcher by profession, and a few weeks back, I was trying to find the residential address of a prominent personality for a client.  Pretty soon I bumped into a document which listed not only his address, but also his property value and his mortgage details, not to mention is social security number. 

To do this kind of thing, you need to use some logic. Let me give you an example. You search for John Doe's telepone number in New York.  Now there's likely to a hundred John Doe's in New York. How do you know which one is yours? You search background information first. What's his wife's name? Maybe Jane Doe. You search for his wife. You find her listed, alongwith a lot of Jane Doe's. You compare the John Doe results with Jane Doe results. You find a match, that's it. If you don't you apply similar logic to other parameters like general area of residence, work address ( likely to be close to residential address ), and so on.  Just about everything is available online, and for free. You just need to type in the magic passwords - meaning the right keyword.