How do Search Engine Bots Work?

[jasonquey]

I heard more about search engine bots like Googlebot, Yahoo slurp, Bing bot but I was wondering how they work on my website? Are they machines or real people?

Do Bots help improve search rankings if they visit my site regularly?

 

[Mark007]

Bots are search engine programs developed to read through your webpage source. They are responsible for webpage crawling and cache for indexing them in search engines. These bots are automated search engine programs developed with instructions to perform certain tasks online.

 

[sajid israq]

Crawler-based search engines are those that use automated software agents (called crawlers) that visit a Web site, read the information on the actual site, read the site's meta tags and also follow the links that the site connects to performing indexing on all linked Web sites as well. The crawler returns all that information back to a central depository, where the data is indexed. The crawler will periodically return to the sites to check for any information that has changed. The frequency with which this happens is determined by the administrators of the search engine.

 

[Newille]

A Web crawler starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.
The large volume implies the crawler can only download a limited number of the Web pages within a given time, so it needs to prioritize its downloads. The high rate of change can imply the pages might have already been updated or even deleted.