Web Design Discussion

Please read What is a robots.txt file?

How to Create and Use a robots.txt file

1. Open notepad and save a file named robots.txt

2. Create records

The content of a robots.txt file consists of so-called “records”. A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more disallow lines.

User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]

Here’s an example:

User-agent: googlebot
Disallow: /cgi-bin/

This robots.txt file would allow the “googlebot”, to retrieve every page from your site except for files from the “cgi-bin” directory. All files in the “cgi-bin” directory will be ignored by the googlebot.

3. Upload the robots.txt file to your server.

Make sure you upload the robots.txt file to the root directory (same location as your home page).

Read the rest of this entry »

Sometimes you may not want the search engines to spider specific directories of your site because you don’t want the informationto be read by the public. You can accomplish this by creating a robots.txt file and placing those files in it.

Read the rest of this entry »

I always create an html site map for every site I build. It helps search engines index all the pages of your site. This is especially important if your home page does not contain a lot of text ie has a flash introduction, includes many images, uses javascript or is a
database-driven site.

The search engine spider will find all the other pages of your site (besides the home page) by following the links that link from the html site map link on the home page.

Read the rest of this entry »