robots.txt File
The robots.txt file is a fast, easy, and popular way to make your website more search engine friendly. Scroll down to discover even more about robot.txt, and how it can it can help your website in achieving search engine optimization (SEO)!
What is a robots.txt File?
The Robots.txt is a text file that is found (or placed) on a website's server, with the purpose of telling web robots how to decipher that site's pages. Web robots (also known as web wanderers, crawlers or spiders) are programs that decipher web pages and index them in search engines. This is known as search engine indexing and can make you rank higher in search engines.
For example, Google is a search engine that has a robot called the "googlebot" that uses robots.txt file for search engine indexing and and ranking web pages.
The robots.txt file gives web robots instructions about a website; this is called the Robots Exclusion Protocol. In this process, robots look for the robots.txt file, because it gives directions on how to decipher and index your site. When a robots.txt file is used correctly, web traffic and search engine rankings can increase - significantly helping your SEO!
What Does A robots.txt File Look Like?
The robots.txt file is one of the simplest piece of HTML code that you will ever have to write or edit. Most all WordPress platforms already include a robots.txt file (in the root folder of your website), so all that's left to do is edit it according to your needs.
For most small websites, the following robots.txt code is used to tell web robots to visit ALL pages of your site, without any specific instructions:
User-Agent *
Disallow:
- The value of the "User-agent" field is the name of the robot you want to give instructions for, and when an asterisk (*) is present this means that ALL search engines are allowed to perform search engine indexing on your site.
- The "Disallow" field states the name of the URL that is not to be visited and indexed in search engines. This can be a full path or a partial path. If left empty, the "Disallow" field indicates that all URL's can be retrieved.
Creating A robot.txt - Tips & Examples
- In the robots.txt file, it's important to add forward slashes before and after a folder name, because search engines need to place this onto the end of your domain name (www.myfamouswebsite.com/SEO-Example/. If a website's folder is found at a different URL, then simply rename the folder and adjust accordingly.
- You can edit the "User-Agent" so that only specific engines can track your website or a specific page. To do so, look up the name of a search engine's robot (such as Google's "googlebot" and/or Yahoo's "slurp") and then add it like this:
User-Agent: googlebot
Disallow: /SEO-Example/
- If you want to block an area of your site (like an irrelevant example page), then simply add that page's folder name to the "disallow" line:
User-Agent:*
Disallow: /SEO-Example/
- For every "Disallow" action, there needs to be a different line, and there can be no blank lines in between such as:
User-agent:*
Disallow: /gif/
Disallow: /tmp/
- To specify that NO web robots should visit any URL starting with /tmp/ or /example/:
User-agent: *
Disallow: /tmp/
Disallow: /example/
- The robots.txt file is a publicly available file, so ANYONE can see what sections of your server you don't want web robots to use. In other words, don't try to use robots.txt to hide information.
robots.txt Video
Check out this video to learn how to create a robots.txt file for your WordPress website or blog:
Check out our basic HTML tutorial or .htaccess File Tutorial to learn even more about the behind-the-scene actions of a website. Want even more? Feel free to browse internet marketing tools and discover online business opportunities here at My Famous Website!
No related posts.
Related posts brought to you by Yet Another Related Posts Plugin.
Comments
Got something to say?
You must be logged in to post a comment.