Dealing with

  • Make effective use of robots.txt
  • Be aware of rel=”nofollow” for links

Advantages are,

  • A “robots.txt” file tells search engines whether they can access and therefore crawl parts of your site.
  • Setting the value of the “rel” attribute of a link to “nofollow” will tell search engine that certain links on your site shouldn’t be followed or pass your page’s reputation to the pages linked to.