FollowLinks

Description

FollowLinks defines what kind of links between documents should be followed by the crawler.

The following argument values are understood:

• yes - follow all links. This is the default behavior.

• no - do not follow any links.

• a - follow the a HTML tags:

  <a href="http://www.site.com/">link text</a>
  

• area - follow the area HTML tags:

  <area shape="rect" coords="0,0,82,126" href="page.htm">
  

• frame - follow the frame HTML tags:

  <frame src="frame.html">
  

• htdb - follow the links generated by one of the HTDB routines.

• iframe - follow the iframe HTML tags:

  <iframe src="http://www.site.com"></iframe>
  

• link - follow the link HTML tags:

  <link rel="alternate" href="page2.html">
  

• meta - follow the meta HTML tags:

  <meta http-equiv="refresh" content="5;URL='http://www.site.com/'">
  

• redir - follow the URL specified in the Location header of HTTP redirects.

• xml - follow the links in XML files, e.g.:

  <item>
  <title>title1</title>
  <link>http://site.com/</link>
  </item>
  

Multiple arguments are possible in the same command:

# Follow only URLs from the "a" and "links" tags, ignore all other links.
FollowLinks a link

If a link type is given with - (minus sign) prefix, then this type of links are not followed.

# Follow all links except frame and iframe 
FollowLinks yes -frame -iframe

Scope

FollowLinks affects all Server and Realm commands until the end of the configuration file, or until the next FollowLinks command.

Examples

FollowLinks yes
      

See also

CollectLinks, HrefOnly.