URLSelectCacheSize

Name

URLSelectCacheSize -- sets URL cache size for indexer

indexer.conf

Synopsis

URLSelectCacheSize {number}

Description

URLSelectCacheSize defines the number of URLs to load from the database as crawler targets in a single portion. The bigger number makes crawling performance better, however in case when indexer crashes or is killed or is terminated using Ctrl-Break, some documents will get stuck in crawling queue and won't be available for crawling for 4 hours.

Note: To make stuck documents available for crawling use indexer -a flag, in combination with suitable subsection control option, like -s, -u, etc.

The default value is 128.

Examples


URLSelectCacheSize 1024
      

See also

URLSelectSkipLock.