Tweet
It is already possible to limit the number of pages indexed per web site (start point), but in V4 of the software the limit had to be the same for all sites.
In V5 you can specify a different number of files to index for each start point. When indexing multiple web sites this allows some sites to be indexed deeply and other to have just a few pages indexed. If you are running a search engine portal type site, there is also the possibility you can sell web site owners the service of deep indexing.
This option is set in the "Edit start point" window, by clicking on "More" in the Spider tab, and selecting the URL with "Edit". Note that you will also be able to import this option in as part of an imported text file containing your start points. They will have to be in the format of:
http://www.mysite.com/, INDEX_AND_FOLLOW, http://www.mysite.com/, USELIMIT, 15
http://www.wrensoft.com/, INDEX_AND_FOLLOW, http://www.wrensoft.com/, NOLIMIT, 25
The parameters are optional and can be omitted, in which case it will default to no limit.
In V5 you can specify a different number of files to index for each start point. When indexing multiple web sites this allows some sites to be indexed deeply and other to have just a few pages indexed. If you are running a search engine portal type site, there is also the possibility you can sell web site owners the service of deep indexing.
This option is set in the "Edit start point" window, by clicking on "More" in the Spider tab, and selecting the URL with "Edit". Note that you will also be able to import this option in as part of an imported text file containing your start points. They will have to be in the format of:
http://www.mysite.com/, INDEX_AND_FOLLOW, http://www.mysite.com/, USELIMIT, 15
http://www.wrensoft.com/, INDEX_AND_FOLLOW, http://www.wrensoft.com/, NOLIMIT, 25
The parameters are optional and can be omitted, in which case it will default to no limit.