findinsite-ms indexing advanced options

An individual file-spec can include zero or more * or ? wildcard characters, where ? matches exactly one character, and * matches zero or more characters. For example file???.ht* would match:
    file001.htm, file101.html and file111.ht
but not
    file1001.htm

A list of file-specs can be given directly in the property, or indirectly in a file.

Direct file-specs

Direct file-specs are semi-colon separated, eg:
Include=iso*;*12*
Exclude=file???.ht*

This specifies two Include file-specs and one Exclude file-spec.

Indirect file-specs in a file

An indirect value consists of @ followed by a file name, where file-specs are specified one per line in plain text. The above direct example may be expressed indirectly as follows:
[email protected]
[email protected]

where includes.txt contains:
iso*
*12*

and excludes.txt contains:
file???.ht*

If an indirect file cannot be opened, an error message is reported.

Username/password credentials

The Credentials property must consist of a semi-colon separated list of credentials. Each credential contains comma-separated fields: a username, a password and an optional path. Spaces are trimmed at the ends of all fields. To use a blank password, specify a period (.) in that field.

For example, for a single username (uname) and password (pwd), use this:

Credentials=uname,pwd

Only one credential can be supplied for each path on the web site. Therefore, if you are using more than one credential, then the paths must be different. Suppose you are indexing www.example.org. If username/password uname1/pwd1 is required for directory www.example.org/manager/ and uname2/pwd2 is required for all other directories, then use this:

Credentials=uname1,pwd1,manager/ ; uname2,pwd2

Indexing rules

Each rule must have one or more conditions, and must have one action. For example, this rule has two conditions (that the file being indexed is a PDF, and its URL starts with Default) and one action (store its referer as the file URL):

rule1=C:type==PDF;C:url==Default;A:url=referer;

• Each rule consists of several elements, each separated by a semi-colon (;)
• Condition elements start with C:
• The Action element starts with A:
• If all the conditions are true, then the action is performed

Conditions that start with C:type== check that the file is a certain type, from this list: html pdf txt doc xls ppt image pub.

Conditions that start with C:url== check that the file starts with the subsequent characters. Note that the "base URL" should not be included here, ie if the indexing run started at http://www.example.com/subdir/ and you want to check for files that start http://www.example.com/subdir/another/ then use condition C:url==another/

The Action A:url=referer sets the URL for this page to its referer page, if it exists. In practice this restricts this rule to standard URL indexing runs. Note that as a consequence, the referer URL will appear twice in the search database.

Example

	Description=My web site
	ScanType=url
	ScanURL=http://www.mycompany.com/
	ParseHTML=true
	HTML_Files=*.htm,*.html,*.asp
	ParseTXT=false
	ParsePDF=true
	PDF_Files=*.pdf
	CaseSignificant=false
	StoreStopWords=true
	StopWordFile=
	NoTitleIgnorePageLinks=true
	ParseUpHierarchy=false
	StorePositions=true
	StoreLoneWords=true
	UseMetaDescriptionAsAbstract=true
	UseMetaAbstractAsAbstract=true
	AbstractWords=0
	Include=
	Exclude=