Webarc:Lemur Indexer (modified) - Revision history

Scsong at 16:37, 4 December 2009

2009-12-04T16:37:35Z

← Older revision		Revision as of 16:37, 4 December 2009
Line 13:		Line 13:

	== How To Build ==		== How To Build ==
	The same way as you build the original Lemur/Indri toolkit. I.e. under the toolkit directory,		The same way as you'd build the original Lemur/Indri toolkit. I.e. under the toolkit directory,
	<pre>		<pre>
	make; make install		make; make install
Line 19:		Line 19:

	== How To Run ==		== How To Run ==
	The same way as you run the original IndriBuildIndex (http://www.lemurproject.org/tutorials/interm_indexing-2.php).		The same way as you'd run the original IndriBuildIndex (http://www.lemurproject.org/tutorials/interm_indexing-2.php).

	</pre>		<pre>
	./IndriBuildIndex <Indri indexer parameter file>		./IndriBuildIndex <Indri indexer parameter file>
	</pre>		</pre>

Scsong at 00:01, 10 November 2009

2009-11-10T00:01:36Z

← Older revision		Revision as of 00:01, 10 November 2009
Line 24:		Line 24:
	./IndriBuildIndex <Indri indexer parameter file>		./IndriBuildIndex <Indri indexer parameter file>
	</pre>		</pre>






	== Input File ==		== Input File ==
Line 43:		Line 38:
	</corpus>		</corpus>
	</parameters>		</parameters>
			</pre>

	Example parameter file for indexing a monthly snapshot.		Example parameter file for indexing a monthly snapshot.

Scsong at 00:01, 10 November 2009

2009-11-10T00:01:08Z

← Older revision		Revision as of 00:01, 10 November 2009
Line 9:		Line 9:
	Note that the the new index also contains the previous statistics that count documents and terms over the entire collection (monthly snapshot + carryover revisions).		Note that the the new index also contains the previous statistics that count documents and terms over the entire collection (monthly snapshot + carryover revisions).

	* Berkeley DB as a new input source: a new class indri.parse.BDBTaggedDocumentIterator is added. Minor changes in a number of related classes.		* Berkeley DB support as a new input source: a new class indri.parse.BDBTaggedDocumentIterator is added. Minor changes in a number of related classes.

Scsong at 00:00, 10 November 2009

2009-11-10T00:00:44Z

New page

== What It Does ==
Given an input source, the Lemur/Indri Indexer (modified) either builds a new index or incrementally adds new documents on an existing index. Below are new features we added on top of the original Lemur/Indri toolkit. The original Lemur/Indri toolkit can be found at http://www.lemurproject.org/.
* New statistics parameters are included in the index. They are:
** fresh document count: the number of wiki revisions in the entire index in the monthly snapshot, not including those carried over from the previous snapshot.
** term count in fresh documents: the number of terms appearing in the monthly snapshot, not including those carried over from the previous snapshot.
** fresh document count for each term: the number of wiki revisions that contain a given term in the monthly snapshot, not including those carried over from the previous snapshot.
** term count in fresh documents for each term: the number of the occurrences of a given term in the monthly snapshot, not including those carried over from the previous snapshot.

Note that the the new index also contains the previous statistics that count documents and terms over the entire collection (monthly snapshot + carryover revisions).

* Berkeley DB as a new input source: a new class indri.parse.BDBTaggedDocumentIterator is added. Minor changes in a number of related classes.

== How To Build ==
The same way as you build the original Lemur/Indri toolkit. I.e. under the toolkit directory,
<pre>
make; make install
</pre>

== How To Run ==
The same way as you run the original IndriBuildIndex (http://www.lemurproject.org/tutorials/interm_indexing-2.php).

</pre>
./IndriBuildIndex <Indri indexer parameter file>
</pre>

== Input File ==
Mostly the same format as defined in the original Indri indexer parameter file format (http://www.lemurproject.org/lemur/indexing.php#IndriBuildIndex), with an additional support for the Berkley DB as an input source.

Example parameter file for indexing carryovers via Berkley DB.
<pre>
<parameters>
<index>/fs/webarc3/data/wikipedia/lemur_index/monthly/month-003</index>
<indexType>indri</indexType>
<corpus>
<path>/fs/webarc3/data/wikipedia/bdb-monthly/month-003-co</path>
<class>trectext_from_bdb</class>
</corpus>
</parameters>

Example parameter file for indexing a monthly snapshot.
<pre>
<parameters>
<index>/fs/webarc3/data/wikipedia/lemur_index/monthly/month-003</index>
<indexType>indri</indexType>
<corpus>
<path>/fs/webarc3/data/wikipedia/preprocessed-monthly/trec-month-003.xml</path>
<class>trectext</class>
</corpus>
</parameters>
</pre>

== Output Files ==
An Indri index directory at the location specified in the input parameter file (<index>...</index>).

== Notes ==
* As a new index format contains extra statistics, the new index is not compatible with the original Indri index, thus won't be readable by the original Lemur toolkit.

== Source Codes ==
svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/lemur-4.10