Webarc:Tools Developed - Revision history

Scsong at 22:40, 10 November 2009

2009-11-10T22:40:42Z

← Older revision		Revision as of 22:40, 10 November 2009
Line 13:		Line 13:

	== Indexers ==		== Indexers ==
	*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), ~~not only~~ for the entire index~~, but only~~ for each individual term in the index.		*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), both for the entire index and for each individual term in the index.

	[[Image:index.png\|480px\|Indexing Flowchart]]		[[Image:index.png\|480px\|Indexing Flowchart]]

Scsong at 20:44, 10 November 2009

2009-11-10T20:44:14Z

← Older revision		Revision as of 20:44, 10 November 2009
Line 29:		Line 29:
	*[[Webarc:PBS Jobs Scripts for Searching\|PBS Jobs Scripts for Searching]] (Bash): These scripts allow search servers and clients to be deployed over multiple nodes in the Chimera cluster. For n nodes, and i indexes, each node is made responsible for i/n indexes. Indexes are distributed in an interlaced fashion: i.e. node 1 has index 1, n+1, 2n+1, ... and node 2 has index 2, n+2, 2n+2, ..., and so on.		*[[Webarc:PBS Jobs Scripts for Searching\|PBS Jobs Scripts for Searching]] (Bash): These scripts allow search servers and clients to be deployed over multiple nodes in the Chimera cluster. For n nodes, and i indexes, each node is made responsible for i/n indexes. Indexes are distributed in an interlaced fashion: i.e. node 1 has index 1, n+1, 2n+1, ... and node 2 has index 2, n+2, 2n+2, ..., and so on.

	[[Image:search.png\|~~640px~~\|Distributed Search]]		[[Image:search.png\|480px\|Distributed Search]]

	== Result Analysis ==		== Result Analysis ==

Scsong at 20:19, 10 November 2009

2009-11-10T20:19:29Z

← Older revision		Revision as of 20:19, 10 November 2009
Line 15:		Line 15:
	*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), not only for the entire index, but only for each individual term in the index.		*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), not only for the entire index, but only for each individual term in the index.

	[[Image:index.png\|~~640px~~\|Indexing Flowchart]]		[[Image:index.png\|480px\|Indexing Flowchart]]

	== Distributed Index Support ==		== Distributed Index Support ==

Scsong at 20:18, 10 November 2009

2009-11-10T20:18:58Z

← Older revision		Revision as of 20:18, 10 November 2009
Line 15:		Line 15:
	*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), not only for the entire index, but only for each individual term in the index.		*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), not only for the entire index, but only for each individual term in the index.

	[[Image:index.~~gif~~\|640px\|Indexing Flowchart]]		[[Image:index.png\|640px\|Indexing Flowchart]]

	== Distributed Index Support ==		== Distributed Index Support ==
Line 29:		Line 29:
	*[[Webarc:PBS Jobs Scripts for Searching\|PBS Jobs Scripts for Searching]] (Bash): These scripts allow search servers and clients to be deployed over multiple nodes in the Chimera cluster. For n nodes, and i indexes, each node is made responsible for i/n indexes. Indexes are distributed in an interlaced fashion: i.e. node 1 has index 1, n+1, 2n+1, ... and node 2 has index 2, n+2, 2n+2, ..., and so on.		*[[Webarc:PBS Jobs Scripts for Searching\|PBS Jobs Scripts for Searching]] (Bash): These scripts allow search servers and clients to be deployed over multiple nodes in the Chimera cluster. For n nodes, and i indexes, each node is made responsible for i/n indexes. Indexes are distributed in an interlaced fashion: i.e. node 1 has index 1, n+1, 2n+1, ... and node 2 has index 2, n+2, 2n+2, ..., and so on.

	[[Image:search.~~gif~~\|640px\|Distributed Search]]		[[Image:search.png\|640px\|Distributed Search]]

	== Result Analysis ==		== Result Analysis ==

Scsong at 20:12, 10 November 2009

2009-11-10T20:12:57Z

← Older revision		Revision as of 20:12, 10 November 2009
Line 10:		Line 10:
	For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.		For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.

	[[Image:preprocess.png\|~~480px~~\|Preprocessing Flowchart]]		[[Image:preprocess.png\|640px\|Preprocessing Flowchart]]

	== Indexers ==		== Indexers ==

Scsong at 20:12, 10 November 2009

2009-11-10T20:12:43Z

← Older revision		Revision as of 20:12, 10 November 2009
Line 10:		Line 10:
	For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.		For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.

	[[Image:preprocess.png\|Preprocessing Flowchart]]		[[Image:preprocess.png\|480px\|Preprocessing Flowchart]]

	== Indexers ==		== Indexers ==

Scsong at 20:11, 10 November 2009

2009-11-10T20:11:50Z

← Older revision		Revision as of 20:11, 10 November 2009
Line 10:		Line 10:
	For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.		For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.

	[[Image:preprocess.~~gif\|640px~~\|Preprocessing Flowchart]]		[[Image:preprocess.png\|Preprocessing Flowchart]]

	== Indexers ==		== Indexers ==

Scsong at 20:08, 10 November 2009

2009-11-10T20:08:56Z

← Older revision		Revision as of 20:08, 10 November 2009
Line 10:		Line 10:
	For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.		For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.

			[[Image:preprocess.gif\|640px\|Preprocessing Flowchart]]

	== Indexers ==		== Indexers ==
	*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), not only for the entire index, but only for each individual term in the index.		*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), not only for the entire index, but only for each individual term in the index.

			[[Image:index.gif\|640px\|Indexing Flowchart]]

	== Distributed Index Support ==		== Distributed Index Support ==
Line 25:		Line 28:
	*[[Webarc:Temporal Search Client/Server\|tsearch: Temporal Search Client/Server]] (C): Using RPC, a search server provides two remote procedures for clients: getstats and search. Getstats returns statistics (such as doc count, term count, fresh doc count, ...) upon a provisional query, while search returns the actual search results for a query. Each search server is multithreaded and can handle multiple indexes simultaneously. Each search client is also multithreaded and issues a search request to multiple servers simultaneously.		*[[Webarc:Temporal Search Client/Server\|tsearch: Temporal Search Client/Server]] (C): Using RPC, a search server provides two remote procedures for clients: getstats and search. Getstats returns statistics (such as doc count, term count, fresh doc count, ...) upon a provisional query, while search returns the actual search results for a query. Each search server is multithreaded and can handle multiple indexes simultaneously. Each search client is also multithreaded and issues a search request to multiple servers simultaneously.
	*[[Webarc:PBS Jobs Scripts for Searching\|PBS Jobs Scripts for Searching]] (Bash): These scripts allow search servers and clients to be deployed over multiple nodes in the Chimera cluster. For n nodes, and i indexes, each node is made responsible for i/n indexes. Indexes are distributed in an interlaced fashion: i.e. node 1 has index 1, n+1, 2n+1, ... and node 2 has index 2, n+2, 2n+2, ..., and so on.		*[[Webarc:PBS Jobs Scripts for Searching\|PBS Jobs Scripts for Searching]] (Bash): These scripts allow search servers and clients to be deployed over multiple nodes in the Chimera cluster. For n nodes, and i indexes, each node is made responsible for i/n indexes. Indexes are distributed in an interlaced fashion: i.e. node 1 has index 1, n+1, 2n+1, ... and node 2 has index 2, n+2, 2n+2, ..., and so on.

			[[Image:search.gif\|640px\|Distributed Search]]

	== Result Analysis ==		== Result Analysis ==

Scsong at 19:42, 10 November 2009

2009-11-10T19:42:36Z

Scsong at 01:20, 10 November 2009

2009-11-10T01:20:53Z

← Older revision		Revision as of 01:20, 10 November 2009
Line 27:		Line 27:

	== Result Analysis ==		== Result Analysis ==
	*[[Webarc:Trec_Eval (modified)\|Trec_Eval (modified)]] (C): We added Kendall's Tau ~~(http://en.wikipedia.org/wiki/Kendall_tau_rank_correlation_coefficient) computation~~ support in the existing Trec_Eval tool ~~(http://trec.nist.gov/trec_eval/)~~.		*[[Webarc:Trec_Eval (modified)\|Trec_Eval (modified)]] (C): We added Kendall's Tau support in the existing Trec_Eval tool.

	== Miscellanea ==		== Miscellanea ==
	*[[Webarc:Berkeley DB Wrapper for Carryover DB\|Berkeley DB Wrapper for Carryover DB]] (Java): A simple and elegant wrapper for any C/C++ implementations that want to access Carryover DB (which is based on Java Berkeley DB) via Java JNI.		*[[Webarc:Berkeley DB Wrapper for Carryover DB\|Berkeley DB Wrapper for Carryover DB]] (Java): A simple and elegant wrapper for any C/C++ implementations that want to access Carryover DB (which is based on Java Berkeley DB) via Java JNI.

← Older revision		Revision as of 19:42, 10 November 2009
Line 1:		Line 1:
	== Input Data Preprocessors ==		== Input Data Preprocessors ==
	*[[Webarc:LRE Monthly Dumper\|LRE Monthly Dumper]] (Java): Based on mwdumper, this tool extracts from the MediaWiki XML dump a monthly snapshot at the end of each month. It also filters out minor edits and redirects.		*[[Webarc:LRE Monthly Dumper\|mwdumper: LRE Monthly Dumper]] (Java): Based on mwdumper, this tool extracts from the MediaWiki XML dump a monthly snapshot at the end of each month. It also filters out minor edits and redirects.

	*[[Webarc:MediaWiki-to-TREC Converter\|MediaWiki-to-TREC Converter]] (Java): Also based on mwdumper, this tool converts the default MediaWiki XML dump format to the TREC-complaint format. For our experimental purposes, it also constructs a Java Berkeley DB for each month, which we call 'Fresh DB'. A 'Fresh DB' contains records corresponding to all wiki articles updated within the month. Each record in DB has the form of { docID, (revision date, file name, offset) }.		*[[Webarc:MediaWiki-to-TREC Converter\|mwprep: MediaWiki-to-TREC Converter]] (Java): Also based on mwdumper, this tool converts the default MediaWiki XML dump format to the TREC-complaint format. For our experimental purposes, it also constructs a Java Berkeley DB for each month, which we call 'Fresh DB'. A 'Fresh DB' contains records corresponding to all wiki articles updated within the month. Each record in DB has the form of { docID, (revision date, file name, offset) }.

	*[[Webarc:Merge DB Constructor\|Merge DB Constructor]] (Java): This tool constructs Merge DB for each month, which contains the union set of records between Merge DB of the previous month and Fresh DB of the current month. I.e. for month m, <math>MergeDB_m = MergeDB_{m-1} \cup FreshDB_m</math>. Since constructing a Merge DB for each month requires an existing Merge DB for the previous month, this tool needs to be run sequentially from the first month to the last month.		*[[Webarc:Merge DB Constructor\|colstate: Merge DB Constructor]] (Java): This tool constructs Merge DB for each month, which contains the union set of records between Merge DB of the previous month and Fresh DB of the current month. I.e. for month m, <math>MergeDB_m = MergeDB_{m-1} \cup FreshDB_m</math>. Since constructing a Merge DB for each month requires an existing Merge DB for the previous month, this tool needs to be run sequentially from the first month to the last month.

	*[[Webarc:Carryover DB Constructor\|Carryover DB Constructor]] (Java): By comparing Fresh DB and Merge DB for each month, this tool identifies the Wiki articles that need to be carried over from the previous month (i.e. those that only appear in Merge DB, not in the monthly snapshot). It constructs yet another DB (Carryover DB) for each month that contains these carryovers. I.e. for month m, <math>CarryoverDB_m = MergeDB_m - FreshDB_m</math>.		*[[Webarc:Carryover DB Constructor\|codb: Carryover DB Constructor]] (Java): By comparing Fresh DB and Merge DB for each month, this tool identifies the Wiki articles that need to be carried over from the previous month (i.e. those that only appear in Merge DB, not in the monthly snapshot). It constructs yet another DB (Carryover DB) for each month that contains these carryovers. I.e. for month m, <math>CarryoverDB_m = MergeDB_m - FreshDB_m</math>.

	For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.		For our experiments, we have run the tools listed above sequentially to obtain monthly snapshots of Wikipedia articles from 2001 to 2007, and also to identify, for each month, the articles that do not have new revisions in the current month, thus need to be carried over and indexed for the current month.
Line 12:		Line 12:

	== Indexers ==		== Indexers ==
	*[[Webarc:Lemur Indexer (modified)\|Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), not only for the entire index, but only for each individual term in the index.		*[[Webarc:Lemur Indexer (modified)\|IndriBuildIndex: Lemur Indexer (modified)]] (C++): Based on the Lemur toolkit, we added additional input source support, namely Berkeley DB (for the Carryover DB). We also added extra statistics parameters for temporal scoring support. In particular, the modified index now also includes 'fresh document counts' (the number of non-carried-over documents) and 'term counts for fresh documents' (the number of terms within non-carried-over documents), not only for the entire index, but only for each individual term in the index.

	== Distributed Index Support ==		== Distributed Index Support ==
Line 23:		Line 23:

	== Distributed Search Support ==		== Distributed Search Support ==
	*[[Webarc:Temporal Search Client/Server\|Temporal Search Client/Server]] (C): Using RPC, a search server provides two remote procedures for clients: getstats and search. Getstats returns statistics (such as doc count, term count, fresh doc count, ...) upon a provisional query, while search returns the actual search results for a query. Each search server is multithreaded and can handle multiple indexes simultaneously. Each search client is also multithreaded and issues a search request to multiple servers simultaneously.		*[[Webarc:Temporal Search Client/Server\|tsearch: Temporal Search Client/Server]] (C): Using RPC, a search server provides two remote procedures for clients: getstats and search. Getstats returns statistics (such as doc count, term count, fresh doc count, ...) upon a provisional query, while search returns the actual search results for a query. Each search server is multithreaded and can handle multiple indexes simultaneously. Each search client is also multithreaded and issues a search request to multiple servers simultaneously.
	*[[Webarc:PBS Jobs Scripts for Searching\|PBS Jobs Scripts for Searching]] (Bash): These scripts allow search servers and clients to be deployed over multiple nodes in the Chimera cluster. For n nodes, and i indexes, each node is made responsible for i/n indexes. Indexes are distributed in an interlaced fashion: i.e. node 1 has index 1, n+1, 2n+1, ... and node 2 has index 2, n+2, 2n+2, ..., and so on.		*[[Webarc:PBS Jobs Scripts for Searching\|PBS Jobs Scripts for Searching]] (Bash): These scripts allow search servers and clients to be deployed over multiple nodes in the Chimera cluster. For n nodes, and i indexes, each node is made responsible for i/n indexes. Indexes are distributed in an interlaced fashion: i.e. node 1 has index 1, n+1, 2n+1, ... and node 2 has index 2, n+2, 2n+2, ..., and so on.