Scsong at 15:21, 11 November 2009

2009-11-11T15:21:30Z

← Older revision		Revision as of 15:21, 11 November 2009
Line 1:		Line 1:
	== What It Does ==		== What It Does ==
	This is an RPC server/client implementation for our temporal search experiments. The server provides two remote procedures for clients to call: getstats and search. ~~Getstats~~ returns statistics (such as doc count, term count, fresh doc count, ...) upon a provisional query, while search returns the actual search results for a query. Each search server is multithreaded and can handle multiple indexes simultaneously. Each search client is also multithreaded and issues a search request to multiple servers simultaneously.		This is an RPC server/client implementation for our temporal search experiments. The server provides two remote procedures for clients to call: ''getstats'' and ''search''. ''getstat''s returns statistics (such as doc count, term count, fresh doc count, ...) upon a provisional query, while ''search'' returns the actual search results for a query. Each search server is multithreaded and can handle multiple indexes simultaneously. Each search client is also multithreaded and issues a search request to multiple servers simultaneously.

	== RPC interface definition ==		== RPC interface definition ==

Scsong at 15:20, 11 November 2009

2009-11-11T15:20:30Z

← Older revision		Revision as of 15:20, 11 November 2009
Line 111:		Line 111:
	</pre>		</pre>

	''<query terms list>'': A file that lists search ~~terms~~ to be used.		''<query terms list>'': A file that lists search phrases to be used.
	Example:		Example:
	<pre>		<pre>
Line 118:		Line 118:
	</pre>		</pre>

	''<query spans list>'': A file that lists {begin time - end time} pairs to be used for each search ~~terms~~ listed in ''<query terms list>''		''<query spans list>'': A file that lists {begin time - end time} pairs to be used for each search phrase listed in ''<query terms list>''
	Example:		Example:
	<pre>		<pre>
Line 161:		Line 161:

	== Output File ==		== Output File ==
	Search server makes no output file. Search client creates ''<query span list>_<time windows list>.out'' under the current directory.		Search server makes no output file. Search client creates a Trec_Eval complaint output file with the name of ''<query span list>_<time windows list>.out'' under the current directory.

	== Notes ==		== Notes ==

Scsong at 02:00, 10 November 2009

2009-11-10T02:00:24Z

New page

== What It Does ==
This is an RPC server/client implementation for our temporal search experiments. The server provides two remote procedures for clients to call: getstats and search. Getstats returns statistics (such as doc count, term count, fresh doc count, ...) upon a provisional query, while search returns the actual search results for a query. Each search server is multithreaded and can handle multiple indexes simultaneously. Each search client is also multithreaded and issues a search request to multiple servers simultaneously.

== RPC interface definition ==
<pre>
struct searchhits_struct {
string docId<>;
double score;
searchhits_struct *next;
};
typedef struct searchhits_struct searchhits;

struct terms_counts_struct {
string term<>;
unsigned int docCount;
unsigned int termCount_high32;
unsigned int termCount_low32;
terms_counts_struct *next;
};
typedef struct terms_counts_struct terms_counts;

struct terms_no_counts_struct {
string term<>;
terms_no_counts_struct *next;
};
typedef struct terms_no_counts_struct terms_no_counts;

struct col_stats_struct {
unsigned int docCount;
float docAverageLength;
unsigned int termCountUnique;
unsigned int termCount_high32;
unsigned int termCount_low32;
};
typedef col_stats_struct col_stats;

struct final_query_struct {
terms_counts terms;
unsigned int beginTime;
unsigned int endTime;
unsigned int maxNoHits;
col_stats colStat;
};
typedef struct final_query_struct final_query;

struct probe_query_struct {
terms_no_counts terms;
unsigned int beginTime;
unsigned int endTime;
};
typedef struct probe_query_struct probe_query;

program tsearch {
version v1 {
searchhits search(final_query) = 1;
final_query getstats(probe_query) = 2;
}=1;
}=0x00001000;
</pre>

== How To Build ==
In tsearch directory,
<pre>
make
</pre>

== How To Run ==
=== Run Search Server ===
<pre>
cd bin
./tsearchsvr <timewindow configuration file>
</pre>
=== Run Search Client ===
<pre>
cd bin
./tsearchcli <rpc server list> <query terms list> <query spans list> <time windows list>
</pre>

== Input File ==
=== Search Server ===
''<timewindow configuration file>'': A file that lists the locations of the indexes and the retrieval method to be used.

Example contents in a list file:
<pre>
1 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-001 TEMP_OKAPI
2 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-002 TEMP_OKAPI
3 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-003 TEMP_OKAPI
4 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-004 TEMP_OKAPI
5 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-005 TEMP_OKAPI
6 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-006 TEMP_OKAPI
7 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-007 TEMP_OKAPI
8 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-008 TEMP_OKAPI
9 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-009 TEMP_OKAPI
10 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-010 TEMP_OKAPI
11 /fs/webarc3/data/wikipedia/lemur_index/monthly/month-011 TEMP_OKAPI
</pre>

=== Search Client ===
''<rpc server list>'': A file that lists the locations of the RPC search servers to connect.
Example:
<pre>
chimera00
chimera02
chimera03
chimera04
chimera05
chimera06
chimera07
chimera08
</pre>

''<query terms list>'': A file that lists search terms to be used.
Example:
<pre>
1901 uk census
2004 taxes income from business partnership deductions
</pre>

''<query spans list>'': A file that lists {begin time - end time} pairs to be used for each search terms listed in ''<query terms list>''
Example:
<pre>
1-64
2-65
3-66
4-67
5-68
6-69
7-70
8-71
9-72
10-73
11-74
12-75
13-76
14-77
15-78
16-79
17-80
18-81
19-82
20-83
</pre>

''<time windows list>'': A file that lists the temporal points where a new time window begins. The last number in the list is when the last time window ends.
Example:
<pre>
0
8
16
24
32
40
48
56
64
72
80
83
</pre>

== Output File ==
Search server makes no output file. Search client creates ''<query span list>_<time windows list>.out'' under the current directory.

== Notes ==
* When running tsearchcli, place ''<query spans list>'' and ''<time windows list>'' in the current directory, and use bare file names without any preceding directory description. (I.e. 'qts.def' rather than './qts.def' or '/tmp/qts.def).
* The current RPC implementation in Linux has the following limitations:
** Although individual procedures can create multithreads to complete the job more quickly, the procedure itself cannot be multithreaded. (I.e. multiple calls to the same procedure will be processed sequentially).
** There is no 8-byte data type support. Even the 'long' type in a 64-bit machine is not properly parsed in the RPC implementation. We walked around this issue by defining two 4-byte integer variables, each of which represents the low/upper half of the 64-bit value.

== Source Codes ==
svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/tsearch

Webarc:Temporal Search Client/Server - Revision history

Scsong at 15:21, 11 November 2009

Scsong at 15:20, 11 November 2009

Scsong at 02:00, 10 November 2009