Webarc:Carryover DB Constructor - Revision history

Scsong at 23:33, 9 November 2009

2009-11-09T23:33:09Z

← Older revision		Revision as of 23:33, 9 November 2009
Line 14:		Line 14:
	In Eclipse, export 'colstate' as a runnable JAR.		In Eclipse, export 'colstate' as a runnable JAR.
	# Right-click on 'colstate' in Package Explorer, select 'export'.		# Right-click on 'colstate' in Package Explorer, select 'export'.
			# Select 'Runnable JAR file' and click 'Next'.
	# Select 'codb - colstate' as Launch configuration.		# Select 'codb - colstate' as Launch configuration.
	# Put <your directory>/codb.jar in Export destination.		# Put <your directory>/codb.jar in Export destination.
Line 19:		Line 20:
	# Click 'Finish'		# Click 'Finish'

			== How To Run ==
	In a shell terminal (or a command line prompt in Windows), change directory to where codb.jar is located (<your directory> above).		In a shell terminal (or a command line prompt in Windows), change directory to where codb.jar is located (<your directory> above).

Scsong at 23:00, 9 November 2009

2009-11-09T23:00:30Z

← Older revision		Revision as of 23:00, 9 November 2009
Line 42:		Line 42:
	* As in the Merge DB Construction process, there is an assumption on the Fresh DB names. For month m, if MergeDB name is <month-m>, FreshDB name is assumed to be <month-m-fresh>.		* As in the Merge DB Construction process, there is an assumption on the Fresh DB names. For month m, if MergeDB name is <month-m>, FreshDB name is assumed to be <month-m-fresh>.
	* This carryover identification step could have been performed during the Merge DB Construction process. However, we detached this step for two reasons:		* This carryover identification step could have been performed during the Merge DB Construction process. However, we detached this step for two reasons:
	** Having four indexes (one Fresh DB and two Merge DBs and one Carryover DB) for each month requires a significant system resource, and often slows down the whole process.		** Having four indexes (one Fresh DB and two Merge DBs and one Carryover DB) simultaneously opened for each month requires a significant system resource, and often slows down the whole process.
	** Unlike the Merge DB Construction process, Carryover DBs can be constructed in parallel.		** Unlike the Merge DB Construction process, Carryover DBs can be constructed in parallel.

	== Source Codes ==		== Source Codes ==
	svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/colstate		svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/colstate

Scsong at 22:59, 9 November 2009

2009-11-09T22:59:47Z

← Older revision		Revision as of 22:59, 9 November 2009
Line 42:		Line 42:
	* As in the Merge DB Construction process, there is an assumption on the Fresh DB names. For month m, if MergeDB name is <month-m>, FreshDB name is assumed to be <month-m-fresh>.		* As in the Merge DB Construction process, there is an assumption on the Fresh DB names. For month m, if MergeDB name is <month-m>, FreshDB name is assumed to be <month-m-fresh>.
	* This carryover identification step could have been performed during the Merge DB Construction process. However, we detached this step for two reasons:		* This carryover identification step could have been performed during the Merge DB Construction process. However, we detached this step for two reasons:
	# Having four indexes (one Fresh DB and two Merge DBs and one Carryover DB) for each month requires a significant system resource, and often slows down the whole process.		** Having four indexes (one Fresh DB and two Merge DBs and one Carryover DB) for each month requires a significant system resource, and often slows down the whole process.
	# Unlike the Merge DB Construction process, Carryover DBs can be constructed in parallel.		** Unlike the Merge DB Construction process, Carryover DBs can be constructed in parallel.

	== Source Codes ==		== Source Codes ==
	svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/colstate		svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/colstate

Scsong at 22:59, 9 November 2009

2009-11-09T22:59:15Z

New page

== What It Does ==
By comparing Fresh DB and Merge DB for each month, this tool identifies the Wiki articles that need to be carried over from the previous month (i.e. those that only appear in Merge DB, not in the monthly snapshot). It constructs yet another DB (Carryover DB) for each month that contains these carryovers. I.e. for month m, <math>CarryoverDB_m = MergeDB_m - FreshDB_m</math>.

== How To Build ==
In Eclipse, configure a run
# Right-click on 'colstate' in Package Explorer, select 'Run As.. --> Run Configurations'.
# On the left pane, right click on 'Java Application --> New'
# Enter 'codb' in the Name field on the right pane.
# Select 'colstate' in the Project field.
# Select 'edu.umd.umiacs.temporalscoring.ConstructCarryOver' in the Main class field
# Click 'Apply'
# Click 'Close'

In Eclipse, export 'colstate' as a runnable JAR.
# Right-click on 'colstate' in Package Explorer, select 'export'.
# Select 'codb - colstate' as Launch configuration.
# Put <your directory>/codb.jar in Export destination.
# Select 'Package required libraries into generated JAR'
# Click 'Finish'

In a shell terminal (or a command line prompt in Windows), change directory to where codb.jar is located (<your directory> above).

<pre>
java -jar codb.jar <MergeDBNames.lst>
</pre>

== Input File ==
''<MergeDBNames.lst>'': A file that lists the locations of the Merge DBs. The name of Fresh DB is assumed to be the concatenation of the name of the Merge DB for the same month and '-fresh'. For example, for Merge DB 'month-003', Fresh DB is assumed to be 'month-003-fresh'. You probably want to break down the entire list into multiple smaller files and run multiple mwprep's in parallel.

Example contents in a list file:
<pre>
/fs/webarc3/data/wikipedia/bdb-monthly/month-000
/fs/webarc3/data/wikipedia/bdb-monthly/month-001
/fs/webarc3/data/wikipedia/bdb-monthly/month-002
/fs/webarc3/data/wikipedia/bdb-monthly/month-003
</pre>

== Output Files ==
Under the same directory under which Fresh DBs are located, new directories for Carryover DBs are generated. The new directories are named by concatenating '-co' at the end of Merge DB names. I.e. given a month, if Merge DB name is <month-003>, Carryover DB name will be given as <month-003-co>.

== Notes ==
* As in the Merge DB Construction process, there is an assumption on the Fresh DB names. For month m, if MergeDB name is <month-m>, FreshDB name is assumed to be <month-m-fresh>.
* This carryover identification step could have been performed during the Merge DB Construction process. However, we detached this step for two reasons:
# Having four indexes (one Fresh DB and two Merge DBs and one Carryover DB) for each month requires a significant system resource, and often slows down the whole process.
# Unlike the Merge DB Construction process, Carryover DBs can be constructed in parallel.

== Source Codes ==
svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/colstate