Webarc:Carryover DB Constructor

What It Does

By comparing Fresh DB and Merge DB for each month, this tool identifies the Wiki articles that need to be carried over from the previous month (i.e. those that only appear in Merge DB, not in the monthly snapshot). It constructs yet another DB (Carryover DB) for each month that contains these carryovers. I.e. for month m, <math>CarryoverDB_m = MergeDB_m - FreshDB_m</math>.

How To Build

In Eclipse, configure a run

1. Right-click on 'colstate' in Package Explorer, select 'Run As.. --> Run Configurations'.
2. On the left pane, right click on 'Java Application --> New'
3. Enter 'codb' in the Name field on the right pane.
4. Select 'colstate' in the Project field.
5. Select 'edu.umd.umiacs.temporalscoring.ConstructCarryOver' in the Main class field
6. Click 'Apply'
7. Click 'Close'

In Eclipse, export 'colstate' as a runnable JAR.

1. Right-click on 'colstate' in Package Explorer, select 'export'.
2. Select 'codb - colstate' as Launch configuration.
3. Put <your directory>/codb.jar in Export destination.
4. Select 'Package required libraries into generated JAR'
5. Click 'Finish'

In a shell terminal (or a command line prompt in Windows), change directory to where codb.jar is located (<your directory> above).

java -jar codb.jar <MergeDBNames.lst>

Input File

<MergeDBNames.lst>: A file that lists the locations of the Merge DBs. The name of Fresh DB is assumed to be the concatenation of the name of the Merge DB for the same month and '-fresh'. For example, for Merge DB 'month-003', Fresh DB is assumed to be 'month-003-fresh'. You probably want to break down the entire list into multiple smaller files and run multiple mwprep's in parallel.

Example contents in a list file:

/fs/webarc3/data/wikipedia/bdb-monthly/month-000
/fs/webarc3/data/wikipedia/bdb-monthly/month-001
/fs/webarc3/data/wikipedia/bdb-monthly/month-002
/fs/webarc3/data/wikipedia/bdb-monthly/month-003

Output Files

Under the same directory under which Fresh DBs are located, new directories for Carryover DBs are generated. The new directories are named by concatenating '-co' at the end of Merge DB names. I.e. given a month, if Merge DB name is <month-003>, Carryover DB name will be given as <month-003-co>.

Notes

• As in the Merge DB Construction process, there is an assumption on the Fresh DB names. For month m, if MergeDB name is <month-m>, FreshDB name is assumed to be <month-m-fresh>.
• This carryover identification step could have been performed during the Merge DB Construction process. However, we detached this step for two reasons:
  • Having four indexes (one Fresh DB and two Merge DBs and one Carryover DB) for each month requires a significant system resource, and often slows down the whole process.
  • Unlike the Merge DB Construction process, Carryover DBs can be constructed in parallel.

Source Codes

svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/colstate