Personal tools

Webarc:Merge DB Constructor

From Adapt

Revision as of 22:45, 9 November 2009 by Scsong (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

What It Does

This tool constructs Merge DB for each month, which contains the union set of records between Merge DB of the previous month and Fresh DB of the current month. I.e. For month m, <math>MergeDB_m = MergeDB_{m-1} \cup FreshDB_m</math>. Since constructing a Merge DB for each month requires an existing Merge DB for the previous month, this tool needs to be run sequentially from the first month to the last month.

How To Build

In Eclipse, configure a run

  1. Right-click on 'colstate' in Package Explorer, select 'Run As.. --> Run Configurations'.
  2. On the left pane, right click on 'Java Application --> New'
  3. Enter 'mdb' in the Name field on the right pane.
  4. Select 'colstate' in the Project field.
  5. Select 'edu.umd.umiacs.temporalscoring.CollectionState' in the Main class field
  6. Click 'Apply'
  7. Click 'Close'

In Eclipse, export 'colstate' as a runnable JAR.

  1. Right-click on 'colstate' in Package Explorer, select 'export'.
  2. Select 'mdb- mwprep' as Launch configuration.
  3. Put mdb.jar in Export destination
  4. Select 'Package required libraries into generated JAR'
  5. Click 'Finish'

In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located. 

java -jar mdb.jar <MergeDBNames.lst>

Input File

<MergeDBNames.lst>: A file that lists the locations of the Merge DBs. The name of Fresh DB is assumed to be the concatenation of the name of the Merge DB for the same month and '-fresh'. For example, for Merge DB 'month-003', Fresh DB is assumed to be 'month-003-fresh'. Note that since this tool cannot be run in parallel, this file needs to contain the entire list for all months.

Example contents in a list file: 

/fs/webarc3/data/wikipedia/bdb-monthly/month-000
/fs/webarc3/data/wikipedia/bdb-monthly/month-001
/fs/webarc3/data/wikipedia/bdb-monthly/month-002
/fs/webarc3/data/wikipedia/bdb-monthly/month-003
/fs/webarc3/data/wikipedia/bdb-monthly/month-004
/fs/webarc3/data/wikipedia/bdb-monthly/month-005
/fs/webarc3/data/wikipedia/bdb-monthly/month-006
/fs/webarc3/data/wikipedia/bdb-monthly/month-007
/fs/webarc3/data/wikipedia/bdb-monthly/month-008
/fs/webarc3/data/wikipedia/bdb-monthly/month-009
/fs/webarc3/data/wikipedia/bdb-monthly/month-010
/fs/webarc3/data/wikipedia/bdb-monthly/month-011
/fs/webarc3/data/wikipedia/bdb-monthly/month-012
/fs/webarc3/data/wikipedia/bdb-monthly/month-013
/fs/webarc3/data/wikipedia/bdb-monthly/month-014
/fs/webarc3/data/wikipedia/bdb-monthly/month-015
/fs/webarc3/data/wikipedia/bdb-monthly/month-016
/fs/webarc3/data/wikipedia/bdb-monthly/month-017
/fs/webarc3/data/wikipedia/bdb-monthly/month-018
/fs/webarc3/data/wikipedia/bdb-monthly/month-019
/fs/webarc3/data/wikipedia/bdb-monthly/month-020
/fs/webarc3/data/wikipedia/bdb-monthly/month-021
/fs/webarc3/data/wikipedia/bdb-monthly/month-022
/fs/webarc3/data/wikipedia/bdb-monthly/month-023
/fs/webarc3/data/wikipedia/bdb-monthly/month-024
/fs/webarc3/data/wikipedia/bdb-monthly/month-025
/fs/webarc3/data/wikipedia/bdb-monthly/month-026
/fs/webarc3/data/wikipedia/bdb-monthly/month-027
/fs/webarc3/data/wikipedia/bdb-monthly/month-028
/fs/webarc3/data/wikipedia/bdb-monthly/month-029
/fs/webarc3/data/wikipedia/bdb-monthly/month-030
/fs/webarc3/data/wikipedia/bdb-monthly/month-031
/fs/webarc3/data/wikipedia/bdb-monthly/month-032
/fs/webarc3/data/wikipedia/bdb-monthly/month-033
/fs/webarc3/data/wikipedia/bdb-monthly/month-034
/fs/webarc3/data/wikipedia/bdb-monthly/month-035
/fs/webarc3/data/wikipedia/bdb-monthly/month-036
/fs/webarc3/data/wikipedia/bdb-monthly/month-037
/fs/webarc3/data/wikipedia/bdb-monthly/month-038
/fs/webarc3/data/wikipedia/bdb-monthly/month-039
/fs/webarc3/data/wikipedia/bdb-monthly/month-040
/fs/webarc3/data/wikipedia/bdb-monthly/month-041
/fs/webarc3/data/wikipedia/bdb-monthly/month-042
/fs/webarc3/data/wikipedia/bdb-monthly/month-043
/fs/webarc3/data/wikipedia/bdb-monthly/month-044
/fs/webarc3/data/wikipedia/bdb-monthly/month-045
/fs/webarc3/data/wikipedia/bdb-monthly/month-046
/fs/webarc3/data/wikipedia/bdb-monthly/month-047
/fs/webarc3/data/wikipedia/bdb-monthly/month-048
/fs/webarc3/data/wikipedia/bdb-monthly/month-049
/fs/webarc3/data/wikipedia/bdb-monthly/month-050
/fs/webarc3/data/wikipedia/bdb-monthly/month-051
/fs/webarc3/data/wikipedia/bdb-monthly/month-052
/fs/webarc3/data/wikipedia/bdb-monthly/month-053
/fs/webarc3/data/wikipedia/bdb-monthly/month-054
/fs/webarc3/data/wikipedia/bdb-monthly/month-055
/fs/webarc3/data/wikipedia/bdb-monthly/month-056
/fs/webarc3/data/wikipedia/bdb-monthly/month-057
/fs/webarc3/data/wikipedia/bdb-monthly/month-058
/fs/webarc3/data/wikipedia/bdb-monthly/month-059
/fs/webarc3/data/wikipedia/bdb-monthly/month-060
/fs/webarc3/data/wikipedia/bdb-monthly/month-061
/fs/webarc3/data/wikipedia/bdb-monthly/month-062
/fs/webarc3/data/wikipedia/bdb-monthly/month-063
/fs/webarc3/data/wikipedia/bdb-monthly/month-064
/fs/webarc3/data/wikipedia/bdb-monthly/month-065
/fs/webarc3/data/wikipedia/bdb-monthly/month-066
/fs/webarc3/data/wikipedia/bdb-monthly/month-067
/fs/webarc3/data/wikipedia/bdb-monthly/month-068
/fs/webarc3/data/wikipedia/bdb-monthly/month-069
/fs/webarc3/data/wikipedia/bdb-monthly/month-070
/fs/webarc3/data/wikipedia/bdb-monthly/month-071
/fs/webarc3/data/wikipedia/bdb-monthly/month-072
/fs/webarc3/data/wikipedia/bdb-monthly/month-073
/fs/webarc3/data/wikipedia/bdb-monthly/month-074
/fs/webarc3/data/wikipedia/bdb-monthly/month-075
/fs/webarc3/data/wikipedia/bdb-monthly/month-076
/fs/webarc3/data/wikipedia/bdb-monthly/month-077
/fs/webarc3/data/wikipedia/bdb-monthly/month-078
/fs/webarc3/data/wikipedia/bdb-monthly/month-079
/fs/webarc3/data/wikipedia/bdb-monthly/month-080
/fs/webarc3/data/wikipedia/bdb-monthly/month-081
/fs/webarc3/data/wikipedia/bdb-monthly/month-082
/fs/webarc3/data/wikipedia/bdb-monthly/month-083

Output Files

  1. Under the same directory under which Fresh DBs are located, new directories for Merge DBs, whose names are as specified in the input file.

Notes

There is an assumption on the name of Fresh DB. For month m, if MergeDB name is <month-m>, FreshDB name is assumed to be <month-m-fresh>.

Source Codes

svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/colstate

Retrieved from "https://wiki.umiacs.umd.edu/adapt/index.php?title=Webarc:Merge_DB_Constructor&oldid=2489"