Webarc:MediaWiki-to-TREC Converter - Revision history

Scsong at 23:33, 9 November 2009

2009-11-09T23:33:14Z

← Older revision		Revision as of 23:33, 9 November 2009
Line 14:		Line 14:
	In Eclipse, export mwprep as a runnable JAR.		In Eclipse, export mwprep as a runnable JAR.
	# Right-click on 'mwprep' in Package Explorer, select 'export'.		# Right-click on 'mwprep' in Package Explorer, select 'export'.
			# Select 'Runnable JAR file' and click 'Next'.
	# Select 'mwprep - mwprep' as Launch configuration.		# Select 'mwprep - mwprep' as Launch configuration.
	# Put mwprep.jar in Export destination.		# Put mwprep.jar in Export destination.
Line 19:		Line 20:
	# Click 'Finish'		# Click 'Finish'

			== How To Run ==
	In a shell terminal (or a command line prompt in Windows), change directory to where mwprep.jar is located (<your directory> above).		In a shell terminal (or a command line prompt in Windows), change directory to where mwprep.jar is located (<your directory> above).

Scsong at 22:59, 9 November 2009

2009-11-09T22:59:53Z

← Older revision		Revision as of 22:59, 9 November 2009
Line 15:		Line 15:
	# Right-click on 'mwprep' in Package Explorer, select 'export'.		# Right-click on 'mwprep' in Package Explorer, select 'export'.
	# Select 'mwprep - mwprep' as Launch configuration.		# Select 'mwprep - mwprep' as Launch configuration.
	# Put mwprep.jar in Export destination		# Put mwprep.jar in Export destination.
	# Select 'Package required libraries into generated JAR'		# Select 'Package required libraries into generated JAR'
	# Click 'Finish'		# Click 'Finish'

	In a shell terminal (or a command line prompt in Windows), change directory to where ~~lremonthlydumper~~.jar is located.		In a shell terminal (or a command line prompt in Windows), change directory to where mwprep.jar is located (<your directory> above).

	<pre>		<pre>

Scsong at 22:45, 9 November 2009

2009-11-09T22:45:57Z

← Older revision		Revision as of 22:45, 9 November 2009
Line 24:		Line 24:
	mkdir preprocessed-monthly		mkdir preprocessed-monthly
	mkdir bdb-monthly		mkdir bdb-monthly
	java -jar mwprep.jar <~~WikiDump~~.lst>		java -jar mwprep.jar < snapshots.lst>
	</pre>		</pre>

	== Input File ==		== Input File ==
	''<~~WikiDump~~.lst>'': A file that lists the locations of the monthly snapshots extracted from the WikiMedia XML dump. You probably want to break down the entire list into multiple smaller files and run multiple mwprep's in parallel.		''<snapshots.lst>'': A file that lists the locations of the monthly snapshots extracted from the WikiMedia XML dump. You probably want to break down the entire list into multiple smaller files and run multiple mwprep's in parallel.

	Example contents in a list file:		Example contents in a list file:

Scsong at 22:16, 9 November 2009

2009-11-09T22:16:24Z

← Older revision		Revision as of 22:16, 9 November 2009
Line 23:		Line 23:
	<pre>		<pre>
	mkdir preprocessed-monthly		mkdir preprocessed-monthly
			mkdir bdb-monthly
	java -jar mwprep.jar <WikiDump.lst>		java -jar mwprep.jar <WikiDump.lst>
	</pre>		</pre>

Scsong at 22:16, 9 November 2009

2009-11-09T22:16:08Z

← Older revision		Revision as of 22:16, 9 November 2009
Line 44:		Line 44:

	== Notes ==		== Notes ==
	The ~~name~~ of the output directories (preprocessed-monthly and bdb-monthly) are hard-coded in edu.umd.umiacs.mw.mwprep. Change these as necessary, or even better, parameterize them.		The names of the output directories (preprocessed-monthly and bdb-monthly) are hard-coded in edu.umd.umiacs.mw.mwprep. Change these as necessary, or even better, parameterize them.

	== Source Codes ==		== Source Codes ==
	svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwprep		svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwprep

Scsong at 22:15, 9 November 2009

2009-11-09T22:15:57Z

← Older revision		Revision as of 22:15, 9 November 2009
Line 46:		Line 46:
	The name of the output directories (preprocessed-monthly and bdb-monthly) are hard-coded in edu.umd.umiacs.mw.mwprep. Change these as necessary, or even better, parameterize them.		The name of the output directories (preprocessed-monthly and bdb-monthly) are hard-coded in edu.umd.umiacs.mw.mwprep. Change these as necessary, or even better, parameterize them.

	== Source ~~Code~~ ==		== Source Codes ==
	svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwprep		svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwprep

Scsong at 22:15, 9 November 2009

2009-11-09T22:15:49Z

← Older revision		Revision as of 22:15, 9 November 2009
Line 26:		Line 26:
	</pre>		</pre>

	== Input ==		== Input File ==
	''<WikiDump.lst>'': A file that lists the locations of the monthly snapshots extracted from the WikiMedia XML dump. You probably want to break down the entire list into multiple smaller files and run multiple mwprep's in parallel.		''<WikiDump.lst>'': A file that lists the locations of the monthly snapshots extracted from the WikiMedia XML dump. You probably want to break down the entire list into multiple smaller files and run multiple mwprep's in parallel.

Line 39:		Line 39:


	== Output ==		== Output Files ==
	# Under the 'preprocessed-monthly' directory, TREC-complaint files. The converted file for the original XML file <filename>.xml is named as trec-<filename>.xml.		# Under the 'preprocessed-monthly' directory, TREC-complaint files. The converted file for the original XML file <filename>.xml is named as trec-<filename>.xml.
	# Under the 'bdb-monthly' directory, directories for Fresh DBs. Each directory corresponds to a DB which in turn corresponds to a single month. The DB directories are named as '<filename>-fresh'.		# Under the 'bdb-monthly' directory, directories for Fresh DBs. Each directory corresponds to a DB which in turn corresponds to a single month. The DB directories are named as '<filename>-fresh'.

Scsong at 22:15, 9 November 2009

2009-11-09T22:15:35Z

← Older revision		Revision as of 22:15, 9 November 2009
Line 1:		Line 1:
	== What ~~it does~~ ==		== What It Does ==
	This tool converts the default MediaWiki XML dump format to the TREC-complaint format. For our experimental purposes, it also constructs a Java Berkeley DB for each month, which we call 'Fresh DB'. A 'Fresh DB' contains records corresponding to all wiki articles updated within a month. Each record in DB has the form of { docID, (revision date, file name, offset) }.		This tool converts the default MediaWiki XML dump format to the TREC-complaint format. For our experimental purposes, it also constructs a Java Berkeley DB for each month, which we call 'Fresh DB'. A 'Fresh DB' contains records corresponding to all wiki articles updated within a month. Each record in DB has the form of { docID, (revision date, file name, offset) }.

	== ~~Usage~~ ==		== How To Build ==
	In Eclipse, configure a run		In Eclipse, configure a run
	# Right-click on 'mwprep' in Package Explorer, select 'Run As.. --> Run Configurations'.		# Right-click on 'mwprep' in Package Explorer, select 'Run As.. --> Run Configurations'.
Line 10:		Line 10:
	# Select 'edu.umd.umiacs.mw.mwprep' in the Main class field		# Select 'edu.umd.umiacs.mw.mwprep' in the Main class field
	# Click 'Apply'		# Click 'Apply'
			# Click 'Close'

			In Eclipse, export mwprep as a runnable JAR.
	export mwprep as a runnable JAR.
	# Right-click on 'mwprep' in Package Explorer, select 'export'.		# Right-click on 'mwprep' in Package Explorer, select 'export'.
	# ~~Right-click on~~ 'mwprep~~' in Package Explorer, select 'export'.~~		# Select 'mwprep - mwprep' as Launch configuration.
	~~# Select 'lremonthlydumper20012007~~ - ~~mwdumper~~' as Launch configuration.		# Put mwprep.jar in Export destination
	# Put ~~lremonthlydumper~~.jar in Export destination
	# Select 'Package required libraries into generated JAR'		# Select 'Package required libraries into generated JAR'
	# Click 'Finish'		# Click 'Finish'
Line 23:		Line 22:

	<pre>		<pre>
	mkdir ~~extracted~~-monthly		mkdir preprocessed-monthly
	java -jar ~~lremonthlydumper~~.jar <WikiDump.lst>		java -jar mwprep.jar <WikiDump.lst>
	</pre>		</pre>

	== Input ==		== Input ==
	''<WikiDump.lst>'': A file that lists the locations of WikiMedia XML dump files		''<WikiDump.lst>'': A file that lists the locations of the monthly snapshots extracted from the WikiMedia XML dump. You probably want to break down the entire list into multiple smaller files and run multiple mwprep's in parallel.

	Example contents in a list file:		Example contents in a list file:
	<pre>		<pre>
	/fs/~~webarc4~~/data/wikipedia/~~enwiki~~-~~20080103~~-~~pages-meta-history-00~~		/fs/webarc3/data/wikipedia/extracted-monthly/month-000.xml
	/fs/~~webarc4~~/data/wikipedia/~~enwiki~~-~~20080103-pages-meta-history-01~~		/fs/webarc3/data/wikipedia/extracted-monthly/month-001.xml
	/~~fs/webarc4/data/wikipedia/enwiki-20080103-pages~~-~~meta-history-02~~		/fs/webarc3/data/wikipedia/extracted-monthly/month-002.xml
	/fs/~~webarc4~~/data/wikipedia/~~enwiki~~-~~20080103-pages-meta-history-03~~		/fs/webarc3/data/wikipedia/extracted-monthly/month-003.xml
	/~~fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta~~-~~history-04~~		/fs/webarc3/data/wikipedia/extracted-monthly/month-004.xml
	/fs/~~webarc4~~/data/wikipedia/~~enwiki~~-~~20080103-pages-meta-history-05~~
	~~/fs~~/~~webarc4/data/wikipedia/enwiki-20080103-pages-meta-history~~-06
	/fs/~~webarc4~~/data/wikipedia/~~enwiki~~-~~20080103-pages-meta-history-07~~
	~~/fs~~/~~webarc4/data/wikipedia/enwiki-20080103-pages-meta-history~~-08
	</pre>		</pre>


	== Output ==		== Output ==
	Under the '~~extracted~~-monthly' directory, ~~WikiMedia XML dump~~ files ~~(one for each month) are generated~~. The file ~~month~~-''<k>''.~~xml~~ corresponds to ~~the extracted wiki articles for the~~ ''~~k''th month~~.		# Under the 'preprocessed-monthly' directory, TREC-complaint files. The converted file for the original XML file <filename>.xml is named as trec-<filename>.xml.
			# Under the 'bdb-monthly' directory, directories for Fresh DBs. Each directory corresponds to a DB which in turn corresponds to a single month. The DB directories are named as '<filename>-fresh'.

	== Notes ==		== Notes ==
	The ~~range~~ of ~~months is~~ hard-coded ~~as January 2001 ~ December 2007~~ in ~~org~~.~~mediawiki~~.~~dumper~~.~~LREMonthlyDumper20012007, and so is the output directory~~. Change these as necessary, or even better, parameterize them.		The name of the output directories (preprocessed-monthly and bdb-monthly) are hard-coded in edu.umd.umiacs.mw.mwprep. Change these as necessary, or even better, parameterize them.

	== Source Code ==		== Source Code ==
	svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/~~mwdumper~~		svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwprep

Scsong at 22:02, 9 November 2009

2009-11-09T22:02:21Z

New page

== What it does ==
This tool converts the default MediaWiki XML dump format to the TREC-complaint format. For our experimental purposes, it also constructs a Java Berkeley DB for each month, which we call 'Fresh DB'. A 'Fresh DB' contains records corresponding to all wiki articles updated within a month. Each record in DB has the form of { docID, (revision date, file name, offset) }.

== Usage ==
In Eclipse, configure a run
# Right-click on 'mwprep' in Package Explorer, select 'Run As.. --> Run Configurations'.
# On the left pane, right click on 'Java Application --> New'
# Enter 'mwprep' in the Name field on the right pane.
# Select 'mwprep' in the Project field.
# Select 'edu.umd.umiacs.mw.mwprep' in the Main class field
# Click 'Apply'

export mwprep as a runnable JAR.
# Right-click on 'mwprep' in Package Explorer, select 'export'.
# Right-click on 'mwprep' in Package Explorer, select 'export'.
# Select 'lremonthlydumper20012007 - mwdumper' as Launch configuration.
# Put lremonthlydumper.jar in Export destination
# Select 'Package required libraries into generated JAR'
# Click 'Finish'

In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located.

<pre>
mkdir extracted-monthly
java -jar lremonthlydumper.jar <WikiDump.lst>
</pre>

== Input ==
''<WikiDump.lst>'': A file that lists the locations of WikiMedia XML dump files
Example contents in a list file:
<pre>
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-00
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-01
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-02
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-03
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-04
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-05
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-06
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-07
/fs/webarc4/data/wikipedia/enwiki-20080103-pages-meta-history-08
</pre>

== Output ==
Under the 'extracted-monthly' directory, WikiMedia XML dump files (one for each month) are generated. The file month-''<k>''.xml corresponds to the extracted wiki articles for the ''k''th month.

== Notes ==
The range of months is hard-coded as January 2001 ~ December 2007 in org.mediawiki.dumper.LREMonthlyDumper20012007, and so is the output directory. Change these as necessary, or even better, parameterize them.

== Source Code ==
svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwdumper