Webarc:LRE Monthly Dumper - Revision history

Scsong at 18:45, 10 November 2009

2009-11-10T18:45:53Z

← Older revision		Revision as of 18:45, 10 November 2009
Line 4:		Line 4:
	== How To Build ==		== How To Build ==
	In Eclipse, configure a run		In Eclipse, configure a run
	# Right-click on 'mwdumper' in Package Explorer, select 'Run As.. --> Run Configurations'.		# Right-click on ''mwdumper'' in Package Explorer, select ''Run As.. --> Run Configurations''.
	# On the left pane, right click on 'Java Application --> New'		# On the left pane, right click on ''Java Application --> New''
	# Enter 'mwdumper' in the Name field on the right pane.		# Enter ''mwdumper'' in the Name field on the right pane.
	# Select 'mwdumper' in the Project field.		# Select ''mwdumper'' in the Project field.
	# Select 'org.mediawiki.dumper.~~LREMonthlyDumper20012007p~~' in the Main class field		# Select ''org.mediawiki.dumper.LREMonthlyDumper20012007'' in the Main class field
	# Click 'Apply'		# Click ''Apply''
	# Click 'Close'		# Click ''Close''

	In Eclipse, export mwdumper as a runnable JAR.		In Eclipse, export mwdumper as a runnable JAR.
Line 16:		Line 16:
	# Select 'Runnable JAR file' and click 'Next'.		# Select 'Runnable JAR file' and click 'Next'.
	# Select 'mwdumper- mwdumper' as Launch configuration.		# Select 'mwdumper- mwdumper' as Launch configuration.
	# Put <your directory>/mwdumper.jar in Export destination		# Put ''<your directory>/mwdumper.jar'' in Export destination
	# Select 'Package required libraries into generated JAR'		# Select 'Package required libraries into generated JAR'
	# Click 'Finish'		# Click 'Finish'

	== How To Run ==		== How To Run ==
	In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located (<your directory> above).		In a shell terminal (or a command line prompt in Windows), change directory to where ''lremonthlydumper.jar'' is located (''<your directory>'' above).

	<pre>		<pre>
Line 45:		Line 45:

	== Output Files ==		== Output Files ==
	Under the 'extracted-monthly' directory, WikiMedia XML dump files (one for each month) are generated. The file month-''<k>''.xml corresponds to the extracted wiki articles for the ''k''th month.		Under the 'extracted-monthly' directory, WikiMedia XML dump files (one for each month) are generated. The file ''month-<k>''.xml corresponds to the extracted wiki articles for the ''k''th month.

	== Notes ==		== Notes ==
	The range of months is hard-coded as January 2001 ~ December 2007 in org.mediawiki.dumper.LREMonthlyDumper20012007, and so is the output directory. Change these as necessary, or even better, parameterize them.		The range of months is hard-coded as January 2001 ~ December 2007 in ''org.mediawiki.dumper.LREMonthlyDumper20012007'', and so is the output directory. Change these as necessary, or even better, parameterize them.

	== Source Codes ==		== Source Codes ==
	svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwdumper		svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwdumper

Scsong at 23:33, 9 November 2009

2009-11-09T23:33:16Z

← Older revision		Revision as of 23:33, 9 November 2009
Line 14:		Line 14:
	In Eclipse, export mwdumper as a runnable JAR.		In Eclipse, export mwdumper as a runnable JAR.
	# Right-click on 'mwdumper' in Package Explorer, select 'export'.		# Right-click on 'mwdumper' in Package Explorer, select 'export'.
	#. Select 'mwdumper- mwdumper' as Launch configuration.		# Select 'Runnable JAR file' and click 'Next'.
			# Select 'mwdumper- mwdumper' as Launch configuration.
	# Put <your directory>/mwdumper.jar in Export destination		# Put <your directory>/mwdumper.jar in Export destination
	# Select 'Package required libraries into generated JAR'		# Select 'Package required libraries into generated JAR'

Scsong at 22:59, 9 November 2009

2009-11-09T22:59:55Z

← Older revision		Revision as of 22:59, 9 November 2009
Line 15:		Line 15:
	# Right-click on 'mwdumper' in Package Explorer, select 'export'.		# Right-click on 'mwdumper' in Package Explorer, select 'export'.
	#. Select 'mwdumper- mwdumper' as Launch configuration.		#. Select 'mwdumper- mwdumper' as Launch configuration.
	# Put mwdumper.jar in Export destination		# Put <your directory>/mwdumper.jar in Export destination
	# Select 'Package required libraries into generated JAR'		# Select 'Package required libraries into generated JAR'
	# Click 'Finish'		# Click 'Finish'

	== How To Run ==		== How To Run ==
	In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located.		In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located (<your directory> above).

	<pre>		<pre>

Scsong at 22:06, 9 November 2009

2009-11-09T22:06:40Z

← Older revision		Revision as of 22:06, 9 November 2009
Line 14:		Line 14:
	In Eclipse, export mwdumper as a runnable JAR.		In Eclipse, export mwdumper as a runnable JAR.
	# Right-click on 'mwdumper' in Package Explorer, select 'export'.		# Right-click on 'mwdumper' in Package Explorer, select 'export'.
	#. Select '~~lremonthlydumper20012007~~ - mwdumper' as Launch configuration.		#. Select 'mwdumper- mwdumper' as Launch configuration.
	# Put ~~lremonthlydumper~~.jar in Export destination		# Put mwdumper.jar in Export destination
	# Select 'Package required libraries into generated JAR'		# Select 'Package required libraries into generated JAR'
	# Click 'Finish'		# Click 'Finish'
Line 24:		Line 24:
	<pre>		<pre>
	mkdir extracted-monthly		mkdir extracted-monthly
	java -jar ~~lremonthlydumper~~.jar <WikiDump.lst>		java -jar mwdumper.jar <WikiDump.lst>
	</pre>		</pre>

Scsong at 22:04, 9 November 2009

2009-11-09T22:04:52Z

← Older revision		Revision as of 22:04, 9 November 2009
Line 1:		Line 1:
	== What ~~it does~~ ==		== What It Does ==
	From the source WikiMedia XML dump files, it extracts the latest revision for each article valid at the end of each month (namely at 11:59:59.99 pm on January 31, February 28, March 31, so on.). It also filters out minor edits and redirects.		From the source WikiMedia XML dump files, it extracts the latest revision for each article valid at the end of each month (namely at 11:59:59.99 pm on January 31, February 28, March 31, so on.). It also filters out minor edits and redirects.

	== How ~~to build~~ ==		== How To Build ==
	In Eclipse, configure a run		In Eclipse, configure a run
	# Right-click on 'mwdumper' in Package Explorer, select 'Run As.. --> Run Configurations'.		# Right-click on 'mwdumper' in Package Explorer, select 'Run As.. --> Run Configurations'.
Line 19:		Line 19:
	# Click 'Finish'		# Click 'Finish'

	== How ~~to run~~ ==		== How To Run ==
	In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located.		In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located.

Scsong at 22:04, 9 November 2009

2009-11-09T22:04:28Z

← Older revision		Revision as of 22:04, 9 November 2009
Line 2:		Line 2:
	From the source WikiMedia XML dump files, it extracts the latest revision for each article valid at the end of each month (namely at 11:59:59.99 pm on January 31, February 28, March 31, so on.). It also filters out minor edits and redirects.		From the source WikiMedia XML dump files, it extracts the latest revision for each article valid at the end of each month (namely at 11:59:59.99 pm on January 31, February 28, March 31, so on.). It also filters out minor edits and redirects.

	== ~~Usage~~ ==		== How to build ==
			In Eclipse, configure a run
			# Right-click on 'mwdumper' in Package Explorer, select 'Run As.. --> Run Configurations'.
			# On the left pane, right click on 'Java Application --> New'
			# Enter 'mwdumper' in the Name field on the right pane.
			# Select 'mwdumper' in the Project field.
			# Select 'org.mediawiki.dumper.LREMonthlyDumper20012007p' in the Main class field
			# Click 'Apply'
			# Click 'Close'

	In Eclipse, export mwdumper as a runnable JAR.		In Eclipse, export mwdumper as a runnable JAR.
	# Right-click on 'mwdumper' in Package Explorer, select 'export'.		# Right-click on 'mwdumper' in Package Explorer, select 'export'.
Line 10:		Line 19:
	# Click 'Finish'		# Click 'Finish'

			== How to run ==
	In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located.		In a shell terminal (or a command line prompt in Windows), change directory to where lremonthlydumper.jar is located.

Line 17:		Line 27:
	</pre>		</pre>

	== Input ==		== Input File ==
	''<WikiDump.lst>'': A file that lists the locations of WikiMedia XML dump files		''<WikiDump.lst>'': A file that lists the locations of WikiMedia XML dump files
	Example contents in a list file:		Example contents in a list file:
Line 33:		Line 43:


	== Output ==		== Output Files ==
	Under the 'extracted-monthly' directory, WikiMedia XML dump files (one for each month) are generated. The file month-''<k>''.xml corresponds to the extracted wiki articles for the ''k''th month.		Under the 'extracted-monthly' directory, WikiMedia XML dump files (one for each month) are generated. The file month-''<k>''.xml corresponds to the extracted wiki articles for the ''k''th month.

Line 39:		Line 49:
	The range of months is hard-coded as January 2001 ~ December 2007 in org.mediawiki.dumper.LREMonthlyDumper20012007, and so is the output directory. Change these as necessary, or even better, parameterize them.		The range of months is hard-coded as January 2001 ~ December 2007 in org.mediawiki.dumper.LREMonthlyDumper20012007, and so is the output directory. Change these as necessary, or even better, parameterize them.

	== Source ~~Code~~ ==		== Source Codes ==
	svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwdumper		svn co http://narasvn.umiacs.umd.edu/repository/src/webarc/mwdumper

Scsong at 21:44, 9 November 2009

2009-11-09T21:44:43Z

← Older revision		Revision as of 21:44, 9 November 2009
Line 34:		Line 34:

	== Output ==		== Output ==
	Under the 'extracted-monthly' directory, WikiMedia XML dump files (one for each month). month-''<k>''.xml corresponds to the ~~output file~~ for the ''k''th month.		Under the 'extracted-monthly' directory, WikiMedia XML dump files (one for each month) are generated. The file month-''<k>''.xml corresponds to the extracted wiki articles for the ''k''th month.

	== Notes ==		== Notes ==

Scsong at 21:43, 9 November 2009

2009-11-09T21:43:34Z

← Older revision		Revision as of 21:43, 9 November 2009
Line 14:		Line 14:
	<pre>		<pre>
	mkdir extracted-monthly		mkdir extracted-monthly
	java -jar lremonthlydumper.jar ''<WikiDump.lst>''		java -jar lremonthlydumper.jar <WikiDump.lst>
	</pre>		</pre>

Scsong at 21:43, 9 November 2009

2009-11-09T21:43:22Z

← Older revision		Revision as of 21:43, 9 November 2009
Line 14:		Line 14:
	<pre>		<pre>
	mkdir extracted-monthly		mkdir extracted-monthly
	java -jar lremonthlydumper.jar ''<WikiDump.lst''>		java -jar lremonthlydumper.jar ''<WikiDump.lst>''
	</pre>		</pre>

Scsong at 21:43, 9 November 2009

2009-11-09T21:43:01Z

← Older revision		Revision as of 21:43, 9 November 2009
Line 18:		Line 18:

	== Input ==		== Input ==
	''<WikiDump.lst''>: A file that lists the locations of WikiMedia XML dump files		''<WikiDump.lst>'': A file that lists the locations of WikiMedia XML dump files
	Example contents in a list file:		Example contents in a list file:
	<pre>		<pre>