WarcManager - Revision history

Shake at 19:07, 29 September 2016

2016-09-29T19:07:58Z

← Older revision		Revision as of 19:07, 29 September 2016
Line 3:		Line 3:


	* Source Code Repository: https://~~subversion~~.umiacs.umd.edu/warc-utils/		* Source Code Repository: https://gitlab.umiacs.umd.edu/adapt/warc-utils
	* Nightly builds: http://adaptci01.umiacs.umd.edu:8080/job/WARC%20Manager%202/		* <strike>Nightly builds: http://adaptci01.umiacs.umd.edu:8080/job/WARC%20Manager%202/</strike>

	=Searching=		=Searching=

Shake at 20:26, 5 December 2014

2014-12-05T20:26:41Z

← Older revision		Revision as of 20:26, 5 December 2014
Line 4:		Line 4:

	* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/		* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/
	* Nightly builds: http://~~adaptvm01~~.umiacs.umd.edu:8080~~/jenkins~~/job/~~Warc~~%20Manager%202/		* Nightly builds: http://adaptci01.umiacs.umd.edu:8080/job/WARC%20Manager%202/

	=Searching=		=Searching=

Toaster at 13:18, 21 June 2011

2011-06-21T13:18:16Z

← Older revision		Revision as of 13:18, 21 June 2011
Line 64:		Line 64:
	type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"		type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"
	validationQuery="SELECT 1"/>		validationQuery="SELECT 1"/>

			Add the following line after the resource line.
			<Parameter name="ingest.url" value="http://localhost:8080/warc/rest/ingest"/>

	4. Start Tomcat		4. Start Tomcat
	$ apache-tomcat-6.0.32/bin/startup.sh		$ apache-tomcat-6.0.32/bin/startup.sh

Toaster at 17:27, 27 April 2011

2011-04-27T17:27:22Z

← Older revision		Revision as of 17:27, 27 April 2011
Line 41:		Line 41:
	The installation consists of a few simple steps.		The installation consists of a few simple steps.

	Before you begin, you will need		Before you begin, you will need to download:
			* Apache Tomcat: http://tomcat.apache.org
			* MySQL jdbc connector: http://dev.mysql.com/downloads/connector/j/
			* context.xml, schema.sql, and the warc webapp from: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/

	1. Create database and setup permissions.		1. Create database and setup permissions.

Toaster at 14:59, 27 April 2011

2011-04-27T14:59:10Z

← Older revision		Revision as of 14:59, 27 April 2011
Line 33:		Line 33:
	The url details page shows detailed information for any URL in the collection. If there are multiple copies or different versions of a URL they will be listed in the left-hand column. Clicking on any entry will load details for that URL.		The url details page shows detailed information for any URL in the collection. If there are multiple copies or different versions of a URL they will be listed in the left-hand column. Clicking on any entry will load details for that URL.

	If you are accessing a URL that contains a html page (As opposed to an image) the warc manager will attempt to extract all links contained in that page and populate the table on the lower right. If the URL is not an html file (ie, image) the link table will remain empty.		If you are accessing a URL that contains a html page (As opposed to an image) the warc manager will attempt to extract all links contained in that page and populate the table on the lower right. links that appear in the archive are shown in black, missing links are red. You can double-click on any page in the table to view details for that url. If the URL is not an html file (ie, image) the link table will remain empty.

	Clicking on the 'Download File' link will allow you to download the selected page from the archive.		Clicking on the 'Download File' link will allow you to download the selected page from the archive.

Toaster at 14:57, 27 April 2011

2011-04-27T14:57:40Z

← Older revision		Revision as of 14:57, 27 April 2011
Line 19:		Line 19:
	* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.		* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.

			==Search Results==
			[[Image:searchresults.png\|thumb\|Search Results]]

			The search results show all matching URL's along with some statistics about that url. Double clicking on any row will load details for any url.

			* First Crawl - The earliest crawl date for the url.
			* Last Crawl - The most recent crawl date for the url.
			* Harvested - Lists the total number of times a page appears in the archive. This includes duplicates and revisits
			* Unique - Shows how many unique copies of the page exist in the archive.

			==URL Details==
			[[Image:urldetails.png\|thumb\|URL Details]]
			The url details page shows detailed information for any URL in the collection. If there are multiple copies or different versions of a URL they will be listed in the left-hand column. Clicking on any entry will load details for that URL.

			If you are accessing a URL that contains a html page (As opposed to an image) the warc manager will attempt to extract all links contained in that page and populate the table on the lower right. If the URL is not an html file (ie, image) the link table will remain empty.

			Clicking on the 'Download File' link will allow you to download the selected page from the archive.

	=Installation=		=Installation=

Toaster at 14:46, 27 April 2011

2011-04-27T14:46:17Z

← Older revision		Revision as of 14:46, 27 April 2011
Line 9:		Line 9:

	The warc manager offers two ways to search collections.		The warc manager offers two ways to search collections.

			[[Image:searchpage.png]]

	1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.		1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.

Toaster at 14:43, 27 April 2011

2011-04-27T14:43:52Z

← Older revision		Revision as of 14:43, 27 April 2011
Line 8:		Line 8:
	=Searching=		=Searching=

	The ~~main page~~ offers two ways to search ~~for~~ collections.		The warc manager offers two ways to search collections.

	1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.		1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.

Toaster at 14:43, 27 April 2011

2011-04-27T14:43:27Z

@@ Line 5: / Line 5: @@
 * Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/
 * Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/
 =Installation=
@@ Line 46: / Line 58: @@
 . Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.

Toaster at 14:42, 27 April 2011

2011-04-27T14:42:45Z

← Older revision		Revision as of 14:42, 27 April 2011
Line 53:		Line 53:
	1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.		1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.

	2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below:		2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:

	* <nowiki>*.html</nowiki> - search for all files ending in '.html'		* <nowiki>*.html</nowiki> - search for all files ending in '.html'
	* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com		* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com
	* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.		* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.