https://wiki.umiacs.umd.edu/adapt/api.php?action=feedcontributions&feedformat=atom&user=ToasterAdapt - User contributions [en]2026-04-29T18:08:21ZUser contributionsMediaWiki 1.43.7https://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Audit_Manager_Installation_Guide&diff=2906Ace:Audit Manager Installation Guide2014-11-24T14:16:48Z<p>Toaster: /* Files */</p>
<hr />
<div>The ACE Audit manager is a web portal to monitor the integrity of any collection you specify. The manager will register any new files found in addition to checking the integrity of files in a collection during an audit. <br />
<br />
==Requirements==<br />
Before you begin, you will need to have the following software installed:<br />
* Java 1.7 or later http://java.sun.com<br />
* Tomcat 6.0 or later http://tomcat.apache.org<br />
* MySQL 4.0 or later http://www.mysql.org<br />
* MySQL connector 5.0.7 or better http://www.mysql.com/products/connector/j/<br />
<br />
==Files==<br />
<br />
The following files are needed to install the Audit Manager <br />
* ace.sql : sql tables<br />
* ace-am.war: ace webapp<br />
* ace-am.xml: ace configuration file<br />
<br />
The latest versions of these can be downloaded from:<br />
https://gitlab.umiacs.umd.edu/adapt/ace/tree/svn/tags/ace-1.8<br />
<br />
==Quick Setup==<br />
<br />
This is for new installations only. To update, follow the directions [[Ace:Update Audit Manager|here]]<br />
<br />
;1. Create Database : Create a new database called 'aceam', grant permissions and setup table structure.<br />
<pre><br />
Welcome to the MySQL monitor. Commands end with ; or \g.<br />
Your MySQL connection id is 2 to server version: 4.1.20<br />
<br />
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.<br />
<br />
mysql> create database aceam;<br />
Query OK, 1 row affected (0.01 sec)<br />
<br />
mysql> grant all on aceam.* to 'aceam'@'localhost' identified by 'YOUR_PASSWORD';<br />
Query OK, 0 rows affected (0.00 sec)<br />
<br />
mysql> use aceam;<br />
Database changed<br />
mysql> source ace.sql;<br />
ERROR 1146 (42S02): Table 'aceam.ACTIVITY_LOG_ENTRY' doesn't exist<br />
ERROR 1146 (42S02): Table 'aceam.ACTIVITY_LOG_ENTRY' doesn't exist<br />
...<br />
...<br />
Query OK, 0 rows affected (0.00 sec)<br />
Records: 0 Duplicates: 0 Warnings: 0<br />
<br />
mysql> exit<br />
Bye<br />
</pre><br />
<br />
The ace.sql listed is the file from above. Set YOUR_PASSWORD to a password to use for this database. This password is only used to connect the Audit Manager to the database.<br />
<br />
;2. Install Webapp : Install tomcat on your server and verify it's working. Shutdown the server and place a copy of the ace-am.war file in your TOMCAT/webapps directory. Place a copy of the configuration file ace-am.xml in your TOMCAT/conf/Catalina/localhost directory. Open up the configuration file and edit the following setting.<br />
<br />
<pre><br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20"<br />
maxIdle="10" maxWait="-1" name="jdbc/aceamdb" <br />
password="ace" username="aceam" <br />
url="jdbc:mysql://localhost/aceam?characterEncoding=UTF-8" <br />
testOnBorrow="true" type="javax.sql.DataSource" validationQuery="SELECT 1"/><br />
</pre><br />
<br />
Change the password from YOUR_PASSWORD to whatever you used above.<br />
<br />
* Place a copy of the mysql-connector-java-5.0.XX-bin.jar in your TOMCAT/lib directory.<br />
<br />
; 3. Register your first collection : When the monitor starts up, you will see a status screen with no collections.<br />
<br />
* Click ''Add Collection'' from the status screen. <br />
**You should be prompted for a password. Enter 'admin' for both the username and password.<br />
* Enter a descriptive name, and the directory you want to monitor. Choose the type of storage where this directory is located. <br />
**Click ''Configure Storage'' when finished.<br />
* If you selected something other than ''local'' for your storage, a box will appear with additional setting you need to fill out.<br />
* Click ''Save'' when finished.<br />
<br />
; 4. Audit your collection : Click the ''Status'' link at the top of the page. Click on the name of the collection and then click the [[Image:file-audit-start.jpg]] icon in the details box. Your collection will now scan all files and register tokens. Please see the [[Audit Manager User Documentation|User Documentation]] for more details.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2893Main Page2012-07-11T14:37:32Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Over the past years, we've developed numerous tools and have active research in many areas.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Open Source Tools'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[Ace|Integrity Management (ACE)]]<li>[[WarcManager|Warc Management Utility]]<li>[https://scm.umiacs.umd.edu/redmine/adapt/projects/ingestion RESTful Bag server Implementation]</ul></td><br />
<td><ul><li>[[Webarc:Main|Web Archive Indexing]]<li>[[Replication:Main|Replication Monitoring]]<br />
<li>[[Chronopolis|Chronopolis]]</ul><br />
</td></tr></table><br />
<br />
<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
<br />
'''ACE V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* <strike>Hudson</strike>Jenkins testing and daily builds http://adaptci01.umiacs.umd.edu:8080/jenkins/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]<br />
<br />
=Old Projects=<br />
<ul><li>[[PAWN:Main|Ingestion (PAWN)]]<br />
<li>[[Focus:Main|Format Stewardship (FOCUS)]]<br />
<li>[[TPAP|TPAP]] - Transcontinental persistent archive prototype</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Devel_Pages&diff=2892Ace:Devel Pages2012-07-11T13:42:49Z<p>Toaster: </p>
<hr />
<div>=General=<br />
<br />
* [https://scm.umiacs.umd.edu/redmine/adapt/projects/ace Bug Tracker, Source code..]<br />
* [http://groups.google.com/group/ace-devel Discussion List]<br />
* [http://adapt-ci.umiacs.umd.edu:8080/hudson/job/ACE%20Audit%20Manager Hudson build tests]<br />
<br />
=Token Store Format=<br />
<br />
Information for creating, parsing and validating Token stores<br />
<br />
* [[Ace:TokenStore|Token Store Format]]<br />
* [[Ace:StorePython|Processing Token Stores in Python]]<br />
<br />
=Audit Manager=<br />
<br />
Developer pages and documentation for the ACE Audit Manager web application<br />
<br />
* [[Ace:JSON Interface|JSON Interface to Audit Manager]]<br />
* [[Ace:Roles|Security Roles]]<br />
* [[Ace:Summary Details|Summary generation]]<br />
* [[Ace:Expanding the Audit Manager|Adding Storage Types]] <br />
* [[Ace:Release Guidlines|Release Guidelines]]<br />
<br />
=Integrity Management Service=<br />
<br />
* [[Ace:IMSPython|Accessing the IMS using python]]<br />
<br />
=Other=<br />
<br />
* [[Ace:Deprecated|Deprecated Pages]]</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=People&diff=2891People2012-07-11T13:40:54Z<p>Toaster: </p>
<hr />
<div>'''Current Personal'''<br />
* [http://www.umiacs.umd.edu/~joseph Joseph JaJa], Principal Investigator<br />
* Mike Smorul, Lead Programmer and Manager of the Persistent Archives Project<br />
* Mike Ritter, Programmer<br />
<br />
<br />
<br />
'''Past Members'''<br />
* [http://www.ece.umd.edu/~scsong Sangchul Song], Graduate Research Assistant<br />
* Mike McGann, Programmer<br />
* Muluwork Geremew, Graduate Research Assistant<br />
* Chris Wamble, Student Programmer<br />
* Yang Wang, Programmer<br />
* Gary Jackson, Programmer<br />
* Anthony DeMartini, Student Programmer <br />
* Qingmin Shi, Postdoctoral Associate <br />
* Damon Earp, Student Programmer<br />
* Malcom Ray, Student Programmer</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=People&diff=2890People2012-07-11T13:40:47Z<p>Toaster: </p>
<hr />
<div>'''Current Personal'''<br />
* [http://www.umiacs.umd.edu/~joseph Joseph JaJa], Principal Investigator<br />
* Mike Smorul, Lead Programmer and Manager of the Persistent Archives Project<br />
* Mike Ritter<br />
<br />
<br />
<br />
'''Past Members'''<br />
* [http://www.ece.umd.edu/~scsong Sangchul Song], Graduate Research Assistant<br />
* Mike McGann, Programmer<br />
* Muluwork Geremew, Graduate Research Assistant<br />
* Chris Wamble, Student Programmer<br />
* Yang Wang, Programmer<br />
* Gary Jackson, Programmer<br />
* Anthony DeMartini, Student Programmer <br />
* Qingmin Shi, Postdoctoral Associate <br />
* Damon Earp, Student Programmer<br />
* Malcom Ray, Student Programmer</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2889Main Page2012-07-11T13:40:19Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Over the past years, we've developed numerous tools and have active research in many areas.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Open Source Tools'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[Ace|Integrity Management (ACE)]]<li>[[WarcManager|Warc Management Utility]]<li>[https://scm.umiacs.umd.edu/redmine/adapt/projects/ingestion RESTful Bag server Implementation]</ul></td><br />
<td><ul><li>[[Webarc:Main|Web Archive Indexing]]<li>[[Replication:Main|Replication Monitoring]]<br />
<li>[[Chronopolis|Chronopolis]]</ul><br />
</td></tr></table><br />
<br />
<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
<br />
'''ACE V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* <strike>Hudson</strike>Jenkins testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]<br />
<br />
=Old Projects=<br />
<ul><li>[[PAWN:Main|Ingestion (PAWN)]]<br />
<li>[[Focus:Main|Format Stewardship (FOCUS)]]<br />
<li>[[TPAP|TPAP]] - Transcontinental persistent archive prototype</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=EAP_decommission&diff=2888EAP decommission2012-02-15T05:24:29Z<p>Toaster: </p>
<hr />
<div>The technology demonstration displaying electronic content on loan from the National Archives and Records Administration has been retired due to hardware obsolescence. For copies of any content hosted in this collection, please contact NARA at inquire@nara.gov.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Main&diff=2887Ace:Main2011-11-07T22:15:22Z<p>Toaster: </p>
<hr />
<div>ACE (Auditing Control Environment) is a system that incorporates a new methodology to address the integrity of long term archives using rigorous cryptographic techniques. ACE continuously audits the contents of the various objects according to the policy set by the archive, and provides mechanisms for an independent third-party auditor to certify the integrity of any object.<br />
<br />
ACE consists of two components, the first an Audit Manager(AM) that checks files locally to ensure they have not been compromised. The second part, the Integrity Management Service (IMS), issues tokens that the AM can use to verify that its local store of file digests has not been tampered with. <br />
<br />
Anyone wishing to audit their files with ACE only needs to install an Audit Manager locally to manage their files. The ADAPT project runs a publically available IMS at ims.umiacs.umd.edu that any group may freely use.<br />
<br />
{| width="75%" style="margin-left: 75px;"<br />
| '''Audit Manager''' || '''Integrity Management Service'''<br />
|-<br />
| [[Ace:Audit Manager Installation Guide|Download Audit Manager]] || [[Ace:Ace IMS System|System Documentation]]<br />
|-<br />
| [[Ace:Audit Manager User Guide|User Guide]] || [http://ims.umiacs.umd.edu:8080/ace-ims/IMSWebService IMS Webservice]<br />
|-<br />
| [http://groups.google.com/group/ace-devel Discussion List] || [[Ace:Validating Witness and Tokens|Validating Tokens and Witnesses]]<br />
|-<br />
| [[Ace:Audit Manager Upgrading|Upgrade Audit Manager]] || [http://groups.google.com/group/ace-ims-witness?hl=en IMS Witness Google Goup]<br />
|- <br />
| [[Ace:AM Config Reference|AM Configuration Options]] || [http://mailman.umiacs.umd.edu/mailman/listinfo/ims-witness IMS Witness UMIACS listserv]<br />
|-<br />
| || [[Ace:IMS Installation| IMS Installation Guide]]<br />
|-<br />
| '''Resources'''<br />
|-<br />
| [https://scm.umiacs.umd.edu/redmine/adapt/projects/show/ace Files and Issue Tracker]<br />
|-<br />
| [[Ace:Devel Pages|Development Documentation]]<br />
|-<br />
| [[Ace:Webstart Client|Auditor Application]] || <br />
|-<br />
| [http://digitalcommons.uconn.edu/libr_pubs/39/ Step by Step installation instructions]<br />
|}<br />
<br />
<br />
'''Papers and Presentations'''<br />
<br />
* Smorul, M., Song, S., and JaJa, J. An Implementation of the Audit Control Environment(ACE) to Support the Long Term Integrity of Digital Archives. in DigCCurr 2009. 2009: University of North Carolina.[[media:DigCCurr2009_060909.pdf|pdf]] presentation [[media:Smorul-digccurr09.ppt|ppt]]<br />
* ACE Presentation at Archiving 2007 (5/07) [[media: ACE-Archiving2007.ppt|ppt]]<br />
* Song, S. and JaJa, J. ACE: A Novel Software Platform to Ensure the Integrity of Long Term Archives. in Archiving 2007. 2007: IS&T. [[media:rad71E67.pdf|pdf]]<br />
<br />
Development of the ACE Version1.0 release was partially sponsored by the US National Archives and Records Administration. The basic research on ACE was partially sponsored under a DIGARCH grant from NSF/Library of Congress.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Main&diff=2886Ace:Main2011-11-07T22:15:07Z<p>Toaster: </p>
<hr />
<div>ACE (Auditing Control Environment) is a system that incorporates a new methodology to address the integrity of long term archives using rigorous cryptographic techniques. ACE continuously audits the contents of the various objects according to the policy set by the archive, and provides mechanisms for an independent third-party auditor to certify the integrity of any object.<br />
<br />
ACE consists of two components, the first an Audit Manager(AM) that checks files locally to ensure they have not been compromised. The second part, the Integrity Management Service (IMS), issues tokens that the AM can use to verify that its local store of file digests has not been tampered with. <br />
<br />
Anyone wishing to audit their files with ACE only needs to install an Audit Manager locally to manage their files. The ADAPT project runs a publically available IMS at ims.umiacs.umd.edu that any group may freely use.<br />
<br />
{| width="75%" style="margin-left: 75px;"<br />
| '''Audit Manager''' || '''Integrity Management Service'''<br />
|-<br />
| [[Ace:Audit Manager Installation Guide|Download Audit Manager]] || [[Ace:Ace IMS System|System Documentation]]<br />
|-<br />
| [[Ace:Audit Manager User Guide|User Guide]] || [http://ims.umiacs.umd.edu:8080/ace-ims/IMSWebService IMS Webservice]<br />
|-<br />
| [http://groups.google.com/group/ace-devel Discussion List] || [[Ace:Validating Witness and Tokens|Validating Tokens and Witnesses]]<br />
|-<br />
| [[Ace:Audit Manager Upgrading|Upgrade Audit Manager]] || [http://groups.google.com/group/ace-ims-witness?hl=en IMS Witness Google Goup]<br />
|- <br />
| [[Ace:AM Config Reference|AM Configuration Options]] || [http://mailman.umiacs.umd.edu/mailman/listinfo/ims-witness IMS Witness UMIACS listserv]<br />
|-<br />
| || [[Ace:IMS Installation| IMS Installation Guide]]<br />
|-<br />
| '''Resources'''<br />
|-<br />
| [https://scm.umiacs.umd.edu/redmine/adapt/projects/show/ace Files and Issue Tracker]<br />
|-<br />
| [[Ace:Devel Pages|Development Documentation]]<br />
|-<br />
| [[Ace:Webstart Client|Auditor Application]] || <br />
|-<br />
| [[http://digitalcommons.uconn.edu/libr_pubs/39/ Step by Step installation instructions<br />
|}<br />
<br />
<br />
'''Papers and Presentations'''<br />
<br />
* Smorul, M., Song, S., and JaJa, J. An Implementation of the Audit Control Environment(ACE) to Support the Long Term Integrity of Digital Archives. in DigCCurr 2009. 2009: University of North Carolina.[[media:DigCCurr2009_060909.pdf|pdf]] presentation [[media:Smorul-digccurr09.ppt|ppt]]<br />
* ACE Presentation at Archiving 2007 (5/07) [[media: ACE-Archiving2007.ppt|ppt]]<br />
* Song, S. and JaJa, J. ACE: A Novel Software Platform to Ensure the Integrity of Long Term Archives. in Archiving 2007. 2007: IS&T. [[media:rad71E67.pdf|pdf]]<br />
<br />
Development of the ACE Version1.0 release was partially sponsored by the US National Archives and Records Administration. The basic research on ACE was partially sponsored under a DIGARCH grant from NSF/Library of Congress.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2885Main Page2011-08-17T14:23:03Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Over the past years, we've developed numerous tools and have active research in many areas.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Open Source Tools'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[Ace|Integrity Management (ACE)]]<li>[[WarcManager|Warc Management Utility]]<li>[https://scm.umiacs.umd.edu/redmine/adapt/projects/ingestion RESTful Bag server Implementation]</ul></td><br />
<td><ul><li>[[Webarc:Main|Web Archive Indexing]]<li>[[Replication:Main|Replication Monitoring]]<br />
<li>[[Chronopolis|Chronopolis]]</ul><br />
</td></tr></table><br />
<br />
'''Java Programmer Job Opening'''<br />
* ADAPT is currently hiring a full time java programmer. Please see our [[Employment]] page for details.<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
<br />
'''ACE V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* <strike>Hudson</strike>Jenkins testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]<br />
<br />
=Old Projects=<br />
<ul><li>[[PAWN:Main|Ingestion (PAWN)]]<br />
<li>[[Focus:Main|Format Stewardship (FOCUS)]]<br />
<li>[[TPAP|TPAP]] - Transcontinental persistent archive prototype</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Employment&diff=2884Employment2011-08-17T14:21:19Z<p>Toaster: </p>
<hr />
<div>=Full Time Position=<br />
<br />
'''Summary'''<br />
<br />
The UMIACS ADAPT Project is looking for a Java programmer to assist in<br />
a number of open source digital preservation efforts. This programmer<br />
will initially work in a small group to help develop and extend<br />
several Tomcat/MySQL based web applications. Due to the small size of<br />
the research group, applicants must be able to work independently and<br />
will be expected to set their own schedule and identify deliverables<br />
within the context of current projects. Applicants will work with<br />
external collaborators to ensure the software meets the needs of the<br />
broader digital preservation community. Minimal travel of one to two<br />
times a year will be required.<br />
<br />
'''Daily Activity'''<br />
<br />
The developer's will focus on building a new web application in collaboration with the Lab's researchers and faculty. In any given week the developer will be expected to spend:<br />
* at least 20 hours of software development and testing.<br />
* at least 5 hours working with outside collaborators to identify necessary additions and identify software bugs.<br />
* at least 3 hours a week attending conference calls and group meetings.<br />
<br />
'''Required Skills'''<br />
* Experienced in Java 1.6+<br />
* Bachelors degree in Computer Science, Computer Engineering or equivalent work experience.<br />
* Familiar with java persistence or similar ORM technologies<br />
* Experienced in developing database backed Servlet or RESTful web services.<br />
* Experienced developing and deploying applications on Tomcat server installations running on linux.<br />
<br />
Excellent candidates should have the additional experience.<br />
* Familiar with developing command-line python utilities.<br />
* Experienced in developing applications in a CI environment<br />
* Minimal Systems Administration experience in a Linux environment.<br />
This position is a demanding professional position that will expose applicants to a wide variety of technologies. Applicants must show a desire to continually explore new technologies and work with others to identify new challenges in digital preservation.<br />
<br />
'''About ADAPT'''<br />
<br />
The adapt project is a research group housed within the University of<br />
Maryland Institute for Advanced Computer Studies. The group has worked<br />
with numerous partners including the Library of Congress and U.S.<br />
National Archives to develop several technologies to assist in long<br />
term digital preservation at scale. For more information and a list of<br />
projects currently supported by the ADAPT project, please visit<br />
http://adapt.umiacs.umd.edu<br />
<br />
'''Applying'''<br />
<br />
If you are interested, please contact Mike Smorul directly by emailing<br />
the following information to msmorul@umiacs.umd.edu<br />
* Your CV<br />
* Three references.<br />
* A portfolio which includes either pointers to web applications you contributed or descriptions of applications you assisted in developing and your role during development<br />
* Your salary requirements<br />
<br />
<br />
=Undergraduate Positions=<br />
<br />
The ADAPT Project has the following undergraduate positions available at this time.<br />
<br />
===Web Designer/Programmer===<br />
'''Description'''<br />
<br />
The ADAPT Project is currently developing several web-based applications which support digital preservation. These applications consist of two parts, a server component which is controlled through a RESTful API, and a web-based client which controls and displays information from the service using JSON.<br />
<br />
The student will initially work to create an attractive javascript-bsed web interface for these services. Depending on skill and motivation, there are a variety of other programming opportunities available in the lab.<br />
<br />
Due to the amount of development involved, interested students should expect a time commitment of over 10hrs/week. Starting pay is $10.25/hr with semester-end reviews.<br />
<br />
'''Requirements'''<br />
<br />
Candidates must have experience in designing web page using javascript. Ideally, candidates will also have java programming and experience in working with REST/JSON based web technologies.<br />
<br />
=Applying=<br />
Please send a copy of your resume/CV to msmorul at umiacs.umd.edu</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=MediaWiki:Sidebar&diff=2883MediaWiki:Sidebar2011-07-20T18:13:00Z<p>Toaster: </p>
<hr />
<div>* General<br />
** Adapt:About|Overview<br />
** Projects|Projects<br />
** papers|Papers<br />
** people|People<br />
** lab|Lab Resources<br />
** https://scm.umiacs.umd.edu/redmine/adapt/|Redmine<br />
*Projects<br />
** Ace:Main|Integrity Auditing (ACE)<br />
** WarcManager|Warc Management Utility<br />
** https://scm.umiacs.umd.edu/redmine/adapt/projects/ingestion|RESTful Bag server Implementation<br />
* Research<br />
** PAWN:Main|Ingestion (PAWN)<br />
** Replication:Main|Replication<br />
** Webarc:Main|Web Archive Searching<br />
** all_projects|All Research</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=MediaWiki:Sidebar&diff=2882MediaWiki:Sidebar2011-07-20T18:12:16Z<p>Toaster: </p>
<hr />
<div>* General<br />
** Adapt:About|Overview<br />
** Projects|Projects<br />
** papers|Papers<br />
** people|People<br />
** lab|Lab Resources<br />
** https://scm.umiacs.umd.edu/redmine/adapt/|Redmine<br />
*Projects<br />
** Ace:Main|Integrity Auditing (ACE)<br />
** WarcManager|Warc Management Utility<br />
** https://scm.umiacs.umd.edu/redmine/adapt/projects/ingestion|RESTful Bag server Implementation<br />
* Research<br />
** PAWN:Main|Ingestion (PAWN)<br />
** Replication:Main|Replication<br />
** Webarc:Main|Web Archive Searching<br />
** all_projects|All Research<br />
* Demonstrations<br />
** Ace:Webstart_Client|ACE Client<br />
** http://narademo.umiacs.umd.edu/cgi-bin/isadg/main.html|EAP Browser<br />
** Pawn:Public_Demonstration|PAWN</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=MediaWiki:Sidebar&diff=2881MediaWiki:Sidebar2011-07-20T18:11:35Z<p>Toaster: </p>
<hr />
<div>* General<br />
** Adapt:About|Overview<br />
** Projects|Projects<br />
** papers|Papers<br />
** people|People<br />
** lab|Lab Resources<br />
** https://scm.umiacs.umd.edu/redmine/adapt/|Redmine<br />
*Projects<br />
** Ace:Main|Integrity Auditing (ACE)<br />
** WarcManager|Warc Management Utility<br />
** https://scm.umiacs.umd.edu/redmine/adapt/projects/ingestion RESTful Bag server Implementation<br />
* Research<br />
** PAWN:Main|Ingestion (PAWN)<br />
** Replication:Main|Replication<br />
** Webarc:Main|Web Archive Searching<br />
** all_projects|All Research<br />
* Demonstrations<br />
** Ace:Webstart_Client|ACE Client<br />
** http://narademo.umiacs.umd.edu/cgi-bin/isadg/main.html|EAP Browser<br />
** Pawn:Public_Demonstration|PAWN</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2880Main Page2011-07-20T18:07:45Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Over the past years, we've developed numerous tools and have active research in many areas.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Open Source Tools'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[Ace|Integrity Management (ACE)]]<li>[[WarcManager|Warc Management Utility]]<li>[https://scm.umiacs.umd.edu/redmine/adapt/projects/ingestion RESTful Bag server Implementation]</ul></td><br />
<td><ul><li>[[Webarc:Main|Web Archive Indexing]]<li>[[Replication:Main|Replication Monitoring]]<br />
<li>[[Chronopolis|Chronopolis]]</ul><br />
</td></tr></table><br />
'''AEC V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* <strike>Hudson</strike>Jenkins testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]<br />
<br />
=Old Projects=<br />
<ul><li>[[PAWN:Main|Ingestion (PAWN)]]<br />
<li>[[Focus:Main|Format Stewardship (FOCUS)]]<br />
<li>[[TPAP|TPAP]] - Transcontinental persistent archive prototype</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2879Main Page2011-07-18T13:55:00Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Our approach uses a distributed object architecture that operates on different levels of abstractions built around grid technologies and web services. Major software components have been prototyped and are currently in use in a number of pilot projects such as the Transcontinental Persistent Archives Project and Chronopolis.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Projects and Tools'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[Chronopolis|Chronopolis]]<li>[[Ace|Integrity Management (ACE)]]<li>[[WarcManager|Warc Management Utility]]<li>[https://scm.umiacs.umd.edu/redmine/adapt/projects/ingestion RESTful Bag server Implementation]</ul></td><br />
<td><ul><li>[[Webarc:Main|Web Archive Indexing]]<li>[[Replication:Main|Replication Monitoring]]</ul><br />
</td></tr></table><br />
'''AEC V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* <strike>Hudson</strike>Jenkins testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]<br />
<br />
=Old Projects=<br />
<ul><li>[[PAWN:Main|Ingestion (PAWN)]]<br />
<li>[[Focus:Main|Format Stewardship (FOCUS)]]<br />
<li>[[TPAP|TPAP]] - Transcontinental persistent archive prototype</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2878WarcManager2011-06-21T13:18:16Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Searching=<br />
<br />
The warc manager offers two ways to search collections.<br />
<br />
[[Image:searchpage.png]]<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.<br />
<br />
==Search Results==<br />
[[Image:searchresults.png|thumb|Search Results]]<br />
<br />
The search results show all matching URL's along with some statistics about that url. Double clicking on any row will load details for any url.<br />
<br />
* First Crawl - The earliest crawl date for the url.<br />
* Last Crawl - The most recent crawl date for the url.<br />
* Harvested - Lists the total number of times a page appears in the archive. This includes duplicates and revisits<br />
* Unique - Shows how many unique copies of the page exist in the archive. <br />
<br />
==URL Details==<br />
[[Image:urldetails.png|thumb|URL Details]]<br />
The url details page shows detailed information for any URL in the collection. If there are multiple copies or different versions of a URL they will be listed in the left-hand column. Clicking on any entry will load details for that URL.<br />
<br />
If you are accessing a URL that contains a html page (As opposed to an image) the warc manager will attempt to extract all links contained in that page and populate the table on the lower right. links that appear in the archive are shown in black, missing links are red. You can double-click on any page in the table to view details for that url. If the URL is not an html file (ie, image) the link table will remain empty. <br />
<br />
Clicking on the 'Download File' link will allow you to download the selected page from the archive.<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need to download:<br />
* Apache Tomcat: http://tomcat.apache.org<br />
* MySQL jdbc connector: http://dev.mysql.com/downloads/connector/j/<br />
* context.xml, schema.sql, and the warc webapp from: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
<br />
Add the following line after the resource line.<br />
<Parameter name="ingest.url" value="http://localhost:8080/warc/rest/ingest"/><br />
<br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=File:Archiving11-smorul.pdf&diff=2877File:Archiving11-smorul.pdf2011-06-03T19:12:01Z<p>Toaster: </p>
<hr />
<div></div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Papers&diff=2876Papers2011-06-03T19:11:31Z<p>Toaster: </p>
<hr />
<div>==Papers==<br />
===2011===<br />
* M. Smorul and J. JaJa. Implementation of a high performance architecture for managing and storing web-harvested collections. in Archiving 2011. 2011:IS&T [[media:Archiving11-smorul.pdf|pdf]]<br />
===2010===<br />
* S. Song and J. JaJa. Effective Strategies for Temporally Anchored Information Retrieval, :UMIACS-TR-2010-05. 2010, University of Maryland Institute for Advanced Computer Studies. [http://www.lib.umd.edu/drum/bitstream/1903/10108/1/UMIACS-TR-2010-05.pdf pdf]<br />
* S. Song and J. JaJa. Techniques to Audit and Certify the Long Term Integrity of Digital Archives, International Journal of Digital Libraries, 2010. [http://www.umiacs.umd.edu/~joseph/IJDL_published.pdf pdf]<br />
===2009===<br />
* J. JaJa and S. Song, Robust Tools and Services for Long-Term Preservation of Digital Information, Special Issue: Library of Congress National Digital Information and Preservation Parntership, 57:3, 580-594, 2009. [http://www.umiacs.umd.edu/~joseph/library-trends.pdf pdf]<br />
* Song, S. and JaJa, J. Search and Access Strategies for Web Archives. in Archiving 2009. 2009: IS&T. [[media:Archiving2009_submitted.pdf|pdf]]<br />
* Smorul, M. and JaJa, J. A Case Study in Distributed Collection Monitoring and Auditing Using the Audit Control Environment (ACE). in Archiving 2009. 2009: IS&T.<br />
* Smorul, M., Song, S., and JaJa, J. An Implementation of the Audit Control Environment(ACE) to Support the Long Term Integrity of Digital Archives. in DigCCurr 2009. 2009: University of North Carolina.[[media:DigCCurr2009_060909.pdf|pdf]]<br />
* JaJa, J., Smorul, M., and Song, S. Tools and Services for Long-Term Preservation of Digital Archives. in Indo-US Workshop on International Trends in Digital Preservation. 2009. Pune, India [[media:Indo-US-Workshop-jaja.pdf|pdf]]<br />
<br />
===2008===<br />
* Song, S. and JaJa, J. Fast Browsing of Archived Web Contents. in 8th International Web Archiving Workshop. 2008. Aarhus, Denmark. [[media:iwaw08_Song_JaJa_Final.pdf|pdf]]<br />
* Song, S. and JaJa, J., Archiving Temporal Web Information: Organization of Web Contents for Fast Access and Compact Storage:UMIACS-TR-2008-08. 2008, University of Maryland Institute for Advanced Computer Studies. [[media:temporal-web-archiving-final-umiacs-tr-2008-08.pdf|pdf]]<br />
<br />
===2007===<br />
* Smorul, M., McGann, M., and JaJa, J. PAWN: A Policy-Driven Environment for Implementing Producer-Archive Interactions in Support of Long Term Digital Preservation. in Archiving 2007. 2007: IS&T. [[media:Pawn-archiving07-uploaded.pdf|pdf]]<br />
* Song, S. and JaJa, J. ACE: A Novel Software Platform to Ensure the Integrity of Long Term Archives. in Archiving 2007. 2007: IS&T. [[media:rad71E67.pdf|pdf]]<br />
* Smorul, M., McGann, M., and JaJa, J. The Use of the Producer-Archive Workflow Network (PAWN) in Support of Customized Archival Practice. in DigCCurr 2007. 2007. Chapel Hill, North Carolina. [[media:Digccurr07-paper-final.pdf|pdf]]<br />
* Song, S. and JaJa, J., Web Archiving: Organizing Web Objects into Web Containers to Optimize Access:UMIACS-TR-2007-42. 2007, University of Maryland Institute for Advanced Computer Studies. [[media:OrganizingWebObjects_UMIACS-TR-2007_42.pdf|pdf]]<br />
<br />
===2006===<br />
* Geremew, M., Song, S., and JaJa, J. Using Scalable and Secure Web Technologies to Design a Global Format Registry Prototype: Architecture, Implementation, and Testing. in Proceedings of Archiving 2006. 2006. Ottawa, Canada: IS&T. [[media:archiving06_final.pdf|pdf]]<br />
<br />
===2005===<br />
* JaJa, J., Smorul, M., McCall, F., and Wang, Y. Scalable, Reliable Marshalling and Organization of Distributed Large Scale Data Onto Enterprise Storage Environments. in The 22nd IEEE / 13th NASA Goddard Conference on Mass Storage Systems and Technologies. 2005. Monterey, California: IEEE Computer Society. [[media:msst-final5.pdf|pdf]]<br />
<br />
===2004===<br />
* JaJa, J., McCall, F., Smorul, M., Moore, Y., and Chadduck, R. Digital Archiving and Long Term Preservation: An Early Experience with Grid and Digital Library Technologies. [[media:thic-04.doc|doc]]<br />
* Smorul, M., JaJa, J., Wang, Y., and McCall, F., PAWN: Producer – Archive Workflow Network in Support of Digital Preservation:UMIACS-TR-2004-49. 2004. [[media:UMIACS-TR-2004-49.pdf|pdf]]<br />
<br />
===2003===<br />
* Smorul, M., JaJa, J., McCall, F., Brown, S.F., Moore, R., Marciano, R., Chen, S.-Y., Lopez, R., and Chadduck, R., Recovery of a Digital Image Collection Through the SDSC/UMD/NARA Prototype Persistent Archive:UMIACS-TR-2003-105. 2003. [[media:UMIACS-TR-2003-105.pdf|pdf]]<br />
<br />
==Presentations==<br />
===2010===<br />
* Smorul, M. ACE DPIF presentation at NIST, Mar 10[[media:ACE-NIST10.ppt|ppt]]<br />
===2009===<br />
* Smorul, M. ACE Presentation for GeoMapp yearly meeting August 09 [[media:Geomapp-ace.ppt|ppt]]<br />
* Smorul, M. and JaJa, J. Tools and Services for the Long Term Preservation and Access of Digital Archives, Sun Pasig, GW University [[media:Gw-pasig-09.ppt|ppt]]<br />
* Smorul, M. ACE Experiences, DigCcurr 2009 Chapel Hill NC [[media:Smorul-digccurr09.ppt|ppt]]<br />
* Song, S. and JaJa, J. Search and Access Strategies for Web Archives. in Archiving 2009. 2009: IS&T. [[media:webarc-slides-archiving-09-v2.ppt|ppt]]<br />
* JaJa, J., Smorul, M., and Song, S. Tools and Services for Long-Term Preservation of Digital Archives. in Indo-US Workshop on International Trends in Digital Preservation. 2009. Pune, India [[media:JaJa-Indo-US-Workshop-March2009.ppt|ppt]]<br />
<br />
===2008===<br />
* Sun Pasig Baltimore, ADAPT Tools overview [[media:SUN-PASIG-Nov08.ppt|ppt]]<br />
* ERA Research Project: Ingestion and Preservation Tools and Services (Partnerships in Innovation 2008) [[Media:JAJA-NARA-UMD-Conference-08.ppt|ppt]]<br />
* Fast Browsing of Archived Web Contents: International Web Archiving Workshop 2008 at Aarhus, Denmark (August 18-19, 2008) [[Media:IWAW2008.pdf|pdf]]<br />
* ACE: NDIIPP Partners Meeting at Arlington, VA (July 8-10, 2008) [[media:session4_chronopolis.ppt|ppt]]<br />
* NARA Presenation on recent research (6/08)<br />
** Overview [[media:NARA-June08-v2.ppt|ppt]]<br />
** PAWN [[media:pawn-0.7-nara-jun11.ppt|ppt]]<br />
** ACE [[media:ACEAuditingControlEnvironment.ppt|ppt]]<br />
** Replication Monitor [[media:ReplicationMonitoring.ppt|ppt]]<br />
* Technical Overview of Processes [[media: NewPawnProcess.ppt|ppt]]<br />
* Midwest Archives Conference (4/08) [[media: 2008MAC-Smorul.ppt|ppt]]<br />
<br />
===2007===<br />
* PAWN, NAGARA Kansas City Meeting (7/07) [[media: PAWN-nagara-2-smorul.ppt|ppt]]<br />
* ACE Presentation: NDIIPP and NARA review (7/07) [[media: ACE-July26.ppt|ppt]]<br />
* July project review, PAWN update (7/07) [[media: PAWN-updates-07-24.ppt|ppt]]<br />
* PAWN Presentation at Archiving 2007 (5/07) [[media: PAWN-Archiving07-v1.ppt|ppt]]<br />
* ACE Presentation at Archiving 2007 (5/07) [[media: ACE-Archiving2007.ppt|ppt]]<br />
* Long Term Sustainment of Digital Information for Science and Engineering, NIST Interoperability Week(04/07) [[media: ADAPT-NIST-04-07.ppt|ppt]]<br />
* DigCCurr 2007 (4/07) [[media: digccurr07-v1.ppt|ppt]]<br />
* NDIIPP Focus Breakout (1/07) [[media: focus-ndiipp-07.ppt|ppt]]<br />
* NDIIPP PAWN Breakout (1/07) [[media: PAWN-ndiipp-jan07.ppt|ppt]]<br />
* NDIIPP Pecha Kucha Session (1/07) [[media: ndiipp-summary-jan-07.ppt|ppt]]<br />
<br />
===2006===<br />
* Presentation at Archives II (2006-07-28) [[media: pawn-nara-20060728.ppt|ppt]]<br />
* Library of Congress (7/06) [[media: loc-july-06.ppt|ppt]]<br />
* July project review, PAWN update (7/06) [[media: PAWNprogress-Jul-06-2.ppt|ppt]]<br />
* Using Scalable and Secure Web Technologies to Design Global Format Registry (Archiving 2006 - 5/06) [[media: Conference_final.ppt|ppt]]<br />
* Robust Technologies for Automated Ingestion and Long-Term Preservation of Digital Information (5/06) [[media: dgo2006-jaja.ppt|ppt]]<br />
<br />
===2005===<br />
* Grid brick overview (11/05) [[media: GRIDBrickStatus-jun2-05.ppt|ppt]]<br />
* FOCUS overview (10/05) [[media: NARASlides_new.ppt|ppt]]<br />
* Pawn Release III status (10/05) [[media: PAWNTestIII-c.ppt|ppt]]<br />
* June project review, PAWN update (6/05) [[media: Review-Jun2-05.ppt|ppt]]<br />
* March NGDA data supplier meeting, ADAPT/PAWN/GRASP/GLCF (3/05) [[media: NGDA-adapt-glcf-05Mar.ppt|ppt]]<br />
* Esip Federation Spring 05<br />
** Introduction (Mike Smorul / Gary Jackson) [[media: 0.Introduction.ppt|ppt]]<br />
** SRB Overview (Mike Smorul) [[media: 1.srb-overview.ppt|ppt]]<br />
** Globus Overview (Gary Jackson) [[media: 2.globus.ppt|ppt]]<br />
** GRASP (Mike Smorul) [[media: 3.grasp.ppt|ppt]]<br />
** LPE (Gary Jackson) [[media: 4.LPE.ppt|ppt]]<br />
** PAWN for Scientific Collections (Mike Smorul) [[media: 5.pawn-mss.ppt|ppt]]<br />
** Globus vs SRB (Mike Smorul) [[media: globusvssrb.ppt|ppt]]<br />
* PAWN, LPE, GRASP overviews for JAXA/GLCF meeting (2/05) [[media: PAWN_LPE_grasp-05feb18.ppt|ppt]]<br />
<br />
===2004===<br />
* Dec 04 Presentation to OAIS members [[media: oais-pawn.ppt|ppt]]<br />
* PAWN: A Novel Ingestion Workflow Technology for Digital Preservation ([http://www.erpanet.org/events/2004/budapest/index.php ERPANET Workshop on Workflow]) [[media: erpanet.ppt|ppt]]<br />
<br />
==Posters==<br />
<br />
===2009===<br />
* Archiving2009 Web Archiving poster 5/09 [[media:webarc-search-poster-archiving-09-v3.ppt|ppt]]<br />
* Archiving2009 ACE poster 5/09<br />
<br />
===2008===<br />
<br />
===2007===<br />
* NDIIPP PAWN poster 1/07 [[media: globusvssrb.ppt|ppt]]<br />
* NDIIPP ACE poster 1/07 [[media: globusvssrb.ppt|ppt]]<br />
<br />
===2006===<br />
* NDIIPP Focus poster [[media: FOCUS_Poster-3.ppt|ppt]]<br />
* NDIIPP Adapt poster [[media: ndiipp-poster-2.ppt|ppt]]<br />
<br />
===2005===<br />
* PAWN and ADAPT poster 9/05 [[media: adapt-pawn4-final.ppt|ppt]]<br />
* Student jobs brochure 9/05 [[media: adapt-student-brochure.ppt|ppt]]<br />
<br />
===2004===<br />
* PAWN Poster for Nov 04 NARA Conference 11/04 [[media: pawn-poster.ppt|ppt]]</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2875WarcManager2011-04-27T17:27:22Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Searching=<br />
<br />
The warc manager offers two ways to search collections.<br />
<br />
[[Image:searchpage.png]]<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.<br />
<br />
==Search Results==<br />
[[Image:searchresults.png|thumb|Search Results]]<br />
<br />
The search results show all matching URL's along with some statistics about that url. Double clicking on any row will load details for any url.<br />
<br />
* First Crawl - The earliest crawl date for the url.<br />
* Last Crawl - The most recent crawl date for the url.<br />
* Harvested - Lists the total number of times a page appears in the archive. This includes duplicates and revisits<br />
* Unique - Shows how many unique copies of the page exist in the archive. <br />
<br />
==URL Details==<br />
[[Image:urldetails.png|thumb|URL Details]]<br />
The url details page shows detailed information for any URL in the collection. If there are multiple copies or different versions of a URL they will be listed in the left-hand column. Clicking on any entry will load details for that URL.<br />
<br />
If you are accessing a URL that contains a html page (As opposed to an image) the warc manager will attempt to extract all links contained in that page and populate the table on the lower right. links that appear in the archive are shown in black, missing links are red. You can double-click on any page in the table to view details for that url. If the URL is not an html file (ie, image) the link table will remain empty. <br />
<br />
Clicking on the 'Download File' link will allow you to download the selected page from the archive.<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need to download:<br />
* Apache Tomcat: http://tomcat.apache.org<br />
* MySQL jdbc connector: http://dev.mysql.com/downloads/connector/j/<br />
* context.xml, schema.sql, and the warc webapp from: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2874WarcManager2011-04-27T14:59:10Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Searching=<br />
<br />
The warc manager offers two ways to search collections.<br />
<br />
[[Image:searchpage.png]]<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.<br />
<br />
==Search Results==<br />
[[Image:searchresults.png|thumb|Search Results]]<br />
<br />
The search results show all matching URL's along with some statistics about that url. Double clicking on any row will load details for any url.<br />
<br />
* First Crawl - The earliest crawl date for the url.<br />
* Last Crawl - The most recent crawl date for the url.<br />
* Harvested - Lists the total number of times a page appears in the archive. This includes duplicates and revisits<br />
* Unique - Shows how many unique copies of the page exist in the archive. <br />
<br />
==URL Details==<br />
[[Image:urldetails.png|thumb|URL Details]]<br />
The url details page shows detailed information for any URL in the collection. If there are multiple copies or different versions of a URL they will be listed in the left-hand column. Clicking on any entry will load details for that URL.<br />
<br />
If you are accessing a URL that contains a html page (As opposed to an image) the warc manager will attempt to extract all links contained in that page and populate the table on the lower right. links that appear in the archive are shown in black, missing links are red. You can double-click on any page in the table to view details for that url. If the URL is not an html file (ie, image) the link table will remain empty. <br />
<br />
Clicking on the 'Download File' link will allow you to download the selected page from the archive.<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=File:Urldetails.png&diff=2873File:Urldetails.png2011-04-27T14:57:54Z<p>Toaster: </p>
<hr />
<div></div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2872WarcManager2011-04-27T14:57:40Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Searching=<br />
<br />
The warc manager offers two ways to search collections.<br />
<br />
[[Image:searchpage.png]]<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.<br />
<br />
==Search Results==<br />
[[Image:searchresults.png|thumb|Search Results]]<br />
<br />
The search results show all matching URL's along with some statistics about that url. Double clicking on any row will load details for any url.<br />
<br />
* First Crawl - The earliest crawl date for the url.<br />
* Last Crawl - The most recent crawl date for the url.<br />
* Harvested - Lists the total number of times a page appears in the archive. This includes duplicates and revisits<br />
* Unique - Shows how many unique copies of the page exist in the archive. <br />
<br />
==URL Details==<br />
[[Image:urldetails.png|thumb|URL Details]]<br />
The url details page shows detailed information for any URL in the collection. If there are multiple copies or different versions of a URL they will be listed in the left-hand column. Clicking on any entry will load details for that URL.<br />
<br />
If you are accessing a URL that contains a html page (As opposed to an image) the warc manager will attempt to extract all links contained in that page and populate the table on the lower right. If the URL is not an html file (ie, image) the link table will remain empty. <br />
<br />
Clicking on the 'Download File' link will allow you to download the selected page from the archive.<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=File:Searchresults.png&diff=2871File:Searchresults.png2011-04-27T14:47:29Z<p>Toaster: </p>
<hr />
<div></div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2870WarcManager2011-04-27T14:46:17Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Searching=<br />
<br />
The warc manager offers two ways to search collections.<br />
<br />
[[Image:searchpage.png]]<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.<br />
<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=File:Searchpage.png&diff=2869File:Searchpage.png2011-04-27T14:45:25Z<p>Toaster: </p>
<hr />
<div></div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2868WarcManager2011-04-27T14:43:52Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Searching=<br />
<br />
The warc manager offers two ways to search collections.<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.<br />
<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2867WarcManager2011-04-27T14:43:27Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Searching=<br />
<br />
The main page offers two ways to search for collections.<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.<br />
<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2866WarcManager2011-04-27T14:42:45Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.<br />
<br />
=Searching=<br />
<br />
The main page offers two ways to search for collections.<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below</nowiki>:<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2865WarcManager2011-04-27T14:42:13Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.<br />
<br />
=Searching=<br />
<br />
The main page offers two ways to search for collections.<br />
<br />
1. If you know the page you want to view, start typing the full URL into the search box starting with 'http'. You will see a drop-down box containing URL's that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. Optionally you can add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. Some common searches are listed below:<br />
<br />
* <nowiki>*.html</nowiki> - search for all files ending in '.html'<br />
* <nowiki>http://www.somesite.com*</nowiki> - View all URL's for www.somesite.com<br />
* <nowiki>http://www.somesite.com/*.html</nowiki> - view all '.html' files on somesite.com.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2864WarcManager2011-04-27T14:37:17Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
[[Image:ingestfiles.png]]<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.<br />
<br />
=Searching=<br />
<br />
The main page offers two ways to search for collections.<br />
<br />
1. If you know the page you want to view, start typing the full url into the search box starting with 'http'. You will see a drop-down box containing urls that match what you have typed. Clicking on any of them will load details for the URL.<br />
<br />
2. <nowiki>Type a search string and click 'Search'. You can optionally add wildcards to your search. For example, entering '*.html' will search for all URL's ending in .html. You can search for a specific site by entering 'http://www.site.com*' (ie, 'http://www.cnn.com/*')</nowiki></div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2863WarcManager2011-04-27T14:30:27Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
Before you begin, you will need <br />
<br />
1. Create database and setup permissions.<br />
mysql> create database webarc;<br />
mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
mysql> use webarc;<br />
mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
$ tar -xf apache-tomcat-6.0.32.tar.gz<br />
$ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
$ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
$ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
$ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
4. Start Tomcat<br />
$ apache-tomcat-6.0.32/bin/startup.sh<br />
<br />
=Indexing Web Content=<br />
<br />
The Warc Manager is able to index warc data stored on the same server that the manager is running on. To index warc files, you will need to know the directory on the server where they reside.<br />
<br />
1. From the main screen, click 'Ingest Files'. You will see the ingest file screen.<br />
<br />
2. Enter the following information:<br />
* Server Directory - The directory on the warc manager where warc files are stored. The scan will recursively search for any files ending in 'warc.gz'<br />
* Select Collection / New Collection: If you want to load these files into an existing collection, select it from the drop down box. Otherwise, type in the name of a new collection or label for these warc files.<br />
<br />
3. Click 'Scan Directory' When you are ready to index warc files. You can return to the scan page any time to view the progress of your upload.<br />
<br />
=Searching=</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=File:Ingestfiles.png&diff=2862File:Ingestfiles.png2011-04-27T14:29:53Z<p>Toaster: </p>
<hr />
<div></div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2861WarcManager2011-04-27T14:10:16Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
1. Create database and setup permissions.<br />
* mysql> create database webarc;<br />
* mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
* mysql> use webarc;<br />
* mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
* $ tar -xf apache-tomcat-6.0.32.tar.gz<br />
* $ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
* $ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
* $ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
* $ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" <br />
maxIdle="10" maxWait="-1" name="jdbc/warcdb" '''password="PASSWORD"''' testOnBorrow="true"<br />
type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc"<br />
validationQuery="SELECT 1"/><br />
==Detailed Installation==<br />
<br />
=Indexing Web Content=<br />
<br />
=Searching=</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2860WarcManager2011-04-27T14:06:25Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
<br />
* Source Code Repository: https://subversion.umiacs.umd.edu/warc-utils/<br />
* Nightly builds: http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/Warc%20Manager%202/<br />
<br />
=Installation=<br />
<br />
The installation consists of a few simple steps. <br />
<br />
1. Create database and setup permissions.<br />
* mysql> create database webarc;<br />
* mysql> grant all on webarc.* to webarc@localhost identified by 'PASSWORD';<br />
* mysql> use webarc;<br />
* mysql> source schema.sql;<br />
2. Install tomcat and JDBC driver<br />
* $ tar -xf apache-tomcat-6.0.32.tar.gz<br />
* $ cp mysql-connector-java-5.0.7-bin.jar apache-tomcat-6.0.32/lib<br />
3. Install and configure the Warc Manager<br />
* $ cp warc-webapp-1.0-SNAPSHOT.war apache-tomcat-6.0.32/webapps/warc.war<br />
* $ mkdir -p apache-tomcat-6.0.32/conf/Catalina/localhost<br />
* $ cp context.xml apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml <br />
* edit apache-tomcat-6.0.32/conf/Catalina/localhost/warc.xml and make sure the password under the resource line is the same as you specified above:<br />
<br />
<pre><br />
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxActive="20" maxIdle="10" maxWait="-1" name="jdbc/warcdb" password="webarc" testOnBorrow="true" type="javax.sql.DataSource" url="jdbc:mysql://localhost/webarc" username="webarc" validationQuery="SELECT 1"/></pre><br />
==Detailed Installation==<br />
<br />
=Indexing Web Content=<br />
<br />
=Searching=</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=WarcManager&diff=2859WarcManager2011-04-27T13:49:06Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The Warc Manager is a tool to help archives quickly browse, search, and analyze archives of web crawl data. The manager is lightweight database web application which indexes and provides a nice browsing interface to a collection of warc data.<br />
<br />
=Installation=<br />
<br />
=Searching=</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2858Main Page2011-04-27T13:46:46Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Our approach uses a distributed object architecture that operates on different levels of abstractions built around grid technologies and web services. Major software components have been prototyped and are currently in use in a number of pilot projects such as the Transcontinental Persistent Archives Project and Chronopolis.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Projects and Tools'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[Chronopolis|Chronopolis]]<li>[[Ace|Integrity Management (ACE)]]<li>[[WarcManager|Warc Management Utility]]</ul></td><br />
<td><ul><li>[[Webarc:Main|Web Archive Indexing]]<li>[[Replication:Main|Replication Monitoring]]</ul><br />
</td></tr></table><br />
'''AEC V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* <strike>Hudson</strike>Jenkins testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]<br />
<br />
=Old Projects=<br />
<ul><li>[[PAWN:Main|Ingestion (PAWN)]]<br />
<li>[[Focus:Main|Format Stewardship (FOCUS)]]<br />
<li>[[TPAP|TPAP]] - Transcontinental persistent archive prototype</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2857Main Page2011-04-19T04:43:46Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Our approach uses a distributed object architecture that operates on different levels of abstractions built around grid technologies and web services. Major software components have been prototyped and are currently in use in a number of pilot projects such as the Transcontinental Persistent Archives Project and Chronopolis.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Projects'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[Chronopolis|Chronopolis]]</ul></td><br />
<td><ul><li>[[Ace|Integrity Management (ACE)]]<li>[[Webarc:Main|Web Archiving]]<li>[[Replication:Main|Replication Monitoring]]</ul><br />
</td></tr></table><br />
'''AEC V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* <strike>Hudson</strike>Jenkins testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/jenkins/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]<br />
<br />
=Old Projects=<br />
<ul><li>[[PAWN:Main|Ingestion (PAWN)]]<br />
<li>[[Focus:Main|Format Stewardship (FOCUS)]]<br />
<li>[[TPAP|TPAP]] - Transcontinental persistent archive prototype</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2856Main Page2011-02-23T15:48:13Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Our approach uses a distributed object architecture that operates on different levels of abstractions built around grid technologies and web services. Major software components have been prototyped and are currently in use in a number of pilot projects such as the Transcontinental Persistent Archives Project and Chronopolis.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Projects'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[Chronopolis|Chronopolis]]</ul></td><br />
<td><ul><li>[[Ace|Integrity Management (ACE)]]<li>[[Webarc:Main|Web Archiving]]<li>[[Replication:Main|Replication Monitoring]]</ul><br />
</td></tr></table><br />
'''AEC V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* Hudson testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/hudson/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]<br />
<br />
=Old Projects=<br />
<ul><li>[[PAWN:Main|Ingestion (PAWN)]]<br />
<li>[[Focus:Main|Format Stewardship (FOCUS)]]<br />
<li>[[TPAP|TPAP]] - Transcontinental persistent archive prototype</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2855Main Page2011-02-16T04:27:25Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Our approach uses a distributed object architecture that operates on different levels of abstractions built around grid technologies and web services. Major software components have been prototyped and are currently in use in a number of pilot projects such as the Transcontinental Persistent Archives Project and Chronopolis.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Projects'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[TPAP|TPAP]]<li>[[Chronopolis|Chronopolis]]</ul></td><br />
<td><ul><li>[[PAWN:Main|Ingestion (PAWN)]]<li>[[Ace|Integrity Management (ACE)]]<li>[[Webarc:Main|Web Archiving]]<li>[[Replication:Main|Replication Monitoring]]<li>[[Focus:Main|Format Stewardship (FOCUS)]]</ul><br />
</td></tr></table><br />
'''AEC V 1.6 (2/3)'''<br />
* Token store exporting, improved db storing<br />
* Frontend UI improvements, <br />
* Build system (maven) changes<br />
<br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* Hudson testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/hudson/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Release_Guidlines&diff=2854Ace:Release Guidlines2011-02-01T20:40:24Z<p>Toaster: </p>
<hr />
<div>The tests should be performed prior to an ACE release. As many of these as possible should be included in the maven/hudson configs for automated testing. <br />
<br />
Prior to release testing, please make sure all bugs/features for that version have been completed on Redmine. https://scm.umiacs.umd.edu/redmine/adapt/projects/ace/roadmap. Update maven version and footer version prior to creating a release package.<br />
<br />
=Source Testing=<br />
<br />
# export a clean copy of the 'svn export https://subversion.umiacs.umd.edu/ace/trunk ace-test'<br />
# 'mvn package' in root, and make sure the build succeeds.<br />
<br />
=Clean deploy testing=<br />
# Create a new database using the current ace ace-am.sql file.<br />
# From your build, deploy the war file in ace-am/target/ace-am-XXX.war <br />
<br />
=Upgrade testing=<br />
# Grab the previous ace release, install the schema and war file<br />
# register the test collection.<br />
# shutdown instance, run update sql file, drop in new war file<br />
# Check error log for errors, check display for major feature changes, run test audit on existing colleciton, register new collection.<br />
<br />
=Releasing=<br />
When you have fully tested everything, use maven to generate the current release and update the current project with the next snapshot.<br />
# Run mvn release:prepare in the root project<br />
# upload the two new binary versions in ace-dist/target to redmine.<br />
# export the source in the tags/ace-xxx version, tar/gz it and upload to redmine</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Release_Guidlines&diff=2853Ace:Release Guidlines2011-02-01T20:29:20Z<p>Toaster: </p>
<hr />
<div>The tests should be performed prior to an ACE release. As many of these as possible should be included in the maven/hudson configs for automated testing. <br />
<br />
Prior to release testing, please make sure all bugs/features for that version have been completed on Redmine. https://scm.umiacs.umd.edu/redmine/adapt/projects/ace/roadmap. Update maven version and footer version prior to creating a release package.<br />
<br />
=Source Testing=<br />
<br />
# export a clean copy of the 'svn export https://subversion.umiacs.umd.edu/ace/trunk ace-test'<br />
# 'mvn package' in root, and make sure the build succeeds.<br />
<br />
=Clean deploy testing=<br />
# Create a new database using the current ace ace-am.sql file.<br />
# From your build, deploy the war file in ace-am/target/ace-am-XXX.war <br />
<br />
=Upgrade testing=<br />
# Grab the previous ace release, install the schema and war file<br />
# register the test collection.<br />
# shutdown instance, run update sql file, drop in new war file<br />
# Check error log for errors, check display for major feature changes, run test audit on existing colleciton, register new collection.<br />
<br />
=Packaging=<br />
A binary distribution can be created by running 'mvn package' in the root project. This will create a binary tgz and zip file in ace-dist/target that has all necessary war, db, and documenation.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Release_Guidlines&diff=2852Ace:Release Guidlines2011-02-01T20:27:19Z<p>Toaster: </p>
<hr />
<div>The tests should be performed prior to an ACE release. As many of these as possible should be included in the maven/hudson configs for automated testing. <br />
<br />
Prior to release testing, please make sure all bugs/features for that version have been completed on Redmine. https://scm.umiacs.umd.edu/redmine/adapt/projects/ace/roadmap. Update maven version and footer version prior to creating a release package.<br />
<br />
=Source Testing=<br />
<br />
# export a clean copy of the 'svn export https://subversion.umiacs.umd.edu/ace/trunk ace-test'<br />
# 'mvn package' in root, and make sure the build succeeds.<br />
<br />
=Clean deploy testing=<br />
# From your build, deploy the war file in ace-am/target/ace-am-XXX.war <br />
<br />
=Upgrade testing=<br />
<br />
=Packaging=<br />
A binary distribution can be created by running 'mvn package' in the root project. This will create a binary tgz and zip file in ace-dist/target that has all necessary war, db, and documenation.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Main_Page&diff=2851Main Page2011-01-26T17:39:26Z<p>Toaster: </p>
<hr />
<div>=Overview=<br />
The ADAPT project is developing technologies for building a scalable and reliable infrastructure for the long-term access and preservation of digital assets. Our approach uses a distributed object architecture that operates on different levels of abstractions built around grid technologies and web services. Major software components have been prototyped and are currently in use in a number of pilot projects such as the Transcontinental Persistent Archives Project and Chronopolis.<br />
<br />
<table width="75%" style="margin-left: 75px;"><br />
<tr><td>'''Projects'''</td><td>'''Research'''</td></tr><tr><br />
<td valign="top"><ul><li>[[TPAP|TPAP]]<li>[[Chronopolis|Chronopolis]]</ul></td><br />
<td><ul><li>[[PAWN:Main|Ingestion (PAWN)]]<li>[[Ace|Integrity Management (ACE)]]<li>[[Webarc:Main|Web Archiving]]<li>[[Replication:Main|Replication Monitoring]]<li>[[Focus:Main|Format Stewardship (FOCUS)]]</ul><br />
</td></tr></table><br />
'''ACE Development infrastructure improvements (1/26)'''<br />
* public source repository https://subversion.umiacs.umd.edu/ace/<br />
* Hudson testing and daily builds http://adaptvm01.umiacs.umd.edu:8080/hudson/job/ACE%20Audit%20Manager/<br />
* Projects converted from netbeans build to maven3 for easier collaboration<br />
<br />
'''Python tutorials for ACE'''<br />
* [[Ace:IMSPython| Requesting and validating IMS tokens]]<br />
<br />
'''Undergrad positions available'''<br />
* The ADAPT project is hiring undergrad programmers to work on a variety of projects<br />
* Learn more on our [[Employment]] page<br />
<br />
'''Ace V 1.5 Released (6/25)'''<br />
* Source Released under BSD License<br />
* Resource migration<br />
* Lots of internal refactoring, dependency restructuring<br />
<br />
<br />
'''New Issue Tracker'''<br />
The adapt project now has an issue tracker to manage releases of our software and track bugs/feature requests.<br />
* https://scm.umiacs.umd.edu/redmine/adapt/<br />
<br />
<br />
<br />
'''Conference Videos Online (4/21/09)'''<br />
Videos for the two-day conference, "Partnerships in Innovation II: From Vision to Reality and Beyond" held on October 7- 8, 2008 in College Park, Maryland, USA are now [[Partnerships 2008 Videos|online]]</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Devel_Pages&diff=2850Ace:Devel Pages2011-01-26T16:15:42Z<p>Toaster: </p>
<hr />
<div>=General=<br />
<br />
* [https://scm.umiacs.umd.edu/redmine/adapt/projects/ace Bug Tracker, Source code..]<br />
* [http://groups.google.com/group/ace-devel Discussion List]<br />
* [http://adaptvm01.umiacs.umd.edu:8080/hudson/job/ACE%20Audit%20Manager Hudson build tests]<br />
<br />
=Token Store Format=<br />
<br />
Information for creating, parsing and validating Token stores<br />
<br />
* [[Ace:TokenStore|Token Store Format]]<br />
* [[Ace:StorePython|Processing Token Stores in Python]]<br />
<br />
=Audit Manager=<br />
<br />
Developer pages and documentation for the ACE Audit Manager web application<br />
<br />
* [[Ace:JSON Interface|JSON Interface to Audit Manager]]<br />
* [[Ace:Roles|Security Roles]]<br />
* [[Ace:Summary Details|Summary generation]]<br />
* [[Ace:Expanding the Audit Manager|Adding Storage Types]] <br />
* [[Ace:Release Guidlines|Release Guidelines]]<br />
<br />
=Integrity Management Service=<br />
<br />
* [[Ace:IMSPython|Accessing the IMS using python]]<br />
<br />
=Other=<br />
<br />
* [[Ace:Deprecated|Deprecated Pages]]</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Release_Guidlines&diff=2849Ace:Release Guidlines2011-01-24T21:34:26Z<p>Toaster: </p>
<hr />
<div>The tests should be performed prior to an ACE release. As many of these as possible should be included in the maven/hudson configs for automated testing. <br />
<br />
Prior to release testing, please make sure all bugs/features for that version have been completed on Redmine. https://scm.umiacs.umd.edu/redmine/adapt/projects/ace/roadmap. Update maven version and footer version prior to creating a release package.<br />
<br />
=Source Testing=<br />
<br />
# export a clean copy of the 'svn export https://subversion.umiacs.umd.edu/ace/trunk ace-test'<br />
# 'mvn package' in root, and make sure the build succeeds.<br />
<br />
=Clean deploy testing=<br />
# From your build, deploy the war file in ace-am/target/ace-am-XXX.war <br />
<br />
=Upgrade testing=<br />
<br />
=Packaging=<br />
The binary distribution should include a</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Release_Guidlines&diff=2848Ace:Release Guidlines2011-01-24T21:26:19Z<p>Toaster: </p>
<hr />
<div>The tests should be performed prior to an ACE release. As many of these as possible should be included in the maven/hudson configs for automated testing.<br />
<br />
'Source Testing'</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Devel_Pages&diff=2847Ace:Devel Pages2011-01-24T21:24:57Z<p>Toaster: </p>
<hr />
<div>=General=<br />
<br />
* [https://scm.umiacs.umd.edu/redmine/adapt/projects/ace Bug Tracker, Source code..]<br />
* [http://groups.google.com/group/ace-devel Discussion List]<br />
* [http://adaptvm01.umiacs.umd.edu:8080/hudson/job/ACE Audit Manager Hudson build tests]<br />
<br />
=Token Store Format=<br />
<br />
Information for creating, parsing and validating Token stores<br />
<br />
* [[Ace:TokenStore|Token Store Format]]<br />
* [[Ace:StorePython|Processing Token Stores in Python]]<br />
<br />
=Audit Manager=<br />
<br />
Developer pages and documentation for the ACE Audit Manager web application<br />
<br />
* [[Ace:JSON Interface|JSON Interface to Audit Manager]]<br />
* [[Ace:Roles|Security Roles]]<br />
* [[Ace:Summary Details|Summary generation]]<br />
* [[Ace:Expanding the Audit Manager|Adding Storage Types]] <br />
* [[Ace:Release Guidlines|Release Guidelines]]<br />
<br />
=Integrity Management Service=<br />
<br />
* [[Ace:IMSPython|Accessing the IMS using python]]<br />
<br />
=Other=<br />
<br />
* [[Ace:Deprecated|Deprecated Pages]]</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Main&diff=2846Ace:Main2011-01-24T21:22:21Z<p>Toaster: </p>
<hr />
<div>ACE (Auditing Control Environment) is a system that incorporates a new methodology to address the integrity of long term archives using rigorous cryptographic techniques. ACE continuously audits the contents of the various objects according to the policy set by the archive, and provides mechanisms for an independent third-party auditor to certify the integrity of any object.<br />
<br />
ACE consists of two components, the first an Audit Manager(AM) that checks files locally to ensure they have not been compromised. The second part, the Integrity Management Service (IMS), issues tokens that the AM can use to verify that its local store of file digests has not been tampered with. <br />
<br />
Anyone wishing to audit their files with ACE only needs to install an Audit Manager locally to manage their files. The ADAPT project runs a publically available IMS at ims.umiacs.umd.edu that any group may freely use.<br />
<br />
{| width="75%" style="margin-left: 75px;"<br />
| '''Audit Manager''' || '''Integrity Management Service'''<br />
|-<br />
| [[Ace:Audit Manager Installation Guide|Download Audit Manager]] || [[Ace:Ace IMS System|System Documentation]]<br />
|-<br />
| [[Ace:Audit Manager User Guide|User Guide]] || [http://ims.umiacs.umd.edu:8080/ace-ims/IMSWebService IMS Webservice]<br />
|-<br />
| [http://groups.google.com/group/ace-devel Discussion List] || [[Ace:Validating Witness and Tokens|Validating Tokens and Witnesses]]<br />
|-<br />
| [[Ace:Audit Manager Upgrading|Upgrade Audit Manager]] || [http://groups.google.com/group/ace-ims-witness?hl=en IMS Witness Google Goup]<br />
|- <br />
| [[Ace:AM Config Reference|AM Configuration Options]] || [http://mailman.umiacs.umd.edu/mailman/listinfo/ims-witness IMS Witness UMIACS listserv]<br />
|-<br />
| || [[Ace:IMS Installation| IMS Installation Guide]]<br />
|-<br />
| '''Resources'''<br />
|-<br />
| [https://scm.umiacs.umd.edu/redmine/adapt/projects/show/ace Files and Issue Tracker]<br />
|-<br />
| [[Ace:Devel Pages|Development Documentation]]<br />
|-<br />
| [[Ace:Webstart Client|Auditor Application]] || <br />
|}<br />
<br />
<br />
'''Papers and Presentations'''<br />
<br />
* Smorul, M., Song, S., and JaJa, J. An Implementation of the Audit Control Environment(ACE) to Support the Long Term Integrity of Digital Archives. in DigCCurr 2009. 2009: University of North Carolina.[[media:DigCCurr2009_060909.pdf|pdf]] presentation [[media:Smorul-digccurr09.ppt|ppt]]<br />
* ACE Presentation at Archiving 2007 (5/07) [[media: ACE-Archiving2007.ppt|ppt]]<br />
* Song, S. and JaJa, J. ACE: A Novel Software Platform to Ensure the Integrity of Long Term Archives. in Archiving 2007. 2007: IS&T. [[media:rad71E67.pdf|pdf]]<br />
<br />
Development of the ACE Version1.0 release was partially sponsored by the US National Archives and Records Administration. The basic research on ACE was partially sponsored under a DIGARCH grant from NSF/Library of Congress.</div>Toasterhttps://wiki.umiacs.umd.edu/adapt/index.php?title=Ace:Main&diff=2845Ace:Main2011-01-24T21:22:06Z<p>Toaster: </p>
<hr />
<div>ACE (Auditing Control Environment) is a system that incorporates a new methodology to address the integrity of long term archives using rigorous cryptographic techniques. ACE continuously audits the contents of the various objects according to the policy set by the archive, and provides mechanisms for an independent third-party auditor to certify the integrity of any object.<br />
<br />
ACE consists of two components, the first an Audit Manager(AM) that checks files locally to ensure they have not been compromised. The second part, the Integrity Management Service (IMS), issues tokens that the AM can use to verify that its local store of file digests has not been tampered with. <br />
<br />
Anyone wishing to audit their files with ACE only needs to install an Audit Manager locally to manage their files. The ADAPT project runs a publically available IMS at ims.umiacs.umd.edu that any group may freely use.<br />
<br />
{| width="75%" style="margin-left: 75px;"<br />
| '''Audit Manager''' || '''Integrity Management Service'''<br />
|-<br />
| [[Ace:Audit Manager Installation Guide|Download Audit Manager]] || [[Ace:Ace IMS System|System Documentation]]<br />
|-<br />
| [[Ace:Audit Manager User Guide|User Guide]] || [http://ims.umiacs.umd.edu:8080/ace-ims/IMSWebService IMS Webservice]<br />
|-<br />
| [http://groups.google.com/group/ace-devel Discussion List] || [[Ace:Validating Witness and Tokens|Validating Tokens and Witnesses]]<br />
|-<br />
| [[Ace:Audit Manager Upgrading|Upgrade Audit Manager]] || [http://groups.google.com/group/ace-ims-witness?hl=en IMS Witness Google Goup]<br />
|- <br />
| || [http://mailman.umiacs.umd.edu/mailman/listinfo/ims-witness IMS Witness UMIACS listserv]<br />
|-<br />
| [[Ace:AM Config Reference|AM Configuration Options]] || [[Ace:IMS Installation| IMS Installation Guide]]<br />
|-<br />
| '''Resources'''<br />
|-<br />
| [https://scm.umiacs.umd.edu/redmine/adapt/projects/show/ace Files and Issue Tracker]<br />
|-<br />
| [[Ace:Devel Pages|Development Documentation]]<br />
|-<br />
| [[Ace:Webstart Client|Auditor Application]] || <br />
|}<br />
<br />
<br />
'''Papers and Presentations'''<br />
<br />
* Smorul, M., Song, S., and JaJa, J. An Implementation of the Audit Control Environment(ACE) to Support the Long Term Integrity of Digital Archives. in DigCCurr 2009. 2009: University of North Carolina.[[media:DigCCurr2009_060909.pdf|pdf]] presentation [[media:Smorul-digccurr09.ppt|ppt]]<br />
* ACE Presentation at Archiving 2007 (5/07) [[media: ACE-Archiving2007.ppt|ppt]]<br />
* Song, S. and JaJa, J. ACE: A Novel Software Platform to Ensure the Integrity of Long Term Archives. in Archiving 2007. 2007: IS&T. [[media:rad71E67.pdf|pdf]]<br />
<br />
Development of the ACE Version1.0 release was partially sponsored by the US National Archives and Records Administration. The basic research on ACE was partially sponsored under a DIGARCH grant from NSF/Library of Congress.</div>Toaster