Pawn:XFDU Project notes

Overview

The goal is to be able to take items from pawn packages and push them into an xfdu package. In this version, the created packages will be self-contained, meaning that xfdu will generate a gzipped tar, zip or other type that it supports. The final destination of these packages will be a local directory on the receiving server, or an ftp server. In addition, technical documentation will be written and posted to the PawnXfduPublishing page, in addition, weekly progress reports will be posted using the comments section at the page bottom.

Setting up your test environment (1 day)

1. Setup your own pawn receiver.

There is a scheduler and manager running on naradev04. You should install your own receiver that you can use to test (ie, start and stop on your own). In addition, you will need to create accounts, record schedules, and packages to test with. Setup your receivier to run on a port other than 8080, before starting it, you'll need to configure the receiving server. Become familiar with how logging on the receiver works. You'll use this to debug your code.

2. Install the null-receiver resource.

You don't have to worry about ingesting jars as the null resoruce is part of the adapt-common package.

3. Create a new package in the plugins directory called pawn-xfdu.

Use 'edu.umiacs.pawn.resource.xfdu' as your base class namespace. You can assume the following libraries and packages will be supplied by pawn.

The xfdu library has been loaded into the standard library location and should called 'xfdu' in the library manager. In addition to the jars in the xfdu package, the following libraries are also required to use the xfdu library.

• javax.mail
• javax.activation
• javax.xml.bind
• org.apache.axis
• javax.xml.parsers

See the PawnXfduPublishing for a complete list of jars and mappings. Please note, xfdu has not been tested in netbeans, so the above may be slightly off, or may not include all libraries.

4. Load your project as a resource.

To load your project as a resource, you will have to use the pawn scheduler admin tool, click on resources, and att the resource, it's name should be the full class that implements the resource factory. After creating your resource upload the jar your project generated. If you need additional libraries, (ie xfdu) you'll have to load them along with your project jar into the scheduler (see srb resource on naradev02's scheduler for an example).

• javax.mail
• org.apache.axis
• org.apache.log4j
• javax.activation
• javax.xml.parsers

• umiacs-util
• adapt-xml
• adapt-common

Milestones

1. Create the base three interfaces for creating xfdu packages.

Take the selected items that are presented to <nop>PAWN and create a package from them. The package should be stored in /tmp on the receiving server and have a user/client -defined name. For now, don't worry about processing manifests (the processManifest method), just items and metadata. When creating the xfdu package all items should be referenced, do not worry about inlining xml data or metadata for now. You will also need to report back to pawn via the processXXX commands the final destination of where items got placed. This should be some reference to the created package and location in the package.

As there may be multiple (non-overlapping) selections presented to the driver, each selected path/item should be ingested w/ the same structure that appears in the package.

 root
  + dir1 <- selected
  + dir2
     + dir3 <- selected
        + dir5

If the driver is presented w/ dir1 and dir3 selections, they should be the roots of the xfdu package.

2. Add multiple destination support.

As putting the resulting packages in /tmp on a receiving server is not an ideal destination, the next step is to create an interface that can be extended to support different types of final destinations for the package. The two destinations that we will support initially are local file (easy case), where the management gui specifies a directory that exists on a receiving server, and the xfdu resource just places a copy of the package in that directory. The second transport is an ftp transport where the management gui can set a remote ftp server, user, password, and directory to dump the files. Please keep in mind these won't be the only two transports in the future, so adding additional destinations must be easy, and documented on the xfdu publishing page.

3. Add metadata and object type mapping.

Items in a pawn bundle have namespaces and types, and for objects, mimetypes associated with them. An xfdu package has the option of referring to items by either inlining them (base-64, or xml) or providing references to them. Currently you have only included items in a manifest by reference. This part of the project is to allow the management gui to map type + namespace combinations to either referenced, or inlining behavior. For example, the dc xml from the hcil project may have been better inlined, This is where that would occur.

Comments, progress updates

Please drop a weekly progress update in here

%COMMENT{type="below"}%Found a bug in xfdu.packaging.Manifest at method add(Metadata metadata). The id appending of '1' is repeated twice; once for the first object added and then another when the size of the metadataSection is added. Fixed this by incrementing a 1 to the size of the metadataSection when used in the conditional statement for id number appendaging. Also sent an email to Sergey, one of the programmers of the xfdu package, a note about this bug for future developments/modifications of the xfdu package.

-- Main.ChristopherWamble - 25 Jul 2006

Eureka! Reached out goal of creating an xfdumanifest.xml file to preserve the heirarchy of the package along with adding the content to the package. Getting to this step required overcoming some difficulties along the way. In particular, the first difficulty was realizing that the manifest file-the one we manipulated to preserve the order as shown in PAWN- was not returning a Map instance of the objects it contained. The solution after looking at the saveAsJar() method of the Package class was to append the file names to 'file:' since the helper methods of saveAsJar() use 'file:' as a delimiter in adding components to the map. The next challenge was deciding how to create the files and store them into the package. saveAsJar() method performed this operation by creating a URL, using several helper methods to manipulate the URL path, reading the URL into the JarOutputStream created by the methods, and finally adding the manifest file of the package. After a few trials we were able to come up with a solution by creating a HashMap to store each of the InputStream objects passed in through processObject() of XFDUMover (implementing class of edu.umiacs.pawn.resource.DataMover) mapping the file name to it's respective InputStream instance. Afterwards in endTransfer() we created the package, passed it into an implementing class of xfdu.packaging.Packager (with several needed objects passed through to its constructor for internal usage) called XFDUPackager, traveresed the manifest file's objects retrieving the InputStream instances of each object from the map we created and using several of the methods already used by the Package class to import the data. Next step is getting the package to be read by the standalone XFDUWorkbench. This will be somewhat of a challenge however since the workbench throws countless exceptions for nearly every action performed.

-- Main.ChristopherWamble - 29 Jun 2006

Near completion of an algorithm that preserves the heirarchy of folders and items within the xfdumanifest.xml that is in each xfdu package. Worked successfully for one book 2 levels deep (which is typical for a ICDL book stored in PAWN). Next test cases will be for several books and arbitrary items before moving onto data content transfer to the package.

-- Main.ChristopherWamble - 15 Jun 2006

Finally overcame the dreaded jaxb.properties problem by refering to a forum (https://jaxb.dev.java.net/faq/jaxb10faq.html) by changing 'jc = JAXBContext.newInstance( JAXB_CONTEXT)' method to 'jc = JAXBContext.newInstance( JAXB_CONTEXT, this.getClass().getClassLoader() )' within the constructor of xfdu.util.XFDUElementBuilder. Using a different newInstance() method by adding a second argument allowed me to overcome the jaxb.properties location problem. After fixing a few other bugs and exceptions along the way I was able to create an xfdu package successfully in /tmp/pool-xfdu. The xfdumanifest.xml file printed out nicely. With the hardest part completed, the next steps are to place the data content into the package (saved as a jar file) and modify the xfdumanifest.xml file to preserve the heirarchical structure that it came in.

-- Main.ChristopherWamble - 06 Jun 2006

Discovered that the mimetype string representation of the global TransferContext object in DataMover differs slightly from those stored in xfdu's MimeType class. After fixing the TransferContext mimetype string to account for this shortcoming I encountered XFDUExceptions for each object. The cause being that the jaxb.properties files is missing from xfdu.core package. I've read through several forums regarding this problem and currently reading up on jAXB and how it works.

-- Main.ChristopherWamble - 24 May 2006

Successfully found a way to create a package with a valid xfdumanifest.xml file that contains DataObjectPointers to DataObject segments that contain FileLocation pointers. This was done by using the implementing classes of DataObjectType, BytestreamType, and ReferenceType from xfdu.core.impl package. The process is as follows: 1. create Data object using Data(MimeType, InputStream) constructor using transferContext's getInputStream() as the second argument. 2. create and initialize a DataObjectType, BytestreamType, and ReferenceType object with correspoinding implementing classes from xfdu.core.impl (DataObjectTypeImpl, BytestreamTypeImpl, ReferenceTypeImpl respectively). 3. use setHref() for the ReferenceType object to set the file pointer location, add the ReferenceType object to BytestreamType object, add the BytestreamType object to DataObjectType object, and finally the DataObjectType object to the Data object created earlier.

After filling in the Data object I stored it in a List object and refered to it later in DataMover's endTransfer() method. Using the notes from the xfdu wiki's getting started manual on the preferred way of building a package, I copied the code into endTransfer() replacing the Data list with my own list and making up a list of metadata objects to correspond with the Data objects. In addition, I also added the data pointers to the ContentUnitTypeObject using information from the Data object list. So far however the package is empty and only contains the xfdu manifest. All of this was done locally in Netbeans, but when I transfered this code to run on the receiving server I continually get java.lang.IllegalArgumentException for each object in response to it being thrown from xfdu.util.MimeType's fromValue() method, which I use in passing the mimetype to the Data object created in DataMover's processObject(). I'm currently investigating this bug now in hopes to find a solution that will lead into getting some of the same results that were done locally within netbeans.

-- Main.ChristopherWamble - 22 May 2006

My last approach: DataObjectType-(into)->Data-(into)->Manifest-(into)->InformationPackageMapType-(into)->Package was unsuccessful. After reading the Manifest and Package source code and understanding the reason (incompatibility of data types-manifest depends on ByteStreamType from xfdu.core as a List) I decided to start from square one and reread the XFDU Specification (http://sindbad.gsfc.nasa.gov/xfdu/pdfdocs/iprwbv2a.pdf). After reviewing the documentation I gained a better understanding of the purpose of the manifest file/object and it's relation to the package or Package Interchange File as well as how all the package storage is based on XML schema. With this information realized I started from scratch implementing DataObjectType (to store in Data class instance), ByteStreamType (to store bytes of the data), and TransformObjectType (for transforming data object back to readable content when package is opened-see XFDU API of DataObjectType for more details). First I began by taking baby steps and developing an empty package within XFDUMover (implementing class of DataMover from edu.umiacs.pawn.resource). This was done successfully and the package was saved on naradev04 within /tmp/pool-xfdu. However I ran into many troubles after I implemented the 3 interfaces (DataObjectType, ByteStreamType, TransformObjectType) and tried putting Data objects with incapsulated DataObjectType objects into the package. I received the following problems: 1. Continually received NullPointerException from processObject() when I tried adding the Data object to the package using Package.add(Data data) within this method. 2. XFDUException when I stored all the Data objects in a List object and then used a for loop in endTransfer() to add them to the package.

I will try to next few approaches to overcome these obstacles: 1. Trace the XFDUException through to the source code of Package and Manifest and find a way around this exception 2. Try creating a local package to a driver on my Netbeans application and then apply a successful approach to the receiving server testbed.

-- Main.ChristopherWamble - 10 May 2006

Unable to store manifest files in package successfully due to XFDUException: unable to locate jaxb.properties for xfdu.core.

Decided to backtrack and create an XFDU locally to my machine. Ran into ClassCastException: XFDU.core.impl.ContentUnitImpl after storing XFDUManifest objects (a subclass of Manifest and implementing class of InformationPackageMapType) into a XFDU Package object and retrieve information stored by returning a List of InformationPackageMapType objects.

Decided to try a new approach to package creation. Instead of subclassing Manifest and Data classes from xfdu.packaging, I will instead do the following: 1. Implement DataObjectType interface (intention is to store information from a TransferContext object). 2. Store implemented DataObjectType into an xfdu.packaging.Data object. 3. Store Data objects into xfdu.packaging.Manifest object 4. Implement InformationPackageMapType and store it into a Package object.

-- Main.ChristopherWamble - 03 May 2006

created manifest files with data objects stored within them, preparing to store manifests into XFDU package and process to naradev04.

-- Main.ChristopherWamble - 01 May 2006

Browse through XFDU API (http://sindbad.gsfc.nasa.gov/xfdu/apidocs/index.html) to select classes that will be used by the pawn-xfdu tool package.

-- Main.ChristopherWamble - 17 Apr 2006

Testing environment has been completed, pawn-xfdu has been loaded as a resource.

-- Main.ChristopherWamble - 07 Apr 2006