Getting products from a remote FileManager

Given the following scenario:

• FileManager is running on machineA.
• Workflow/PGE tasks should run on machineB.

There are 2 strategies to operate on remote files:

1. Use NFS to simulate a local filesystem on machineB
2. Use fmprod to download the product to machineB from a web URL

When your Workflow/PGE task is done, you can ingest the output files into the remote FileManager using org.apache.oodt.cas.filemgr.datatransfer.RemoteDataTransferFactory as the clientTransferer.

Use NFS

The idea here is to use NFS to mount the data archive onto a common root file path. Then all file paths will appear to be on the local file system.

1. Edit /etc/exports to allow machineB to see the data archive on machineA (for more information, type "man exports")
   machineA: /etc/exports

   /Users/me/filemgr/data/archive -network=10.0.0.0 -mask=255.255.255.0
   

2. Restart the NFS service on machineA

   sudo nfsd restart
   

3. Mount the remote file system on machineB (for more information, type "man mount")

   sudo mkdir -p /net/machineA
   sudo mount -t nfs machineA:/Users/me/filemgr/data/archive /net/machineA
   

4. Create a symbolic link on machineA

   sudo ln -s /Users/me/filemgr/data/archive /net/machineA
   

5. Edit product-types.xml to use the symbolic link
   machineA: product-types.xml

   ...
   <type id="urn:MyProdTypeId" name="MyProdTypeName">
     <repository path="file:///net/machineA"/>
   ...
   

6. The FileManager should now return file paths that are reachable by machineB.

If your products were already previously ingested, you must update the [FileLocation] metadata element for your ingested products. There are several ways to update the catalog with the new NFS file location:

• Re-ingest, making sure the repository path in product-types.xml uses the NFS mount.
• Re-ingest, using a different Versioner, which gives the NFS mount as final file location.
• MetadataBasedProductMover. Run the following locally on machineA:

  java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.MetadataBasedProductMover \
      --fileManagerUrl http://localhost:9000 \
      --typeName MyProdTypeName \
      --pathSpec /net/machineA/[Filename]
  

Use fmprod

The idea here is to deploy a Product Server to allow anyone to download products with an HTTP request.

1. Build fmprod (assumes you have OODT sources checked out)

   cd webapp/fmprod
   mvn clean package
   

2. Deploy fmprod (cas-product-VERSION.war)
3. Download the product with an HTTP request (GET or POST)
   • To get a single product, use the "productID" query parameter.

     curl 'http://webappMachine/fmprod/data?productID=< your product id >'
     

   • To get a dataset (products of a certain type) as a zip file, use the "typeID" query parameter.

     curl 'http://webappMachine/fmprod/dataset?typeID=< your product type id >'
     

4. The previous download step can be wrapped in a "FileStager" task. Subsequent tasks can now operate on the downloaded local file.