Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

Table of Contents

Introduction

Given the following scenario:

...

When your Workflow/PGE task is done, you can ingest the output files into the remote FileManager using org.apache.oodt.cas.filemgr.datatransfer.RemoteDataTransferFactory as the clientTransferer.

Anchor
nfs
nfs

Use NFS

The idea here is to use NFS to mount the data archive onto a common root file path. Then all file paths will appear to be on the local file system.

...

  • Re-ingest, making sure the repository path in product-types.xml uses the NFS mount.
  • Re-ingest, using a different Versioner, which gives the NFS mount as final file location.
  • MetadataBasedProductMover. Run the following locally on machineA:

    No Format
    java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.MetadataBasedProductMover \
        --fileManagerUrl http://localhost:9000 \
        --typeName MyProdTypeName \
        --pathSpec /net/machineA/[Filename]
    

Anchor
fmprod
fmprod

Use fmprod

The idea here is to deploy a Product Server to allow anyone to download products with an HTTP request.

  1. Build fmprod (assumes you have OODT sources checked out)

    No Format
    cd webapp/fmprod
    mvn clean package
    
  2. Deploy fmprod (cas-product-VERSION.war)
  3. Download the product with an HTTP request (GET or POST)
    • To get a file product, use the "productID" query parameter.

      No Format
      curl 'http://webappMachine/fmprod/data?productID=< your product id >'
      
    • To get a hierarchical product, : use additional 'refIndex' parameter for each of the 'n' datastore referencesor format parameter. Note 'refIndex=0' is the directory name.

      No Format
      curl 'http://webappMachine/fmprod/data?productID=< your product id >&refIndex=1'
      curl 'http://webappMachine/fmprod/data?productID=< your product id >&refIndex=2'
      curl 'http://webappMachine/fmprod/data?productID=< your product id >&refIndex=< n >format=application/x-zip'
    • To get a dataset (products of a certain type) as a zip file, use the "typeID" query parameter.

      No Format
      curl 'http://webappMachine/fmprod/dataset?typeID=< your product type id >'
      
    • The filename is sent to to requesting client in the server header. It's possible to force wget and curl to use this information to name the file. To download a product and to figure out the product name from the server header information use one of the following commands:

      No Format
      curl --remote-name --remote-header-name http://webappMachine/fmprod/data?productID=< your product id >
      wget --content-disposition http://webappMachine/fmprod/data?productID=< your product id >
      
  4. The previous download step can be wrapped in a "FileStager" task. Subsequent tasks can now operate on the downloaded local file.

...