Versions Compared

Old Version 2

changes.mady.by.user Claus Ibsen

Saved on

compared with

New Version 3

changes.mady.by.user Claus Ibsen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Wiki Markup
h2. FTP/SFTP Component

This component provides access to remote file systems over the FTP and SFTP protocols.

h3. URI format

{code}
ftp://[username@]hostname[:port]/filename[?options]
sftp://[username@]hostname[:port]/filename[?options]
{code}

Where *filename* represents the underlying file name or directory. Can contain nested folders.
The *username* is currently only possible to provide in the hostname parameter.

If no *username* is provided then {{anonymous}} login is attempted using no password.
If no *port* number is provided. Camel will provide default values according to the protocol. (ftp = 21, sftp = 22)

h4. Examples
{{ftp://someone@someftpserver.com/public/upload/images/holiday2008?password=secret&binary=true}}
{{ftp://someoneelse@someotherftpserver.co.uk:12049/reports/2008/budget.txt?password=secret&binary=false&directory=false}}
{{ftp://publicftpserver.com/download}}

{warningtip:title=TimestampMore examples}
InThis Camel 1.4 or older the FTP consumer usescomponent is an internalextension timestampof forthe last[File2] pollingcomponent. ThisSo timestampthere iscould usedbe tomore matchsamples forand newdetails remoteon files: if remote file modified timestamp > last poll timestamp => file can be consumed.

In Camel 1.5 this algorithm has been disabled by default as its not reliable over the FTP protocol. FTP Servers only return file modified timestamps using HH:mm (not seconds). And of course the clocks between the client and server can also be out of sync. Bottom line is that timestamp check for FTP protocol should *not* be used. That is why this feature is marked as *@deprecated* and will be removed in Camel 2.0.

We encourage you to use a different strategy for matching new remote files: such as deleting or moving the file after download.
{warning}

{tip:title=More examples}
This component is an extension of the [File] component. So there could be more samples and details on the [File] component page as well.
{tip}

h3. Camel 1.x URI Options

{div:class=confluenceTableSmall}
|| Name || Default Value || Description ||
| directory | true| indicates whether or not the given file name should be interpreted by default as a directory or file (as it sometimes hard to be sure with some FTP servers) |
| password | null | specifies the password to use to login to the remote file system |
| binary | false | specifies the file transfer mode BINARY or ASCII. Default is ASCII. |
| ftpClientConfig | null | Camel 1.5: Reference to a bean in the registry as a {{[org.apache.commons.net.ftp.FTPClientConfig|http://commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html]}} class. Use this option if you need to configure the client according to the FTP Server date format, locale, timezone, platform etc. See the javadoc {{[FTPClientConfig|http://commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html]}} for more documentation.
| consumer.recursive | true/false | if a directory, will look for changes in files in all the sub directories. Is *true* as default for Camel 1.4 or older. Will change to *false* as default value as of Camel 1.5 |
| consumer.setNames | true  | Used by FTPConsumer. If true Camel will use the filename the file has on the FTP server. The filename is stored on the IN message in the header {{FileComponent.HEADER_FILE_NAME}}. *Note:* In Camel 1.4 the default value has changed to *true*. |
| consumer.delay | 500 | Delay in millis between each poll |
| consumer.initialDelay | 1000 | Millis before polling starts |
| consumer.userFixedDelay | false | *true* to use fixed delay between pools, otherwise fixed rate is used. See [ScheduledExecutorService|http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ScheduledExecutorService.html] in JDK for details. |
| consumer.regexPattern | null | Used by FTPConsumer. Regular expression to use for matching files when consuming. |
| consumer.exclusiveReadLock | false| Camel 1.5: Used by FTPConsumer. If set to true Camel will only poll the ftp files if it has exclusive read to the file (= the file is not in progress of being written). Camel will wait until it is granted, testing once every second. The test is implemented by Camel will try to rename the file. Setting to false Camel will poll the file even if its in progress of being written. |
| consumer.deleteFile | false | Camel 1.5: Used by FTPConsumer. Flag to set if the consumed file should be deleted after it has been downloaded. |
| consumer.moveNamePrefix | null | Camel 1.5: Used by FTPConsumer. The prefix String perpended to the filename when moving it. For example to move processed files into the _done_ directory, set this value to 'done/' |
| consumer.moveNamePostfix | null | Camel 1.5: Used by FTPConsumer. The postfix String appended to the filename when moving it. For example to rename processed files from _foo_ to _foo.old_ set this value to '.old' |
| consumer.excludedNamePrefix | null | Camel 1.5: Used by FTPConsumer. Is used to exclude files if filename is starting with the given prefix. |
| consumer.excludedNamePostfix | null | Camel 1.5: Used by FTPConsumer. Is used to exclude files if filename is ending with the given postfix. |
| consumer.timestamp | false | Camel 1.5: *@deprecated* will be removed in Camel 2.0. This option is only for backwards comparability. |
| expression | null | Camel 1.5: Use expression to dynamically set the filename. This allows you to very easily set dynamic pattern style filenames. If an expression is set it take precedes over the {{org.apache.camel.file.name}} header. (Note: The header can itself also be an expression). The expression options supports both String and Expression types. If the expression is a String type then its *always* evaluated using the [File Language]. If the expression is an Expression type then this type is of course used as it - this allows for instance to use [OGNL] as expression too. |
| passiveMode | false | Camel 1.5.1/2.0: Set whether to use passive mode connections. Default is active. This feature is only for regular FTP, not SFTP. |
| knownHosts | null | Camel 1.5.1/2.0: Sets the known_hosts file so that the SFTP endpoint can do host key verification. |
| privateKeyFile | null | Camel 1.5.1/2.0: Set the private key file to that the SFTP endpoint can do private key verification. |
| privateKeyFilePassphrase | null | Camel 1.5.1/2.0: Set the private key file passphrase to that the SFTP endpoint can do private key verification. |
{div}

h3. Camel 2.x URI Options
{div:class=confluenceTableSmall}
|| Name || Default Value || Description ||
| delay | 500 | Delay in millis between each poll |
| initialDelay | 1000 | Millis before polling starts |
| userFixedDelay | false | *true* to use fixed delay between pools, otherwise fixed rate is used. See [ScheduledExecutorService|http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ScheduledExecutorService.html] in JDK for details. |
| noop | false | If true then the file is not moved or deleted in any way (see below). This option is good for read only data, or for ETL type requirements. |
| directory | true| indicates whether or not the given file name should be interpreted by default as a directory or file (as it sometimes hard to be sure with some FTP servers) |
| password | null | specifies the password to use to login to the remote file system |
| binary | false | specifies the file transfer mode BINARY or ASCII. Default is ASCII. |
| passiveMode | false | Set whether to use passive mode connections. Default is active. This feature is only for regular FTP, not SFTP. |
| ftpClientConfig | null | Reference to a bean in the registry as a {{[org.apache.commons.net.ftp.FTPClientConfig|http://commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html]}} class. Use this option if you need to configure the client according to the FTP Server date format, locale, timezone, platform etc. See the javadoc {{[FTPClientConfig|http://commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html]}} for more documentation.
| recursive | false | if a directory, will look for changes in files in all the sub directories. |
| setNames | true  | Used by FTPConsumer. If true Camel will use the filename the file has on the FTP server. The filename is stored on the IN message in the header {{FileComponent.HEADER_FILE_NAME}}. |
| regexPattern | null | Used by FTPConsumer. Regular expression to use for matching files when consuming. |
| delete | false | Used by FTPConsumer. Flag to set if the consumed file should be deleted after it has been downloaded. |
| moveNamePrefix | null | Used by FTPConsumer. The prefix String perpended to the filename when moving it. For example to move processed files into the _done_ directory, set this value to 'done/' |
| moveNamePostfix | null | Used by FTPConsumer. The postfix String appended to the filename when moving it. For example to rename processed files from _foo_ to _foo.old_ set this value to '.old' |
| preMoveNamePrefix | null | Used by FTPConsumer. The prefix String perpended to the filename when moving it *before* processing. For example to move processed files into the _done_ directory, set this value to 'done/' |
| preMoveNamePostfix | null | Used by FTPConsumer. The postfix String appended to the filename when moving it *before* processing. For example to rename processed files from _foo_ to _foo.old_ set this value to '.old' |
| preMoveExpression | null | Used by FTPConsumer. Use expression to dynamically set the filename when moving it *before* processing. For example to move in progress file into the order directory and use {{.bak}} as extension set this value to {{'order/$\{file:name.noext}.bak'}} |
| exclusiveReadLockStrategy | null | Pluggable read lock as a {{org.apache.camel.component.file.remote.RemoteFileExclusiveReadLockStrategy}} implementation. |
| excludedNamePrefix | null | Used by FTPConsumer. Is used to exclude files if filename is starting with the given prefix. |
| excludedNamePostfix | null | Used by FTPConsumer. Is used to exclude files if filename is ending with the given postfix. |
| expression | null | Use expression to dynamically set the filename. This allows you to very easily set dynamic pattern style filenames. If an expression is set it take precedes over the {{org.apache.camel.file.name}} header. (Note: The header can itself also be an expression). The expression options supports both String and Expression types. If the expression is a String type then its *always* evaluated using the [File Language]. If the expression is an Expression type then this type is of course used as it - this allows for instance to use [OGNL] as expression too. |
| tempPrefix | null | Option for ftp/sftp producer only. This option is used to upload the file using a temporary name, and then after the upload is complete rename it to the real name. Can be used to identify files being uploaded and also avoid consumers (not using exclusive read locks) reading in progress files. |
| idempotent | false | Option to use the [Idempotent Consumer] EIP pattern to let Camel skip already processed files. Will default use a memory based LRUCache that holds 1000 entries. If noop=true then idempotent will be enabled as well to avoid consuming the same files over and over again. |
| idempotentRepository | null | Pluggable repository as a [org.apache.camel.processor.idempotent.MessageIdRepository|http://camel.apache.org/maven/camel-core/apidocs/org/apache/camel/processor/idempotent/MessageIdRepository.html] class. Will default use MemoryMessageIdRepository if none is specified and idempotent is true. |
| filter | null | Used by FTPConsumer. Pluggable filter as a {{org.apache.camel.component.file.remote.RemoteFileFilter}} class. Will skip files if filter returns false in its accept method. Camel ships with an ANT path matcher filter in the camel-spring component. See below for sample. |
| sorter | null | Pluggable sorter as a {{java.util.Comparator<RemoteFile>}} class. |
| sortBy | null | Build in sort by using the [File Language]. Supports nested sorts so you can have a sort by file name and as a 2nd group sort by modified date. See [File] and the sorting section for details. |
| readLock | none | Used by FTPConsumer, to only poll the files if it has exclusive read lock to the file (= the file is not in progress of being written). Camel will wait until the file lock is granted. This option provides the build in strategies: *rename* and *none*. rename is for using a try to rename the file as a test if we can get exclusive read lock. none is for no read locks at all. |
| readLockTimeout | 0 | Optional timeout in millis for the read lock, if supported by the read lock. If the read lock could not be granted and the timeout triggered then Camel will skip the file. At next poll Camel will try the file again, and this time maybe the read lock could be granted. |
| knownHostsFile | null | Sets the known_hosts file so that the SFTP endpoint can do host key verification. |
| privateKeyFile | null | Set the private key file to that the SFTP endpoint can do private key verification. |
| privateKeyFilePassphrase | null | Set the private key file passphrase to that the SFTP endpoint can do private key verification. |
{div}

h3. New default behavior for FTP/SFTP-Consumers in Camel 1.5
The consumer will always skip any file which name starts with a dot, such as {{".", ".camel", ".m2" or ".groovy"}}. Only files (not directories) is matched for valid filename if options such as: {{consumer.regexPattern, consumer.excludeNamePrefix, consumer.excludeNamePostfix}} is used.

The consumer recursive option will be changed from *true* to *false* as the default value. We don't feel that Camel out-of-the-box should recursive poll.

The consumer will *not* use timestamp algorithm for determine if a remote file is a new file - see warning section above. To use the old behavior of Camel 1.4 or older you can use the option {{consumer.timestamp=true}}.

h3. Important changes in Camel 2.0

In Camel 2.0 we have re factored the file component with:
* total refactor to align it with the [File] component
* removed the @deprecated options and the confusing timestamp check.
* ftp consumer gathers the list of files as a batch list before processing
* build in idempotent, to skip already processed files
* pluggable filtering using org.apache.camel.component.file.remote.RemoteFileFilter
* ANT path matcher for configuring ANT path styles for includes/excludes
* advanced sorting with pluggable strategies and leveraging the [File Language] expression to easy setup sorting incl. sub groups.
* pluggable ExclusiveReadLockStrategy with build in strategy: rename.


h3. Exclusive Read Lock
The option *readLock* can be used to force Camel *not* to consume files that is currently in the progress of being written. However this option is default turned off, as it requires that the user has write access. There are other solutions to avoid consuming files that are currently being written over FTP, for instance you can write the a temporary destination and move the file after it has been written. 

h3. Message Headers

The following message headers can be used to affect the behavior of the component

|| Header || Description ||
| org.apache.camel.file.name | Specifies the output file name (relative to the endpoint directory) to be used for the output message when sending to the endpoint. If this is not present and no expression either then a generated message Id is used as filename instead.  |
| org.apache.camel.file.name.produced | New in Camel 1.5: The actual absolute filepath (path + name) for the output file that was written. This header is set by Camel and its purpose is providing end-users the name of the file that was written. |
| org.apache.camel.file.total | Camel 2.0: Current index out of total number of files being consumed in this batch. |
| org.apache.camel.file.index | Camel 2.0: Total number of files being consumed in this batch. |
| file.remote.host | The hostname of the remote server |
| file.remote.name | The name of the file consumed from the remote server |
| file.remote.fullName | The fullname of the file consumed from the remote server |

h3. Consumer properties

When using FTPConsumer (downloading files from a FTP Server) the consumer specific properties from the [File] component should be prefixed with "consumer.". For example the delay option from File Component should be specified as "consumer.delay=30000" in the URI. See the samples or some of the unit tests of this component.

h3. Filename Expression
In Camel 1.5 we have support for setting the filename using an expression. This can be set either using the *expression* option or as a string based [File Language] expression in the {{org.apache.camel.file.name}} header. See the [File Language] for some samples.

h3. Camel 1.x Known issues

See the timestamp warning.

When consuming files (downloading) you must use type conversation to either String or to InputStream for ASCII and BINARY file types. 
In Camel 1.4 this is fixed, as there are build in type converters for the ASCII and BINARY file types, meaning that you do not need the convertBodyTo expression.

In Camel 1.4 or below Camel FTPConsumer will poll files regardless if the file is currently being written. See the *consumer.exclusiveReadLock* option.

Also in Camel 1.3 since setNames is default *false* then you must explicitly set the filename using the setHeader expression when consuming from FTP directly to File.
The code below illustrates this:

{code}
private String ftpUrl = "ftp://camelrider@localhost:21/public/downloads?password=admin&binary=false";
private String fileUrl = "file:myfolder/?append=false&noop=true";

return new RouteBuilder() {
    public void configure() throws Exception {
        from(ftpUrl).setHeader(FileComponent.HEADER_FILE_NAME, constant("downloaded.txt")).convertBodyTo(String.class).to(fileUrl);
    }
};
{code}

Or you can set the option to *true* as illustrated below:

{code}
private String ftpUrl = "ftp://camelrider@localhost:21/public/downloads?password=admin&binary=false&consumer.setNames=true";
private String fileUrl = "file:myfolder/?append=false&noop=true";

return new RouteBuilder() {
    public void configure() throws Exception {
        from(ftpUrl).convertBodyTo(String.class).to(fileUrl);
    }
};
{code}

h3. Sample

In the sample below we setup Camel to download all the reports from the FTP server once every hour (60 min) as BINARY content and store it as files on the local file system.
{snippet:id=e1|lang=java|url=camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFtpToBinarySampleTest.java}

And the route using Spring DSL:
{code:xml}
  <route>
     <from uri="ftp://scott@localhost/public/reports?password=tiger&amp;binary=true&amp;consumer.delay=60000"/>
     <to uri="file://target/test-reports"/>
  </route>
{code}

h4. Using expression for filenames

In this sample we want to move consumed files to a backup folder using today's date as a sub foldername. Notice that the move happens on the remote FTP server. If you want to store the downloaded file on your local disk then route it to the [File] component as the sample above illustrates.
{code:java}
from(ftpUrl + "&expression=backup/${date:now:yyyyMMdd}/${file:name}").to("...");
{code}

See [File Language] for more samples.

the [File2] component page as well.
{tip}

h3. URI Options
The options below are exclusive for the [FTP2] component. 
{div:class=confluenceTableSmall}
|| Name || Default Value || Description ||
| password | null | specifies the password to use to login to the remote file system |
| binary | false | specifies the file transfer mode BINARY or ASCII. Default is ASCII. |
| passiveMode | false | Set whether to use passive mode connections. Default is active. This feature is only for regular FTP, not SFTP. |
| ftpClientConfig | null | *FTP only*: Reference to a bean in the registry as a {{[org.apache.commons.net.ftp.FTPClientConfig|http://commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html]}} class. Use this option if you need to configure the client according to the FTP Server date format, locale, timezone, platform etc. See the javadoc {{[FTPClientConfig|http://commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html]}} for more documentation.
| knownHostsFile | null | *SFTP only:* Sets the known_hosts file so that the SFTP endpoint can do host key verification. |
| privateKeyFile | null | *SFTP only:* Set the private key file to that the SFTP endpoint can do private key verification. |
| privateKeyFilePassphrase | null | *SFTP only:* Set the private key file passphrase to that the SFTP endpoint can do private key verification. |
{div}

Common options:
TODO: Copy from File2


h3. Exclusive Read Lock
The option *readLock* can be used to force Camel *not* to consume files that is currently in the progress of being written. However this option is default turned off, as it requires that the user has write access. There are other solutions to avoid consuming files that are currently being written over FTP, for instance you can write the a temporary destination and move the file after it has been written. 

h3. Message Headers

The following message headers can be used to affect the behavior of the component

|| Header || Description ||
| CamelFileName | Specifies the output file name (relative to the endpoint directory) to be used for the output message when sending to the endpoint. If this is not present and no expression either then a generated message Id is used as filename instead.  |
| CamelFileNameProduced | The actual absolute filepath (path + name) for the output file that was written. This header is set by Camel and its purpose is providing end-users the name of the file that was written. |
| CamelFileBatchTotal | Current index out of total number of files being consumed in this batch. |
| CamelFileBatchIndex | Total number of files being consumed in this batch. |
| CamelFileHost | The remote hostname. |

h3. Sample

In the sample below we setup Camel to download all the reports from the FTP server once every hour (60 min) as BINARY content and store it as files on the local file system.
{snippet:id=e1|lang=java|url=camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFtpToBinarySampleTest.java}

And the route using Spring DSL:
{code:xml}
  <route>
     <from uri="ftp://scott@localhost/public/reports?password=tiger&amp;binary=true&amp;delay=60000"/>
     <to uri="file://target/test-reports"/>
  </route>
{code}

h4. Consuming a remote FTP server triggered by a route
The FTP consumer is build as a scheduled consumer to be used in the *from* route. However if you want to start consuming from a FTP server triggered within a route it's a bit cumbersome to do this in Camel 1.x (we plan to improve this in Camel 2.x). However it's possible as this code below demonstrates.

In the sample we have a [Seda] queue where a message arrives that holds a message containing a filename to poll from a remote FTP server. So we setup a basic FTP url as:
{snippet:id=e1|lang=java|url=camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromQueueThenConsumeFtpToMockTest.java}

And then we have the route where we use [Processor] within the route so we can use Java code. In this Java code we create the ftp consumer that downloads the file we want. And after the download we can get the content of the file and put it in the original exchange that continues being routed. As this is based on an unit test it routes to a [Mock] endpoint.
{snippet:id=e2|lang=java|url=camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromQueueThenConsumeFtpToMockTest.java}

h3. Filter using {{org.apache.camel.component.file.remote.RemoteFileFilter
*Available in Camel 2.0*

GenericFileFilter

Camel supports pluggable filtering strategies. This strategy it to use the build in {{org.apache.camel.component.file.remote.RemoteFileFilterGenericFileFilter}} in Java. You can then configure the endpoint with such a filter to skip certain filters before being processed.

In the sample we have build our own filter that only accepts files starting with report in the filename.
{snippet:id=e1|lang=java|url=camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFtpRemoteFileFilterTest.java}

And then we can configure our route using the *filter* attribute to reference our filter (using # notation) that we have defines in the spring XML file:
{code:xml}
   <!-- define our sorter as a plain spring bean -->
   <bean id="myFilter" class="com.mycompany.MyFileSorterMyFileFilter"/>

  <route>
    <from uri="ftp://someuser@someftpserver.com?password=secret&amp;filter=#myFilter"/>
    <to uri="bean:processInbox"/>
  </route>
{code}

h4. Filtering using ANT path matcher
*Available in Camel 2.0*

The ANT path matcher is a filter that is shipped out-of-the-box in the *camel-spring* jar. So you need to depend on *camel-spring* if you are using Maven.
The reasons is that we leverage Spring's [AntPathMatcher|http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/util/AntPathMatcher.html] to do the actual matching.

The file paths is matched with the following rules:
- {{?}} matches one character
- {{*}} matches zero or more characters
- {{**}} matches zero or more directories in a path

The sample below demonstrates how to use it:
{snippet:id=example|lang=xml|url=camel/trunk/tests/camel-itest/src/test/resources/org/apache/camel/itest/ftp/SpringFileAntPathMatcherRemoteFileFilterTest-context.xml}

h3. Debug logging
This component has log level *TRACE* that can be helpful if you have problems.

{include:Endpoint See Also}
- [File2]