FTP/SFTP/FTPS Component
This component provides access to remote file systems over the FTP and SFTP protocols.
Maven users will need to add the following dependency to their pom.xml
for this component:
Code Block | ||||
---|---|---|---|---|
| ||||
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-ftp</artifactId> <version>x.x.x</version>See the documentation of the Apache Commons <!-- use the same version as your Camel core version --> </dependency> |
Info | ||
---|---|---|
| ||
See File for more options as all the options from File is inherited. |
Info | ||
---|---|---|
| ||
Absolute path is not supported. Camel 2.16 will translate absolute paths to relative ones by trimming all leading slashes from |
Tip | ||
---|---|---|
| ||
Make sure you read the section titled Default when consuming files further below for details related to consuming files. |
URI format
Code Block |
---|
ftp://[username@]hostname[:port]/directoryname[?options] sftp://[username@]hostname[:port]/directoryname[?options] ftps://[username@]hostname[:port]/directoryname[?options] |
Where directoryname represents the underlying directory. The directory name is a relative path. Absolute paths are not supported. The relative path can contain nested folders, such as /inbox/us.
For Camel versions before Camel 2.16, the directoryName must exist already as this component does not support the autoCreate
option (which the file component does). The reason is that its the FTP administrator (FTP server) task to properly setup user accounts, and home directories with the right file permissions etc.
For Camel 2.16, autoCreate
option is supported. When consumer starts, before polling is scheduled, there's additional FTP operation performed to create the directory configured for endpoint. The default value for autoCreate
is true
.
If no username is provided, then anonymous
login is attempted using no password.
If no port number is provided, Camel will provide default values according to the protocol (ftp = 21, sftp = 22, ftps = 2222).
You can append query options to the URI in the following format, ?option=value&option=value&...
This component uses two different libraries for the actual FTP work. FTP and FTPS uses Apache Commons Net while SFTP uses JCraft JSCH.
The FTPS component is only available in Camel 2.2 or newer.
FTPS (also known as FTP Secure) is an extension to FTP that adds support for the Transport Layer Security (TLS) and the Secure Sockets Layer (SSL) cryptographic protocols.
URI Options
The options below are exclusive for the FTP component.
Info | ||
---|---|---|
| ||
See File for more options as all the options from File is inherited. |
Div | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Info | ||
---|---|---|
| ||
When using the When using |
You can configure additional options on the ftpClient
and ftpClientConfig
from the URI directly by using the ftpClient.
or ftpClientConfig.
prefix.
For example to set the setDataTimeout
on the FTPClient
to 30 seconds you can do:
Code Block |
---|
from("ftp://foo@myserver?password=secret&ftpClient.dataTimeout=30000").to("bean:foo"); |
You can mix and match and have use both prefixes, for example to configure date format or timezones.
Code Block |
---|
from("ftp://foo@myserver?password=secret&ftpClient.dataTimeout=30000&ftpClientConfig.serverLanguageCode=fr").to("bean:foo"); |
You can have as many of these options as you like.
See the documentation of the Apache Commons FTP FTPClientConfig for possible options and more details. And as well for Apache Commons FTP FTPClient.
If you do not like having many and long configuration in the url you can refer to the ftpClient
or ftpClientConfig
to use by letting Camel lookup in the Registry for it.
For example:
Code Block |
---|
<bean id="myConfig" class="org.apache.commons.net.ftp.FTPClientConfig"> <property name="lenientFutureDates" value="true"/> <property name="serverLanguageCode" value="fr"/> </bean> |
And then let Camel lookup this bean when you use the # notation in the url.
Code Block |
---|
from("ftp://foo@myserver?password=secret&ftpClientConfig=#myConfig").to("bean:foo"); |
More URI options
Info | ||||
---|---|---|---|---|
| ||||
See File2 as all the options there also applies for this component. |
Examples
ftp://someone@someftpserver.com/public/upload/images/holiday2008?password=secret&binary=true
ftp://someoneelse@someotherftpserver.co.uk:12049/reports/2008/password=secret&binary=false
ftp://publicftpserver.com/download
Warning | ||
---|---|---|
| ||
The FTP consumer (with the same endpoint) does not support concurrency (the backing FTP client is not thread safe). The FTP producer does not have this issue, it supports concurrency. |
Tip | ||
---|---|---|
| ||
This component is an extension of the File component. So there are more samples and details on the File component page. |
Default when consuming files
The FTP consumer will by default leave the consumed files untouched on the remote FTP server. You have to configure it explicitly if you want it to delete the files or move them to another location. For example you can use delete=true
to delete the files, or use move=.done
to move the files into a hidden done sub directory.
The regular File consumer is different as it will by default move files to a .camel
sub directory. The reason Camel does not do this by default for the FTP consumer is that it may lack permissions by default to be able to move or delete files.
limitations
The option readLock can be used to force Camel not to consume files that are currently being written. However, this option is turned off by default, as it requires that the user has write access. See the options table at File2 for more details about read locks.
There are other solutions to avoid consuming files that are currently being written over FTP; for instance, you can write to a temporary destination and move the file after it has been written.
When moving files using move
or preMove
option the files are restricted to the FTP_ROOT folder. That prevents you from moving files outside the FTP area. If you want to move files to another area you can use soft links and move files into a soft linked folder.
Message Headers
The following message headers can be used to affect the behavior of the component
Div | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
|
In addition the FTP/FTPS consumer and producer will enrich the Camel Message
with the following headers
Div | ||||||
---|---|---|---|---|---|---|
| ||||||
|
About timeouts
The two set of libraries (see top) have different APIs for setting timeout. You can use the connectTimeout
option for both of them to set a timeout in millis to establish a network connection. An individual soTimeout
can also be set on the FTP/FTPS, which corresponds to using ftpClient.soTimeout
. Notice SFTP will automatically use connectTimeout
as its soTimeout
. The timeout
option only applies for FTP/FTSP as the data timeout, which corresponds to the ftpClient.dataTimeout
value. All timeout values are in millis.
Using Local Work Directory
Camel supports consuming from remote FTP servers and downloading the files directly into a local work directory. This avoids reading the entire remote file content into memory as it is streamed directly into the local file using FileOutputStream
.
Camel will store to a local file with the same name as the remote file, though with .inprogress
as extension while the file is being downloaded. Afterwards, the file is renamed to remove the .inprogress
suffix. And finally, when the Exchange is complete the local file is deleted.
So if you want to download files from a remote FTP server and store it as files then you need to route to a file endpoint such as:
Code Block | ||||
---|---|---|---|---|
| ||||
from("ftp://someone@someserver.com?password=secret&localWorkDirectory=/tmp").to("file://inbox"); |
Tip | ||
---|---|---|
| ||
The route above is ultra efficient as it avoids reading the entire file content into memory. It will download the remote file directly to a local file stream. The |
Stepwise changing directories
Camel FTP can operate in two modes in terms of traversing directories when consuming files (eg downloading) or producing files (eg uploading)
- stepwise
- not stepwise
You may want to pick either one depending on your situation and security issues. Some Camel end users can only download files if they use stepwise, while others can only download if they do not. At least you have the choice to pick (from Camel 2.6 onwards).
In Camel 2.0 - 2.5 there is only one mode and it is:
- before 2.5 not stepwise
- 2.5 stepwise
From Camel 2.6 onwards there is now an option stepwise
you can use to control the behavior.
Note that stepwise changing of directory will in most cases only work when the user is confined to it's home directory and when the home directory is reported as "/"
.
The difference between the two of them is best illustrated with an example. Suppose we have the following directory structure on the remote FTP server we need to traverse and download files:
Code Block |
---|
/ /one /one/two /one/two/sub-a /one/two/sub-b |
And that we have a file in each of sub-a (a.txt) and sub-b (b.txt) folder.
Using stepwise=true (default mode)
Code Block |
---|
TYPE A 200 Type set to A PWD 257 "/" is current directory. CWD one 250 CWD successful. "/one" is current directory. CWD two 250 CWD successful. "/one/two" is current directory. SYST 215 UNIX emulated by FileZilla PORT 127,0,0,1,17,94 200 Port command successful LIST 150 Opening data channel for directory list. 226 Transfer OK CWD sub-a 250 CWD successful. "/one/two/sub-a" is current directory. PORT 127,0,0,1,17,95 200 Port command successful LIST 150 Opening data channel for directory list. 226 Transfer OK CDUP 200 CDUP successful. "/one/two" is current directory. CWD sub-b 250 CWD successful. "/one/two/sub-b" is current directory. PORT 127,0,0,1,17,96 200 Port command successful LIST 150 Opening data channel for directory list. 226 Transfer OK CDUP 200 CDUP successful. "/one/two" is current directory. CWD / 250 CWD successful. "/" is current directory. PWD 257 "/" is current directory. CWD one 250 CWD successful. "/one" is current directory. CWD two 250 CWD successful. "/one/two" is current directory. PORT 127,0,0,1,17,97 200 Port command successful RETR foo.txt 150 Opening data channel for file transfer. 226 Transfer OK CWD / 250 CWD successful. "/" is current directory. PWD 257 "/" is current directory. CWD one 250 CWD successful. "/one" is current directory. CWD two 250 CWD successful. "/one/two" is current directory. CWD sub-a 250 CWD successful. "/one/two/sub-a" is current directory. PORT 127,0,0,1,17,98 200 Port command successful RETR a.txt 150 Opening data channel for file transfer. 226 Transfer OK CWD / 250 CWD successful. "/" is current directory. PWD 257 "/" is current directory. CWD one 250 CWD successful. "/one" is current directory. CWD two 250 CWD successful. "/one/two" is current directory. CWD sub-b 250 CWD successful. "/one/two/sub-b" is current directory. PORT 127,0,0,1,17,99 200 Port command successful RETR b.txt 150 Opening data channel for file transfer. 226 Transfer OK CWD / 250 CWD successful. "/" is current directory. QUIT 221 Goodbye disconnected. |
As you can see when stepwise is enabled, it will traverse the directory structure using CD xxx.
Using stepwise=false
Code Block |
---|
230 Logged on TYPE A 200 Type set to A SYST 215 UNIX emulated by FileZilla PORT 127,0,0,1,4,122 200 Port command successful LIST one/two 150 Opening data channel for directory list 226 Transfer OK PORT 127,0,0,1,4,123 200 Port command successful LIST one/two/sub-a 150 Opening data channel for directory list 226 Transfer OK PORT 127,0,0,1,4,124 200 Port command successful LIST one/two/sub-b 150 Opening data channel for directory list 226 Transfer OK PORT 127,0,0,1,4,125 200 Port command successful RETR one/two/foo.txt 150 Opening data channel for file transfer. 226 Transfer OK PORT 127,0,0,1,4,126 200 Port command successful RETR one/two/sub-a/a.txt 150 Opening data channel for file transfer. 226 Transfer OK PORT 127,0,0,1,4,127 200 Port command successful RETR one/two/sub-b/b.txt 150 Opening data channel for file transfer. 226 Transfer OK QUIT 221 Goodbye disconnected. |
As you can see when not using stepwise, there are no CD operation invoked at all.
Samples
In the sample below we set up Camel to download all the reports from the FTP server once every hour (60 min) as BINARY content and store it as files on the local file system.
Wiki Markup |
---|
{snippet:id=e1|lang=java|url=camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFtpToBinarySampleTest.java} |
Code Block | ||||
---|---|---|---|---|
| ||||
<route> <from uri="ftp://scott@localhost/public/reports?password=tiger&binary=true&delay=60000"/> <to uri="file://target/test-reports"/> </route> |
Consuming a remote FTPS server (implicit SSL) and client authentication
Code Block |
---|
from("ftps://admin@localhost:2222/public/camel?password=admin&securityProtocol=SSL&isImplicit=true &ftpClient.keyStore.file=./src/test/resources/server.jks &ftpClient.keyStore.password=password&ftpClient.keyStore.keyPassword=password") .to("bean:foo"); |
Consuming a remote FTPS server (explicit TLS) and a custom trust store configuration
Code Block |
---|
from("ftps://admin@localhost:2222/public/camel?password=admin&ftpClient.trustStore.file=./src/test/resources/server.jks&ftpClient.trustStore.password=password") .to("bean:foo"); |
Filter using org.apache.camel.component.file.GenericFileFilter
Camel supports pluggable filtering strategies. This strategy can be provided by implementing org.apache.camel.component.file.GenericFileFilter
in Java. You can then configure the endpoint with such a filter to skip certain filters before being processed.
In the sample we have built our own filter that only accepts files starting with report in the filename.
Wiki Markup |
---|
{snippet:id=e1|lang=java|url=camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFtpRemoteFileFilterTest.java} |
#
notation) that we have defined in the spring XML file:Code Block | ||||
---|---|---|---|---|
| ||||
<!-- define our sorter as a plain spring bean --> <bean id="myFilter" class="com.mycompany.MyFileFilter"/> <route> <from uri="ftp://someuser@someftpserver.com?password=secret&filter=#myFilter"/> <to uri="bean:processInbox"/> </route> |
Filtering using ANT path matcher
The ANT path matcher is a filter that is shipped out-of-the-box in the camel-spring jar. So you need to depend on camel-spring if you are using Maven.
The reason is that we leverage Spring's AntPathMatcher to do the actual matching.
The file paths are matched with the following rules:
?
matches one character*
matches zero or more characters**
matches zero or more directories in a path
The sample below demonstrates how to use it:
Wiki Markup |
---|
{snippet:id=example|lang=xml|url=camel/trunk/tests/camel-itest/src/test/resources/org/apache/camel/itest/ftp/SpringFileAntPathMatcherRemoteFileFilterTest-context.xml} |
Using a proxy with SFTP
To use an HTTP proxy to connect to your remote host, you can configure your route in the following way:
Code Block | ||||
---|---|---|---|---|
| ||||
<!-- define our sorter as a plain spring bean --> <bean id="proxy" class="com.jcraft.jsch.ProxyHTTP"> <constructor-arg value="localhost"/> <constructor-arg value="7777"/> </bean> <route> <from uri="sftp://localhost:9999/root?username=admin&password=admin&proxy=#proxy"/> <to uri="bean:processFile"/> </route> |
You can also assign a user name and password to the proxy, if necessary. Please consult the documentation for com.jcraft.jsch.Proxy
to discover all options.
Setting preferred SFTP authentication method
If you want to explicitly specify the list of authentication methods that should be used by sftp
component, use preferredAuthentications
option. If for example you would like Camel to attempt to authenticate with private/public SSH key and fallback to user/password authentication in the case when no public key is available, use the following route configuration:
Code Block |
---|
from("sftp://localhost:9999/root?username=admin&password=admin&preferredAuthentications=publickey,password"). to("bean:processFile"); |
Consuming a single file using a fixed name
When you want to download a single file and know the file name, you can use fileName=myFileName.txt
to tell Camel the name of the file to download. By default the consumer will still do a FTP LIST command to do a directory listing and then filter these files based on the fileName
option. Though in this use-case it may be desirable to turn off the directory listing by setting useList=false
. For example the user account used to login to the FTP server may not have permission to do a FTP LIST command. So you can turn off this with useList=false
, and then provide the fixed name of the file to download with fileName=myFileName.txt
, then the FTP consumer can still download the file. If the file for some reason does not exist, then Camel will by default throw an exception, you can turn this off and ignore this by setting ignoreFileNotFoundOrPermissionError=true
.
For example to have a Camel route that picks up a single file, and deletes it after use you can write
Code Block |
---|
from("ftp://admin@localhost:21/nolist/?password=admin&stepwise=false&useList=false&ignoreFileNotFoundOrPermissionError=true&fileName=report.txt&delete=true") .to("activemq:queue:report"); |
Notice that we have used all the options we talked above.
You can also use this with ConsumerTemplate
. For example to download a single file (if it exists) and grab the file content as a String type:
Code Block |
---|
String data = template.retrieveBodyNoWait("ftp://admin@localhost:21/nolist/?password=admin&stepwise=false&useList=false&ignoreFileNotFoundOrPermissionError=true&fileName=report.txt&delete=true", String.class); |
Debug logging
This component has log level TRACE that can be helpful if you have problems.
Include Page | ||||
---|---|---|---|---|
|