Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PLUGIN-675] add hadoop compatible filesystem changes to sftp-put plugin #35

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

[PLUGIN-675] add hadoop compatible filesystem changes to sftp-put plugin #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions docs/SFTPCopy-action.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
SFTP Copy
=========

<a href="https://cdap-users.herokuapp.com/"><img alt="Join CDAP community" src="https://cdap-users.herokuapp.com/badge.svg?t=sftp-actions"/></a>
[![Build Status](https://travis-ci.org/hydrator/sftp-actions.svg?branch=develop)](https://travis-ci.org/hydrator/sftp-actions) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) <img src="https://cdap-users.herokuapp.com/assets/cdap-action.svg"/>


SFTP copy allows copying of the files from the specified directory on SFTP servers and write them to HDFS as the destination.
The files that are copied can be optionally uncompressed before storing. The files are copied directly to HDFS without needing any additional staging area.

Expand Down
4 changes: 0 additions & 4 deletions docs/SFTPDelete-action.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
SFTP Delete
===========

<a href="https://cdap-users.herokuapp.com/"><img alt="Join CDAP community" src="https://cdap-users.herokuapp.com/badge.svg?t=sftp-actions"/></a>
[![Build Status](https://travis-ci.org/hydrator/sftp-actions.svg?branch=develop)](https://travis-ci.org/hydrator/sftp-actions) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) <img src="https://cdap-users.herokuapp.com/assets/cdap-action.svg"/>


CDAP Action plugin to delete the specified files from the SFTP server.


Expand Down
17 changes: 9 additions & 8 deletions docs/SFTPPut-action.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,22 @@
SFTP Put
========

<a href="https://cdap-users.herokuapp.com/"><img alt="Join CDAP community" src="https://cdap-users.herokuapp.com/badge.svg?t=sftp-actions"/></a>
[![Build Status](https://travis-ci.org/hydrator/sftp-actions.svg?branch=develop)](https://travis-ci.org/hydrator/sftp-actions) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) <img src="https://cdap-users.herokuapp.com/assets/cdap-action.svg"/>


SFTP put allows uploading of file(s) from filesystem (local/HDFS) to FTP server. File Regex can also be used
SFTP put allows uploading of file(s) from a hadoop-compatible filesystem (gcs/local/HDFS) to FTP server. File Regex can also be used
to filter only the files that are of interest.


Usage Notes
-----------
In order perform SFTP Put, we require host and port on which the SFTP server is running. SFTP implements secure file
transfer over SSH. Typically port number 22 is used for SFTP(which is also default port for SSH). We also require valid
credentials in the form of user name and password. Please make sure that you are able to SSH to the SFTP server using
credentials in the form of username and password. Please make sure that you are able to SSH to the SFTP server using
specified user and password. SSH connection to SFTP server can be customized by providing additional configurations
such as enable host key checking by setting configuration property 'StrictHostKeyChecking' to 'yes'. These additional
configurations can be specified using `Properties for SSH` section.

Directory/File on the Filesystem which needs to be copied can be specified using `Source Path` property. The specified
Directory/File on the hadoop-compatible filesystem which needs to be copied can be specified using `Source Path` property. The specified
path should exist. `Destination directory` is the absolute path of the directory on the FTP Server where the files
will be copied. If destination directory does not exists, then it will be created first.
will be copied. If destination directory does not exist, then it will be created first.

Plugin Configuration
--------------------
Expand All @@ -39,6 +35,11 @@ Plugin Configuration
| **Continue execution on error** | **N** | false | Boolean flag to determine whether to proceed with next files in case there is a failure in deletion of any particular file. |
| **Properties for SSH** | **N** | N/A | Specifies the properties that are used to configure SSH connection to the FTP server. For example to enable verbose logging add property 'LogLevel' with value 'VERBOSE'. To enable host key checking set 'StrictHostKeyChecking' to 'yes'. SSH can be configured with the properties described here 'https://linux.die.net/man/5/ssh_config'. |

# Example Directory Configuration Using GCS
**Source Directory:** gs://my-bucket/my-file.csv

**Destination Directory:** /my/sftp/directory


Build
-----
Expand Down
3 changes: 2 additions & 1 deletion src/main/java/io/cdap/plugin/SFTPPutAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@ public String getFileNameRegex() {
@Override
public void run(ActionContext context) throws Exception {
Path source = new Path(config.getSrcPath());
FileSystem fileSystem = FileSystem.get(new Configuration());
Configuration conf = new Configuration();
FileSystem fileSystem = source.getFileSystem(conf);
if (!fileSystem.exists(source)) {
throw new RuntimeException(String.format("Source Path doesn't exist at %s", source));
}
Expand Down