From 7471b92b631cff7c5eb7abe6cabde8c487cffc69 Mon Sep 17 00:00:00 2001 From: Justin Taras Date: Wed, 2 Aug 2023 22:33:54 -0400 Subject: [PATCH] add hadoop compatible changes to sftp-put plugin cleaned up some formatting in the docs --- docs/SFTPCopy-action.md | 4 ---- docs/SFTPDelete-action.md | 4 ---- docs/SFTPPut-action.md | 17 +++++++++-------- src/main/java/io/cdap/plugin/SFTPPutAction.java | 3 ++- 4 files changed, 11 insertions(+), 17 deletions(-) diff --git a/docs/SFTPCopy-action.md b/docs/SFTPCopy-action.md index a5c1744..8343d9f 100644 --- a/docs/SFTPCopy-action.md +++ b/docs/SFTPCopy-action.md @@ -1,10 +1,6 @@ SFTP Copy ========= -Join CDAP community -[![Build Status](https://travis-ci.org/hydrator/sftp-actions.svg?branch=develop)](https://travis-ci.org/hydrator/sftp-actions) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) - - SFTP copy allows copying of the files from the specified directory on SFTP servers and write them to HDFS as the destination. The files that are copied can be optionally uncompressed before storing. The files are copied directly to HDFS without needing any additional staging area. diff --git a/docs/SFTPDelete-action.md b/docs/SFTPDelete-action.md index dc9634b..708c985 100644 --- a/docs/SFTPDelete-action.md +++ b/docs/SFTPDelete-action.md @@ -1,10 +1,6 @@ SFTP Delete =========== -Join CDAP community -[![Build Status](https://travis-ci.org/hydrator/sftp-actions.svg?branch=develop)](https://travis-ci.org/hydrator/sftp-actions) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) - - CDAP Action plugin to delete the specified files from the SFTP server. diff --git a/docs/SFTPPut-action.md b/docs/SFTPPut-action.md index b960c51..449930c 100644 --- a/docs/SFTPPut-action.md +++ b/docs/SFTPPut-action.md @@ -1,11 +1,7 @@ SFTP Put ======== -Join CDAP community -[![Build Status](https://travis-ci.org/hydrator/sftp-actions.svg?branch=develop)](https://travis-ci.org/hydrator/sftp-actions) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) - - -SFTP put allows uploading of file(s) from filesystem (local/HDFS) to FTP server. File Regex can also be used +SFTP put allows uploading of file(s) from a hadoop-compatible filesystem (gcs/local/HDFS) to FTP server. File Regex can also be used to filter only the files that are of interest. @@ -13,14 +9,14 @@ Usage Notes ----------- In order perform SFTP Put, we require host and port on which the SFTP server is running. SFTP implements secure file transfer over SSH. Typically port number 22 is used for SFTP(which is also default port for SSH). We also require valid -credentials in the form of user name and password. Please make sure that you are able to SSH to the SFTP server using +credentials in the form of username and password. Please make sure that you are able to SSH to the SFTP server using specified user and password. SSH connection to SFTP server can be customized by providing additional configurations such as enable host key checking by setting configuration property 'StrictHostKeyChecking' to 'yes'. These additional configurations can be specified using `Properties for SSH` section. -Directory/File on the Filesystem which needs to be copied can be specified using `Source Path` property. The specified +Directory/File on the hadoop-compatible filesystem which needs to be copied can be specified using `Source Path` property. The specified path should exist. `Destination directory` is the absolute path of the directory on the FTP Server where the files -will be copied. If destination directory does not exists, then it will be created first. +will be copied. If destination directory does not exist, then it will be created first. Plugin Configuration -------------------- @@ -39,6 +35,11 @@ Plugin Configuration | **Continue execution on error** | **N** | false | Boolean flag to determine whether to proceed with next files in case there is a failure in deletion of any particular file. | | **Properties for SSH** | **N** | N/A | Specifies the properties that are used to configure SSH connection to the FTP server. For example to enable verbose logging add property 'LogLevel' with value 'VERBOSE'. To enable host key checking set 'StrictHostKeyChecking' to 'yes'. SSH can be configured with the properties described here 'https://linux.die.net/man/5/ssh_config'. | +# Example Directory Configuration Using GCS +**Source Directory:** gs://my-bucket/my-file.csv + +**Destination Directory:** /my/sftp/directory + Build ----- diff --git a/src/main/java/io/cdap/plugin/SFTPPutAction.java b/src/main/java/io/cdap/plugin/SFTPPutAction.java index 5afe1ba..3d06370 100644 --- a/src/main/java/io/cdap/plugin/SFTPPutAction.java +++ b/src/main/java/io/cdap/plugin/SFTPPutAction.java @@ -83,7 +83,8 @@ public String getFileNameRegex() { @Override public void run(ActionContext context) throws Exception { Path source = new Path(config.getSrcPath()); - FileSystem fileSystem = FileSystem.get(new Configuration()); + Configuration conf = new Configuration(); + FileSystem fileSystem = source.getFileSystem(conf); if (!fileSystem.exists(source)) { throw new RuntimeException(String.format("Source Path doesn't exist at %s", source)); }