Snowflake Ingest Service Java SDK

The Snowflake Ingest Service SDK allows users to ingest files into their Snowflake data warehouse in a programmatic fashion via key-pair authentication. Currently, we support ingestion through the following APIs:

1. Snowpipe
2. Snowpipe Streaming - Under Private Preview

Prerequisites

Java 8+

The Snowflake Ingest Service SDK can only be used with Java 8 or higher. Backwards compatibility with Java 7 and prior is not planned at this time.

A 2048-bit RSA key pair

Snowflake Authentication for the Ingest Service requires creating a 2048 bit RSA key pair and, registering the public key with Snowflake. For detailed instructions, please visit the relevant Snowflake Documentation Page.

Maven (Developers only)

This SDK is developed as a Maven project. As a result, you'll need to install Maven to build the projects and, run tests.

Adding as a Dependency

You can add the Snowflake Ingest Service SDK by adding the following to your project

<!-- Add this to your Maven project's pom.xml -->
<dependency>
    <groupId>net.snowflake</groupId>
    <artifactId>snowflake-ingest-sdk</artifactId>
    <version>{version}</version>
</dependency>

// in Gradle project
dependencies {
    compile 'net.snowflake:snowflake-ingest-sdk:{version}'
}

Example

Snowpipe

Check out SnowflakeIngestBasicExample.java

Snowpipe Streaming

Check out SnowflakeStreamingIngestExample.java, which performs following operations:

1. Reads a JSON file which contains details regarding Snowflake Account, User, Role and Private Key. Take a look at profile_streaming.json.example for more details.
   1. Here are the steps required to generate a private key.
2. Creates a SnowflakeStreamingIngestClient which can be used to open one or more Streaming Channels pointing to the same or different tables.
3. Creates a SnowflakeStreamingIngestChannel against a Database, Schema and Table.
   1. Please note: The database, schema and table is expected to be present before opening the Channel. Example SQL queries to create them:

create or replace database MY_DATABASE;
create or replace schema MY_SCHEMA;
create or replace table MY_TABLE(c1 number);

4. Inserts 1000 rows into the channel created in 3rd step using the insertRows API on the Channel object
   1. insertRows API also takes in an optional offsetToken String which can be associated to this batch of rows.
5. Calls getLatestCommittedOffsetToken on the channel until the appropriate offset is found in Snowflake.
6. Close the channel when the ingestion is done to make sure everything is committed.

Building From Source

If you would like to build this project from source you can run the following to install the artifact to your local maven repository.

mvn install

If you would just like to build the jar in the source directory, you can run

mvn package

However, for general usage, pulling a pre-built jar from maven is recommended.

If you would like to run SnowflakeIngestBasicExample.java or SnowflakeStreamingIngestExample.java in the example folder, you would need to remove the following scope limits in pom.xml

<!-- Remove provided scope from slf4j-api -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <scope>provided</scope>
</dependency>

Testing (SimpleIngestIT Test)

• Modify TestUtils.java file and replace PROFILE_PATH with profile.json.example for testing.

  • profile.json is used because an encrypted file will be decrypted for Github Actions testing. Check End2EndTest.yml

• Use an unencrypted version(Only for testing) of private key while generating keys(private and public pair) using OpenSSL.

  • Here is the link for documentation Key Pair Generator

Code style

We use Google Java format to format the code. To format all files, run:

./format.sh