Skip to content

Commit

Permalink
Fix embeds with relative path to honor versioning. (#212)
Browse files Browse the repository at this point in the history
  • Loading branch information
ajhunyady authored Aug 21, 2024
1 parent 4e8a402 commit 98fae6d
Show file tree
Hide file tree
Showing 36 changed files with 165 additions and 154 deletions.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

apiVersion: 0.1.0
meta:
version: 0.3.8
Expand All @@ -9,4 +8,4 @@ meta:

http:
endpoint: "https://catfact.ninja/fact"
interval: 10s
interval: 10s
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ meta:
name: cat-facts-transformed
type: http-source
topic: cat-facts-data-transform
create-topic: true

http:
endpoint: https://catfact.ninja/fact
Expand Down
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion docs/connectors/how-to/run-local.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ sidebar_position: 10
title: Run Local
description: "Run a Connector on your local machine"
---
import Versions from '@site/embeds/versions';
import Versions from '../../_embeds/versions';

# Run Connectors on Your Local Machine

Expand Down
20 changes: 5 additions & 15 deletions docs/fluvio/tutorials/connector-transformations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ title: "Connector Transformations"
description: "Part 2 of 3: HTTP to SQL Sink tutorial series."
---
import CodeBlock from '@theme/CodeBlock';
import HubSmDownload from '!!raw-loader!../../_embeds/cmds/hub-sm-download-jolt.bash';
import HubSmList from '!!raw-loader!../../_embeds/cmds/sm-list-jolt.bash';
import CatFactBasic from '!!raw-loader!../../_embeds/connectors/http-cat-fact-transform.yaml';
import CdkDeployHttpSrc from '!!raw-loader!../../_embeds/cmds/cdk-deploy-start-http-source-cat-facts-xform.bash';


# Connector Transformations

Expand Down Expand Up @@ -38,17 +43,10 @@ The SmartModule we will use in this tutorial implements domain specific language

To use jolt SmartModule, we need to download it to the cluster.


import HubSmDownload from '!!raw-loader!@site/embeds/cmds/hub-sm-download-jolt.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {HubSmDownload}</CodeBlock>



Once you have the SmartModule downloaded, you can list them:

import HubSmList from '!!raw-loader!@site/embeds/cmds/sm-list-jolt.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {HubSmList}</CodeBlock>


Expand All @@ -60,22 +58,14 @@ Before we start modified connector, please terminate existing running connector.

Coyp and paste following config and save it as `http-cat-facts-transform.yaml`. This configuration use the same endpoint as the previous tutorial except different topic name.


import CatFactBasic from '!!raw-loader!@site/embeds/connectors/http-cat-fact-transform.yaml';

<CodeBlock language="yaml">{CatFactBasic}</CodeBlock>


Noticed that we just added `transforms` section which added JOLT specification to insert a new field `source` with value `http` to every record.

Once you have the config file, you can create the connector using the `cdk deploy start` command as before.


import CdkDeployHttpSrc from '!!raw-loader!@site/embeds/cmds/cdk-deploy-start-http-source-cat-facts-xform.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {CdkDeployHttpSrc}</CodeBlock>


You can use `cdk deploy list` to view the status of the connector.

```bash copy="fl"
Expand Down
12 changes: 5 additions & 7 deletions docs/fluvio/tutorials/http-source.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,11 @@ title: "HTTP Source -> Topic"
description: "Part 1 of 3: HTTP to SQL Sink tutorial series."
---
import CodeBlock from '@theme/CodeBlock';
import Versions from '@site/embeds/versions';
import Versions from '../../_embeds/versions';
import CatFactBasic from '!!raw-loader!../../_embeds/connectors/http-cat-fact-basic.yaml';
import CdkHubDownload from '!!raw-loader!../../_embeds/cmds/cdk-hub-download-http-source.bash';
import CdkDeployHttpSrc from '!!raw-loader!../../_embeds/cmds/cdk-deploy-start-http-source-cat-facts.bash';


# HTTP Source -> Topic

Expand Down Expand Up @@ -54,8 +58,6 @@ The connector you downloaded contains binary executable. The connector is packa

In this tutorial, we will be using the `HTTP Source Connector` from the Hub.

import CdkHubDownload from '!!raw-loader!@site/embeds/cmds/cdk-hub-download-http-source.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {CdkHubDownload}</CodeBlock>


Expand All @@ -69,8 +71,6 @@ After downloading, you need to create a configuration file to provision the conn
Coyp and paste following config and save it as `http-cat-facts.yaml`.


import CatFactBasic from '!!raw-loader!@site/embeds/connectors/http-cat-fact-basic.yaml';

<CodeBlock language="yaml">{CatFactBasic}</CodeBlock>


Expand All @@ -86,8 +86,6 @@ The `interval` is set to `10s` which means the connector will poll the endpoint
Once you have the config file, you can create the connector using the `cdk deploy start` command.


import CdkDeployHttpSrc from '!!raw-loader!@site/embeds/cmds/cdk-deploy-start-http-source-cat-facts.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {CdkDeployHttpSrc}</CodeBlock>


Expand Down
26 changes: 6 additions & 20 deletions docs/fluvio/tutorials/sql-sink.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ title: "Topic -> SQL Sink"
description: "Part 1 of 3: HTTP to SQL Sink tutorial series."
---
import CodeBlock from '@theme/CodeBlock';
import HubDownloadSqlSink from '!!raw-loader!../../_embeds/cmds/cdk-hub-download-sql-sink.bash';
import SmDownloadJsonSql from '!!raw-loader!../../_embeds/cmds/hub-sm-download-json-sql.bash';
import SmListJsonSqlJolt from '!!raw-loader!../../_embeds/cmds/sm-list-json-sql-jolt.bash';
import CatSQL from '!!raw-loader!../../_embeds/connectors/sql-cat-fact.yaml';
import CdkDeploySqlSink from '!!raw-loader!../../_embeds/cmds/cdk-deploy-start-sql-sink-cat-facts.bash';


# Topic -> SQL Sink

Expand Down Expand Up @@ -35,38 +41,22 @@ We will be using topics from first tutorial [Streaming from HTTP Source] which s

As in previous tutorials, we will use `cdk` to manage the connectors. Run following command to download the connector from the Hub.


import HubDownloadSqlSink from '!!raw-loader!@site/embeds/cmds/cdk-hub-download-sql-sink.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {HubDownloadSqlSink}</CodeBlock>


Then download SQL SmartModule from the Hub.


import SmDownloadJsonSql from '!!raw-loader!@site/embeds/cmds/hub-sm-download-json-sql.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {SmDownloadJsonSql}</CodeBlock>


Then you should see two smartmodules downloaded assuming you have already downloaded the `jolt` SmartModule from previous tutorial.


import SmListJsonSqlJolt from '!!raw-loader!@site/embeds/cmds/sm-list-json-sql-jolt.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {SmListJsonSqlJolt}</CodeBlock>


## Sink Connector configuration

Coyp and paste following config and save it as `sql-cat-fact.yaml`.


import CatSQL from '!!raw-loader!@site/embeds/connectors/sql-cat-fact.yaml';

<CodeBlock language="yaml">{CatSQL}</CodeBlock>


This configuration will read data from `cat-facts` topic and insert into `animalfacts` table in the database. The `json-sql` SmartModule will transform the JSON data into SQL insert statement.

Please change line containing `url` to your database connection string.
Expand Down Expand Up @@ -111,12 +101,8 @@ You can confirm table is created:

Once you have the config file, you can create the connector using the `cdk deploy start` command.


import CdkDeploySqlSink from '!!raw-loader!@site/embeds/cmds/cdk-deploy-start-sql-sink-cat-facts.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {CdkDeploySqlSink}</CodeBlock>


You can use `cdk deploy list` to view the status of the connector.

```bash copy="fl"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
cdk deploy start --ipkg infinyon-http-source-0.3.8.ipkg --config ./http-cat-facts-transform.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
cdk deploy start --ipkg infinyon-http-source-0.3.8.ipkg --config ./http-cat-facts.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
cdk deploy start --ipkg infinyon-sql-sink-0.4.3.ipkg --config ./sql-cat-fact.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
cdk hub download infinyon/http-source@0.3.8
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
cdk hub download infinyon/sql-sink@0.4.3
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
fluvio hub sm download infinyon/jolt@0.4.1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
fluvio hub sm download infinyon/json-sql@0.2.1
3 changes: 3 additions & 0 deletions versioned_docs/version-0.11.11/_embeds/cmds/sm-list-jolt.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
fluvio sm list
SMARTMODULE SIZE
infinyon/jolt@0.4.1 589.3 KB
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
fluvio sm list
SMARTMODULE SIZE
infinyon/json-sql@0.2.1 559.6 KB
infinyon/jolt@0.4.1 589.3 KB
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ meta:
name: cat-facts-transformed
type: http-source
topic: cat-facts-data-transform
create-topic: true

http:
endpoint: https://catfact.ninja/fact
Expand All @@ -16,4 +15,4 @@ transforms:
spec:
- operation: default
spec:
source: "http"
source: "http"
5 changes: 5 additions & 0 deletions versioned_docs/version-0.11.11/_embeds/versions.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
const versions = {
infinyon_http_source: 'infinyon-http-source@0.3.8',
}

export default versions;
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ sidebar_position: 10
title: Run Local
description: "Run a Connector on your local machine"
---

import Versions from '@site/embeds/versions';
import Versions from '../../_embeds/versions';

# Run Connectors on Your Local Machine

Expand Down
Loading

0 comments on commit 98fae6d

Please sign in to comment.