Skip to content

Commit

Permalink
fix jolt metions (#305)
Browse files Browse the repository at this point in the history
#299



Also, I changed CodeBlock to support more than one line of code.
  • Loading branch information
fraidev authored Nov 23, 2024
1 parent b6db406 commit 02ffd3b
Show file tree
Hide file tree
Showing 63 changed files with 614 additions and 377 deletions.
5 changes: 5 additions & 0 deletions docs/_embeds/cmds/cdk-hub-download-http-source-output.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
trying connection to fluvio router.infinyon.cloud:9003
downloading infinyon/jolt@0.4.1 to infinyon-jolt-0.4.1.ipkg
... downloading complete
... checking package
... cluster smartmodule install complete
2 changes: 1 addition & 1 deletion docs/_embeds/cmds/cdk-hub-download-http-source.bash
Original file line number Diff line number Diff line change
@@ -1 +1 @@
cdk hub download infinyon/http-source@0.3.8
cdk hub download infinyon/http-source@0.3.8
5 changes: 5 additions & 0 deletions docs/_embeds/cmds/hub-sm-download-jolt-output.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
trying connection to fluvio router.infinyon.cloud:9003
downloading infinyon/jolt@0.4.1 to infinyon-jolt-0.4.1.ipkg
... downloading complete
... checking package
... cluster smartmodule install complete
29 changes: 11 additions & 18 deletions docs/cloud/tutorials/mqtt-to-sql.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -137,27 +137,20 @@ $ fluvio cloud connector create --config sql.yml

Download the [Jolt] and [Json-Sql] SmartModules used by this example connector

Example output

import CodeBlock from '@theme/CodeBlock';
import SmDownloadJolt from '!!raw-loader!../../_embeds/cmds/hub-sm-download-jolt.bash';
import SmDownloadJoltOutput from '!!raw-loader!../../_embeds/cmds/hub-sm-download-jolt-output.bash';
import SmDownloadJsonSql from '!!raw-loader!../../_embeds/cmds/hub-sm-download-json-sql.bash';
import SmDownloadJsonSqlOutput from '!!raw-loader!../../_embeds/cmds/hub-sm-download-json-sql-output.bash';

```bash
$ fluvio hub sm download infinyon/json-sql@x.y.z
downloading infinyon/json-sql@x.y.z to infinyon-json-sql-x.y.z.ipkg
... downloading complete
... checking package
trying connection to fluvio router.infinyon.cloud:9003
... cluster smartmodule install complete
```
<CodeBlock language="bash" copyBehavior="fl">$ {SmDownloadJsonSql} $ {SmDownloadJolt}</CodeBlock>

Expect the following output:

```bash
$ fluvio hub sm download infinyon/jolt@x.y.z
downloading infinyon/jolt@x.y.z to infinyon-jolt-x.y.z.ipkg
... downloading complete
... checking package
trying connection to fluvio router.infinyon.cloud:9003
... cluster smartmodule install complete
```
<CodeBlock language="bash">
{SmDownloadJsonSqlOutput}
{SmDownloadJoltOutput}
</CodeBlock>

##### Connector with JSON to JSON transformation config

Expand Down
10 changes: 5 additions & 5 deletions docs/fluvio/concepts/transformations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,22 @@ Example:

```yaml
transforms:
- uses: infinyon/jolt@0.3.0
- uses: infinyon/jolt@0.4.1
with:
spec:
- operation: default
spec:
source: "http"
```
we have one transformation, which is a SmartModule named [`infinyon/jolt@0.3.0`].
we have one transformation, which is a SmartModule named [`infinyon/jolt@0.4.1`].
The name must match a SmartModule previously downloaded to the Cluster:


```bash
fluvio sm list
SMARTMODULE SIZE
infinyon/jolt@0.3.0 608.4 KB
infinyon/jolt@0.4.1 608.4 KB
```


Expand Down Expand Up @@ -93,7 +93,7 @@ Multiple SmartModules can be activated in series. The output of the previous Sma
```yaml
transforms:
- uses: infinyon/jolt@0.3.0
- uses: infinyon/jolt@0.4.1
with:
spec:
- operation: shift
Expand All @@ -108,4 +108,4 @@ transforms:
In this example, the `jolt` transformation is performed first, then its output is the input the `regex-filter`.

[SmartModules]: smartmodules/overview.mdx
[`infinyon/jolt@0.3.0`]: hub/smartmodules/jolt.mdx
[`infinyon/jolt@0.4.1`]: hub/smartmodules/jolt.mdx
13 changes: 7 additions & 6 deletions docs/hub/smartmodules/jolt.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -52,22 +52,23 @@ The Jolt SmartModule applies the operations in sequence: `remove` followed by `s
To demonstrate how **Jolt** SmartModule works we will use [SMDK] tool.

:::info
All smartmodule versions are marked with `x.y.z`. To find the latest version, run:
All smartmodule versions are marked with `0.4.1`. To find the latest version, run:
* `fluvio hub smartmodule list`
:::

First, we need to download it to our cluster:

```bash copy="fl"
$ fluvio hub smartmodule download infinyon/jolt@x.y.z
```
import CodeBlock from '@theme/CodeBlock';
import SmDownloadJolt from '!!raw-loader!../../_embeds/cmds/hub-sm-download-jolt.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {SmDownloadJolt}</CodeBlock>

Second, we create a file `transform.yaml` with transformation specification defined above:

```yaml
# transform.yaml
transforms:
- uses: infinyon/jolt@x.y.z
- uses: infinyon/jolt@0.4.1
spec:
- operation: remove
spec:
Expand Down Expand Up @@ -113,4 +114,4 @@ For additional examples checkout the tutorials:

[Fluvio Jolt]: https://github.com/infinyon/fluvio-jolt
[map-type]:smartmodules/features/operators/map.mdx
[SMDK]: smartmodules/smdk.mdx
[SMDK]: smartmodules/smdk.mdx
11 changes: 10 additions & 1 deletion src/theme/CodeBlock/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,16 @@ function maybeStringifyChildren(children: ReactNode): ReactNode {
return children;
}
// The children is now guaranteed to be one/more plain strings
return Array.isArray(children) ? children.join('') : (children as string);

// If child is "$" or "$$" jump to the next line, if not add a space
return React.Children.toArray(children).map((child, index) => {
if (typeof child === 'string') {
if (index > 0) {
return child.replace(/\$\$?/g, (match) => (match === '$' ? '\n$' : ' '));
}
}
return child;
});
}

export default function CodeBlock({
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
cdk deploy start --ipkg infinyon-http-sink-0.2.11.ipkg --config <CONFIG>
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
cdk hub download infinyon/http-sink@0.2.11
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
trying connection to fluvio router.infinyon.cloud:9003
downloading infinyon/jolt@0.4.1 to infinyon-jolt-0.4.1.ipkg
... downloading complete
... checking package
... cluster smartmodule install complete
Original file line number Diff line number Diff line change
@@ -1 +1 @@
cdk hub download infinyon/http-source@0.3.8
cdk hub download infinyon/http-source@0.3.8
13 changes: 13 additions & 0 deletions versioned_docs/version-0.11.11/_embeds/cmds/hub-conn-list.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
fluvio hub connector list
CONNECTOR Visibility
infinyon-labs/graphite-sink@0.2.0 public
infinyon/duckdb-sink@0.1.0 public
infinyon/http-sink@0.2.11 public
infinyon/http-source@0.4.3 public
infinyon/ic-webhook-source@0.1.4 public
infinyon/inf-webhook-source@0.1.9 public
infinyon/kafka-sink@0.2.9 public
infinyon/kafka-source@0.2.7 public
infinyon/mqtt-source@0.2.9 public
infinyon/sql-sink@0.4.3 public
qdrant/qdrant-sink@0.1.0 public
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
trying connection to fluvio router.infinyon.cloud:9003
downloading infinyon/jolt@0.4.1 to infinyon-jolt-0.4.1.ipkg
... downloading complete
... checking package
... cluster smartmodule install complete
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
fluvio hub smartmodule download infinyon/jolt@0.4.1
... cluster smartmodule install complete
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
trying connection to fluvio router.infinyon.cloud:9003
downloading infinyon/json-sql@0.2.1 to infinyon-json-sql-0.2.1.ipkg
... downloading complete
... checking package
... cluster smartmodule install complete
15 changes: 15 additions & 0 deletions versioned_docs/version-0.11.11/_embeds/cmds/hub-sm-list.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
fluvio hub smartmodule list
SMARTMODULE Visibility
infinyon-labs/array-map-json@0.1.1 public
infinyon-labs/csv-json@0.2.0 public
infinyon-labs/dedup-filter@0.0.2 public
infinyon-labs/json-formatter@0.1.3 public
infinyon-labs/key-gen-json@0.1.1 public
infinyon-labs/regex-map-json@0.1.2 public
infinyon-labs/regex-map@0.1.1 public
infinyon-labs/rss-json@0.1.1 public
infinyon-labs/stars-forks-changes@0.1.4 public
infinyon/csv-json@0.2.0 public
infinyon/jolt@0.4.1 public
infinyon/json-sql@0.2.1 public
infinyon/regex-filter@0.2.0 public
31 changes: 12 additions & 19 deletions versioned_docs/version-0.11.11/cloud/tutorials/mqtt-to-sql.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ transforms:
json-key: "payload.device.device_id"
value:
type: "int"
default: "0"
defau1t: "0"
required: true
"record":
json-key: "$"
Expand All @@ -137,27 +137,20 @@ $ fluvio cloud connector create --config sql.yml

Download the [Jolt] and [Json-Sql] SmartModules used by this example connector

Example output
import CodeBlock from '@theme/CodeBlock';
import SmDownloadJolt from '!!raw-loader!../../_embeds/cmds/hub-sm-download-jolt.bash';
import SmDownloadJoltOutput from '!!raw-loader!../../_embeds/cmds/hub-sm-download-jolt-output.bash';
import SmDownloadJsonSql from '!!raw-loader!../../_embeds/cmds/hub-sm-download-json-sql.bash';
import SmDownloadJsonSqlOutput from '!!raw-loader!../../_embeds/cmds/hub-sm-download-json-sql-output.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {SmDownloadJsonSql} $ {SmDownloadJolt}</CodeBlock>

```bash
$ fluvio hub sm download infinyon/json-sql@x.y.z
downloading infinyon/json-sql@x.y.z to infinyon-json-sql-x.y.z.ipkg
... downloading complete
... checking package
trying connection to fluvio router.infinyon.cloud:9003
... cluster smartmodule install complete
```

Expect the following output:

```bash
$ fluvio hub sm download infinyon/jolt@x.y.z
downloading infinyon/jolt@x.y.z to infinyon-jolt-x.y.z.ipkg
... downloading complete
... checking package
trying connection to fluvio router.infinyon.cloud:9003
... cluster smartmodule install complete
```
<CodeBlock language="bash">
{SmDownloadJsonSqlOutput}
{SmDownloadJoltOutput}
</CodeBlock>

##### Connector with JSON to JSON transformation config

Expand Down
13 changes: 4 additions & 9 deletions versioned_docs/version-0.11.11/fluvio/cli/fluvio/hub/list.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ title: "list"
description: "Retrieves all SmartModules available for download in the Hub for a specific user"
---

import CodeBlock from '@theme/CodeBlock';
import HubSmListCmd from '!!raw-loader!../../../../_embeds/cmds/hub-sm-list.bash';

# The `fluvio hub list` command

List operation retrieves all SmartModules available for download in the SmartModule Hub for a specific user.
Expand All @@ -14,15 +17,7 @@ By default, users can retrieve all public SmartModules and all private SmartModu

Listing Hub SmartModule from the CLI is straight forward:

```bash copy="fl"
$ fluvio hub list

SMARTMODULE Visibility
infinyon/jolt@0.1.0 public
infinyon/json-sql@0.2.1 public
infinyon/regex-filter@0.1.0 public
...
```
<CodeBlock language="bash" copyBehavior="fl">$ {HubSmListCmd}</CodeBlock>

[`InfinyOn Cloud`]: https://infinyon.cloud/
[ Download Fluvio CLI ]: /cli
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,22 @@ Example:

```yaml
transforms:
- uses: infinyon/jolt@0.3.0
- uses: infinyon/jolt@0.4.1
with:
spec:
- operation: default
spec:
source: "http"
```
we have one transformation, which is a SmartModule named [`infinyon/jolt@0.3.0`].
we have one transformation, which is a SmartModule named [`infinyon/jolt@0.4.1`].
The name must match a SmartModule previously downloaded to the Cluster:


```bash
fluvio sm list
SMARTMODULE SIZE
infinyon/jolt@0.3.0 608.4 KB
infinyon/jolt@0.4.1 608.4 KB
```


Expand Down Expand Up @@ -93,7 +93,7 @@ Multiple SmartModules can be activated in series. The output of the previous Sma
```yaml
transforms:
- uses: infinyon/jolt@0.3.0
- uses: infinyon/jolt@0.4.1
with:
spec:
- operation: shift
Expand All @@ -108,4 +108,4 @@ transforms:
In this example, the `jolt` transformation is performed first, then its output is the input the `regex-filter`.

[SmartModules]: smartmodules/overview.mdx
[`infinyon/jolt@0.3.0`]: hub/smartmodules/jolt.mdx
[`infinyon/jolt@0.4.1`]: hub/smartmodules/jolt.mdx
15 changes: 8 additions & 7 deletions versioned_docs/version-0.11.11/hub/smartmodules/jolt.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -52,22 +52,23 @@ The Jolt SmartModule applies the operations in sequence: `remove` followed by `s
To demonstrate how **Jolt** SmartModule works we will use [SMDK] tool.

:::info
All smartmodule versions are marked with `x.y.z`. To find the latest version, run:
All smartmodule versions are marked with `0.4.1`. To find the latest version, run:
* `fluvio hub smartmodule list`
:::

First, we need to download it to our cluster:

```bash copy="fl"
$ fluvio hub smartmodule download infinyon/jolt@x.y.z
```
import CodeBlock from '@theme/CodeBlock';
import SmDownloadJolt from '!!raw-loader!../../_embeds/cmds/hub-sm-download-jolt.bash';

<CodeBlock language="bash" copyBehavior="fl">$ {SmDownloadJolt}</CodeBlock>

Second, we create a file `transform.yaml` with transformation specification defined above:

```yaml
# transform.yaml
transforms:
- uses: infinyon/jolt@x.y.z
- uses: infinyon/jolt@0.4.1
spec:
- operation: remove
spec:
Expand All @@ -86,7 +87,7 @@ Let's use `smdk test` to see it in action:

```bash copy="fl"
$ smdk test --text '{}' --transforms-file ./transform.yaml
1 records outputed
1 records outputted
{"data":{"source":"http-connector"}}
```

Expand All @@ -113,4 +114,4 @@ For additional examples checkout the tutorials:

[Fluvio Jolt]: https://github.com/infinyon/fluvio-jolt
[map-type]:smartmodules/features/operators/map.mdx
[SMDK]: smartmodules/smdk.mdx
[SMDK]: smartmodules/smdk.mdx
Loading

0 comments on commit 02ffd3b

Please sign in to comment.