Skip to content

Commit

Permalink
chore: updates for new release
Browse files Browse the repository at this point in the history
  • Loading branch information
@morenol
morenol committed Oct 28, 2024
1 parent 0bf5ed9 commit 0077176
Show file tree
Hide file tree
Showing 3 changed files with 10 additions and 302 deletions.
295 changes: 4 additions & 291 deletions sdf/composition/overview.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,166 +91,7 @@ The sections are as follows:
* `functions` - defines functions in the package
* `dev` - defines substitutions used during development and test


#### Build and Test a Package

We'll use the [SDF] command line tool with the package definition file to generate the WebAssembly glue code and the placeholder for the custom logic.

```bash copy="fl"
$ sdf -h
Stateful Dataflow Command Line Interface

Usage: sdf <COMMAND>

Commands:
clean Clean generated artifacts such as state and internal objects
build Build package (requires: sdf-package.yaml)
generate Generate package (requires: sdf-package.yaml)
update Update package (requires: sdf-package.yaml)
test Test command shell (requires sdf-package.yaml)
run Run dataflow (requires: dataflow.yaml)
setup Setup pre-requisites for Stateful Dataflows
version Prints binary version
log Print dataflow logs
```

Let's start with an example. Create a package that split sentences into words and counts the number of characters in each word.

##### 1. Create the Package file

Create a fresh project directory `split-sentence` with two subdirectories: `packages` and `sentence`:

```bash
$ mkdir -p split-sentence/packages/sentence
$ cd split-sentence/packages/sentence
```

Inside the `sentence` directory and create the `sdf-package.yaml` and add the following content:

```yaml
#sdf-package.yaml
apiVersion: 0.5.0

meta:
name: sentence-pkg
version: 0.1.0
namespace: example

functions:

sentence-to-words:
operator: flat-map
inputs:
- name: sentence
type: string
output:
type: string

augment-count:
operator: map
inputs:
- name: word
type: string
output:
type: string

dev:
converter: raw
```
##### 2. Generate the Package Project
Use SDF generate command to build the project:
```bash copy="fl"
$ sdf generate
```

The generator created several directories and files that we'll edit next.


##### 3. Add the Custom Code

First, let's update the first function, `sentence-to-words`. Open `rust/sentence-pkg/src/sentence_to_words.rs` and update the function body with the following code:

```rust
pub(crate) fn sentence_to_words(sentence: String) -> Result<Vec<String>> {
Ok(sentence.split_whitespace().map(String::from).collect())
}
```

Next update `augment_count`. Open `rust/sentence-pkg/src/augment_count.rs` and replace the function body:

```rust
pub(crate) fn augment_count(word: String) -> Result<String> {
Ok(format!("{}({})", word, word.chars().count()))
}
```

Let's add some tests as well:

```rust
#[cfg(test)]
mod test {
use super::*;

#[test]
fn test_augment_count() {
let input = "Hello".to_string();
let output = augment_count(input);
assert_eq!(output.unwrap(), "Hello(5)");
}
}
```

We've implemented both functions; it's time to compile and test our work.


##### 4. Build and Test the Package

To build the package, run:

```bash
$ sdf build
```

Use `sdf test` interactive shell to test the code:

```bash
$ sdf test
```

In the test shell, you can view the functions available for testing:

```bash copy="fl"
>> show functions
sentence-to-words
augment-count
```

Let's test `sentence-to-words` first:

```bash copy="fl"
>> test function sentence-to-words --value "Hello World"
Hello
World
```

Next, test `augment-count`:

```bash copy="fl"
>> test function augment-count --value "Hello"
Hello(5)
```

You may also test the `rust` code via Cargo:

```bash
$ cd rust/sentence-pkg
$ cargo test
```

The tests passed, and the package is now ready to use in the dataflow file.
# Dataflows

## The Dataflow

Expand Down Expand Up @@ -326,144 +167,16 @@ The sections are as follows:

The dataflow file has other variants such as `window` and `partitions`, which are omitted for simplicity. For additional details check the [Dataflow file] section.

# Getting started

#### Create a Dataflow

We'll import the package built in the [previous](#build-and-test-a-package) section to create the `split-sentence` dataflow.

##### 1. Create the Dataflow file

Create a dataflow file in the directory `split-sentence` directory:

```bash
$ cd split-sentence
```

Create the `dataflow.yaml` and add the following content:

```yaml name="dataflow.yaml"
apiVersion: 0.5.0
meta:
name: split-sentence
version: 0.1.0
namespace: example
imports:
- pkg: example/sentence-pkg@0.1.0
functions:
- name: sentence-to-words
- name: augment-count
config:
converter: raw
topics:
sentence:
schema:
value:
type: string
converter: raw
words:
schema:
value:
type: string
converter: raw
services:
sentence-words:
sources:
- type: topic
id: sentence
transforms:
- operator: flat-map
uses: sentence-to-words
- operator: map
uses: augment-count
sinks:
- type: topic
id: words
dev:
imports:
- pkg: example/sentence-pkg@0.1.0
path: ./packages/sentence
```

##### 2. Run the Dataflow
See [next section] for a quickstart.

Use `sdf` command line tool to run the dataflow:

```bash copy="fl"
$ sdf run --dev
```

Use `--dev` to ask the engine to change the path to the local package. Without this flag, the engine will look for the package in `InfinyOn Hub`.


##### 3. Test the Dataflow

1. Produce sentences to in `sentence` topic:

```bash copy="fl"
$ fluvio produce sentence
```

```bash
Hello world
Hi there
```

Consume from `words` to retrieve the result:

```bash
$ fluvio consume words -Bd
```

```bash
Hello(5)
world(5)
Hi(2)
there(5)
```

##### 4. Show State

The dataflow collects runtime metrics that you can inspect in the runtime terminal.

Check the `sentence-to-words` counters:

```bash copy="fl"
>> show state sentence-words/sentence-to-words/metrics
Key Window succeeded failed
stats * 2 0
```

Check the `augment-count` counters:

```bash copy="fl"
>> show state sentence-words/augment-count/metrics
Key Window succeeded failed
stats * 4 0
```

Congratulations! You've successfully built and run a composable dataflow! The project is available for download in [github].

##### 5. Clean-up

Exit the `sdf` terminal and remove the topics:

```bash
fluvio topic delete sentence
fluvio topic delete words
```

### References

* [Example Workflows in github]

[next section]: quickstart.mdx
[Inline Dataflows]: sdf/quickstart.mdx
[Dataflow file]: sdf/concepts/dataflow-yaml.mdx
[SDF]: sdf/cli/index.mdx
Expand Down
13 changes: 4 additions & 9 deletions sdf/composition/quickstart.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,6 @@ sidebar_position: 20
This guide will get you started with [SDF], an utility that helps developers build, troubleshoot, and run full-featured event-driven dataflows.


#### Overview

This [SDF] release includes several new features and improvements. The main feature is [composition], which allows you to create individual packages and import them to create dataflows.


#### Example Dataflow

As an example, we'll create a dataflow that splits sentences into words, and counts the number of characters in each word.
Expand Down Expand Up @@ -66,7 +61,7 @@ Your Fluvio cluster is ready for use.
SDF is in beta and it requires the following image:

```bash
fvm install sdf-beta2
fvm install sdf-beta3
```

You can validate prerequisites with:
Expand Down Expand Up @@ -145,15 +140,15 @@ The generator created several directories and files that we'll edit next.

#### 1.3 Add the Custom Code

First let's update the first function `sentence-to-words`. Open `rust/sentence-to-words/src/lib.rs` and update the function body with the following code:
First let's update the first function `sentence-to-words`. Open `rust/sentence-pkg/src/sentence_to_words.rs` and update the function body with the following code:

```rust
fn sentence_to_words(sentence: String) -> Result<Vec<String>> {
Ok(sentence.split_whitespace().map(String::from).collect())
}
```

Next update `augment_count`. Open `rust/augment-count/src/lib.rs` and replace the function body:
Next update `augment_count`. Open `rust/sentence-pkg/src/augment_count.rs` and replace the function body:

```rust
fn augment_count(word: String) -> Result<String> {
Expand Down Expand Up @@ -374,7 +369,7 @@ For additional examples, check out [stateful-dataflows-examples] in github. The
Stateful Dataflows is an ambitious project with many possibilities and just as many hazards. Please get in touch, we would love to hear your feedback and help us steer the product in the right direction.


[SDF]: /sdf
[SDF]: sdf
[composition]: overview.mdx
[stateful-dataflows-examples]: https://github.com/infinyon/stateful-dataflows-examples
[github]: https://github.com/infinyon/stateful-dataflows-examples/tree/main/dataflows-composed/split-sentence
Expand Down
4 changes: 2 additions & 2 deletions sdf/whatsnew.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,11 @@ We suggest to regenerate the packages again and copy the operator logic in the n
- Support connecting to other fluvio cluster topics via profile. See how to access external clusters in [topics configuration].

## Improvements
- Upgrade wasmtime to 26
- Support `sdf cleanup` in packages.
- Improvements on SDF packages.

## Bug Fixes
- Terminate daemon sdf if parent terminates
- Sdf dataflow properly terminates in host worker.

## InfinyOn Support

Expand Down

0 comments on commit 0077176

Please sign in to comment.