Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement node role registration #314

Merged
merged 7 commits into from
Aug 16, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ Here you can learn all you need to know about BanyanDB.

- **Installation**. Instruments about how to download and onboard BanyanDB server, Banyand.
- **Clients**. Some native clients to access Banyand.
- **Schema**. Pivotal database native resources.
- **CRUD Operations**. To create, read, update, and delete data points or entities on resources in the schema.
- **Observability**. Learn how to effectively monitor, diagnose and optimize Banyand.
- **Concept**. Learn the concepts of Banyand. Includes the architecture, data model, and so on.
- **CRUD Operations**. To create, read, update, and delete data points or entities on resources in the schema.

You might also find these links interesting:

Expand Down
6 changes: 5 additions & 1 deletion docs/concept/clustering.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ In addition to persistent raw data, Data Nodes also handle TopN aggregation calc

### 1.2 Meta Nodes

Meta Nodes are responsible for maintaining high-level metadata of the cluster, which includes:
Meta Nodes is implemented by etcd. They are responsible for maintaining high-level metadata of the cluster, which includes:

- All nodes in the cluster
- All database schemas
Expand Down Expand Up @@ -52,6 +52,10 @@ All nodes within a BanyanDB cluster communicate with other nodes according to th
- Query Nodes interact with Data Nodes to execute queries and return results to the Liaison Nodes.
- Liaison Nodes distribute incoming requests to the appropriate Query Nodes or Data Nodes.

### Nodes Discovery

All nodes in the cluster are discovered by the Meta Nodes. When a node starts up, it registers itself with the Meta Nodes. The Meta Nodes then share this information with the Liaison Nodes and Query Nodes, which use it to route requests to the appropriate nodes.

## 3. **Data Organization**

Different nodes in BanyanDB are responsible for different parts of the database, while Query and Liaison Nodes manage the routing and processing of queries.
Expand Down
68 changes: 68 additions & 0 deletions docs/installation/binaries.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Get Binaries

This page shows how to get binaries of Banyand.

## Prebuilt Released binaries

Get binaries from the [download](https://skywalking.apache.org/downloads/).

## Build From Source

### Requirements

Users who want to build a binary from sources have to set up:

* Go 1.20
* Node 18.16
* Git >= 2.30
* Linux, macOS or Windows+WSL2
* GNU make

### Windows

BanyanDB is built on Linux and macOS that introduced several platform-specific characters to the building system. Therefore, we highly recommend you use [WSL2+Ubuntu](https://ubuntu.com/wsl) to execute tasks of the Makefile.

### Build Binaries

To issue the below command to get basic binaries of banyand and bydbctl.

```shell
$ make generate
...
$ make build
...
--- banyand: all ---
make[1]: Entering directory '<path_to_project_root>/banyand'
...
chmod +x build/bin/banyand-server
Done building banyand server
make[1]: Leaving directory '<path_to_project_root>/banyand'
...
--- bydbctl: all ---
make[1]: Entering directory '<path_to_project_root>/bydbctl'
...
chmod +x build/bin/bydbctl
Done building bydbctl
make[1]: Leaving directory '<path_to_project_root>/bydbctl'
```

The build system provides a series of binary options as well.

* `make -C banyand banyand-server` generates a basic `banyand-server`.
* `make -C banyand release` builds out a static binary for releasing.
* `make -C banyand debug` gives a binary for debugging without the complier's optimizations.
* `make -C banyand debug-static` is a static binary for debugging.
* `make -C bydbctl release` cross-builds several binaries for multi-platforms.

Then users get binaries as below

``` shell
$ ls banyand/build/bin
banyand-server
banyand-server-debug
banyand-server-debug-static
banyand-server-static

$ ls banyand/build/bin
bydbctl
```
45 changes: 45 additions & 0 deletions docs/installation/cluster.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Cluster Installation<TBD>

## Setup Meta Nodes

Meta nodes are a etcd cluster which is required for the metadata module to provide the metadata service and nodes discovery service for the whole cluster.

The etcd cluster can be setup by the [etcd installation guide](https://etcd.io/docs/v3.5/install/)

## Role-base Banyand Cluster

There is an example: The etcd cluster is spread across three nodes with the addresses `10.0.0.1:2379`, `10.0.0.2:2379`, and `10.0.0.3:2379`.

Data nodes, query nodes and liaison nodes are running as independent processes by

```shell
$ ./banyand-server storage --mode data --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
$ ./banyand-server storage --mode data --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
$ ./banyand-server storage --mode data --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
$ ./banyand-server storage --mode query --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
$ ./banyand-server storage --mode query --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
$ ./banyand-server liaison --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
```

The data node, query node and liaison node would be listening on the `<ports>` if no errors occurred.

## Mix-mode Data Nodes

If you want to use a `mix` mode instead of separate query and data nodes, you can run the banyand-server as processes by

```shell
$ ./banyand-server storage --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
$ ./banyand-server storage --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
$ ./banyand-server storage --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
$ ./banyand-server liaison --etcd-endpoints=http://10.0.0.1:2379,http://10.0.0.2:2379,http://10.0.0.3:2379 <flags>
```

## Node Discovery

The node discovery is based on the etcd cluster. The etcd cluster is required for the metadata module to provide the metadata service and nodes discovery service for the whole cluster.

The host is registered to the etcd cluster by the `banyand-server` automatically based on `node-host-provider` :
hanahmily marked this conversation as resolved.
Show resolved Hide resolved

- `node-host-provider=hostname` : The OS's hostname is registered as the host part in the address.
- `node-host-provider=ip` : The OS's the first public active IP address(IPv4) is registered as the host part in the address.
hanahmily marked this conversation as resolved.
Show resolved Hide resolved
- `node-host-provider=flag` : `node-host` is registered as the host part in the address.
21 changes: 21 additions & 0 deletions docs/installation/standalone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Standalone Mode

The standalone mode is the simplest way to run Banyand. It is suitable for the development and testing environment. The standalone mode is running as a standalone process by

```shell
$ ./banyand-server standalone
██████╗ █████╗ ███╗ ██╗██╗ ██╗ █████╗ ███╗ ██╗██████╗ ██████╗
██╔══██╗██╔══██╗████╗ ██║╚██╗ ██╔╝██╔══██╗████╗ ██║██╔══██╗██╔══██╗
██████╔╝███████║██╔██╗ ██║ ╚████╔╝ ███████║██╔██╗ ██║██║ ██║██████╔╝
██╔══██╗██╔══██║██║╚██╗██║ ╚██╔╝ ██╔══██║██║╚██╗██║██║ ██║██╔══██╗
██████╔╝██║ ██║██║ ╚████║ ██║ ██║ ██║██║ ╚████║██████╔╝██████╔╝
╚═════╝ ╚═╝ ╚═╝╚═╝ ╚═══╝ ╚═╝ ╚═╝ ╚═╝╚═╝ ╚═══╝╚═════╝ ╚═════╝
***starting as a standalone server****
...
...
***Listening to**** addr::17912 module:LIAISON-GRPC
```

The banyand-server would be listening on the `0.0.0.0:17912` to access gRPC requests. if no errors occurred.

At the same time, the banyand-server would be listening on the `0.0.0.0:17913` to access HTTP requests. if no errors occurred. The HTTP server is used for CLI and Web UI.
8 changes: 7 additions & 1 deletion docs/menu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,13 @@ catalog:
- name: "Welcome"
path: "/readme"
- name: "Installation"
path: "/installation"
catalog:
- name: "Get Binaries"
path: "/installation/binaries"
- name: "Standalone Mode"
path: "/installation/standalone"
- name: "Cluster Mode"
path: "/installation/cluster"
- name: "Clients"
path: "/clients"
- name: "Observability"
Expand Down
Loading