Skip to content

Commit

Permalink
readme: Add more info
Browse files Browse the repository at this point in the history
  • Loading branch information
tontinton committed Jul 27, 2024
1 parent b19f2db commit 88f2bdf
Show file tree
Hide file tree
Showing 3 changed files with 23 additions and 0 deletions.
1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ authors = ["Tony Solomonik @ tony.solomonik@gmail.com"]
description = "A search engine on object storage."
repository = "https://github.com/tontinton/toshokan"
homepage = "https://github.com/tontinton/toshokan"
readme = "./README.md"
license = "Apache-2.0 OR MIT"

[profile.release]
Expand Down
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
## Introduction

`toshokan` is a search engine (think Elasticsearch, Splunk), but storing the data on object storage, most similar to <a href="https://github.com/quickwit-oss/quickwit">Quickwit</a>.

It uses:
* <a href="https://github.com/quickwit-oss/tantivy">tantivy</a> - for building and searching the inverted index data structure.
* <a href="https://github.com/apache/opendal">Apache OpenDAL</a> - for an abstraction over object storages.

I've also posted a <a href="https://tontinton.com/posts/new-age-data-intensive-apps/">blog post</a> explaining the benefits and drawbacks of using an object storage for data intensive applications.

## Architecture

<kbd style="background-color: #1e1e1e">
<img src="./architecture.svg">
</kbd>

## How to use

```sh
toshokan create example_config.yaml

Expand All @@ -22,5 +40,8 @@ toshokan search test "tenant_id:[60 TO 65} AND severity_text:INFO" --limit 1 | j
# "timestamp": "2016-04-13T06:46:54Z"
# }

# Merge index files for faster searching.
toshokan merge test

toshokan drop test
```
1 change: 1 addition & 0 deletions architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 88f2bdf

Please sign in to comment.