Skip to content

Commit

Permalink
Refactor README.md: Update Token Count React App description and add …
Browse files Browse the repository at this point in the history
…usage instructions
  • Loading branch information
aeltorio committed Oct 7, 2024
1 parent e72b526 commit 8a1f7b0
Showing 1 changed file with 64 additions and 2 deletions.
66 changes: 64 additions & 2 deletions tokenCount/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,65 @@
# React + Sentencepiece-js
# Token Count React App

This is a simple React app showing sentencepiece-js in a browser. No data is exchanged with your browser.
This directory contains a simple React 18 application that demonstrates the use of the Meta Llama 3.1 tokenizer model for counting tokens. The application is built with React 18, Vite, and the Fluent UI v9 framework.

## Overview

The main component of the application is defined in `App.tsx`. It uses Fluent UI components to create a card layout that includes a header, a text analyzer, and a footer. The text analyzer allows users to input text and count the tokens using the Llama 3.1 tokenizer model.

### Key Features

- **React 18**: Utilizes the latest features of React 18.
- **Vite**: A fast build tool for modern web projects.
- **Fluent UI v9**: Provides a consistent and modern UI framework.

## File Structure

- `App.tsx`: The main component that sets up the layout and integrates the text analyzer.
- `TextAnalyzer.tsx`: A component that handles text input and token counting.
- `vite.config.ts`: Configuration file for Vite.

## Usage

To run the application locally, follow these steps:

1. **Install dependencies**:

```sh
npm install
```

2. **Start the development server**:

```sh
npm run dev
```

3. **Build the application**:

```sh
npm run build
```

4. **Preview the production build**:

```sh
npm run preview
```

## Deployment

The application can be deployed to GitHub Pages. Ensure that the `base` property in `vite.config.ts` is set correctly to the URL where the app will be hosted.

## Example

You can see the application in action on GitHub Pages: [Sentencepiece Token Counter](https://sctg-development.github.io/sentencepiece-js/).

## License

This project is licensed under the MIT License. See the [LICENSE](../LICENSE) file for more details.

## Author

Ronan Le Meillat

For more information, visit the [GitHub repository](https://github.com/sctg-development/sentencepiece-js).

0 comments on commit 8a1f7b0

Please sign in to comment.