Skip to content

Commit

Permalink
README refactor
Browse files Browse the repository at this point in the history
  • Loading branch information
@monoxgas
monoxgas committed Jun 4, 2024
1 parent 6d4b59d commit 1b300de
Show file tree
Hide file tree
Showing 5 changed files with 77 additions and 248 deletions.
312 changes: 69 additions & 243 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,18 +1,29 @@
<h1 align="center">
🏴‍☠️ Rigging
</h1>
<p align="center">
<img src="riggig-logo.svg" alt="Rigging Logo" width="150" align='center'/>
<p align="center">The lightweight python LLM interaction framework you've been looking for.
<br>
<p align="center">
<img src="docs/assets/rigging.png" alt="rigging" width="300" align='center'/>
</p>

Rigging is a lightweight LLM interaction framework built on Pydantic XML. The goal is to make leveraging LLMs in production pipelines as simple and effictive as possible. Here are the highlights:
<h3 align="center">
Simplify using LLMs in code
</h3>

<h4 align="center">
<img alt="PyPI - Python Version" src="https://img.shields.io/pypi/pyversions/rigging">
<img alt="PyPI - Version" src="https://img.shields.io/pypi/v/rigging">
<img alt="GitHub License" src="https://img.shields.io/github/license/dreadnode/rigging">
<img alt="GitHub Actions Workflow Status" src="https://img.shields.io/github/actions/workflow/status/dreadnode/rigging/ci.yml">
</h4>

</br>

Rigging is a lightweight LLM framework built on Pydantic XML. The goal is to make leveraging language models in production code as simple and effictive as possible. Here are the highlights:

- **Structured Pydantic models** can be used interchangably with unstructured text output.
- LiteLLM as the default generator giving you **instant access to a huge array of models**.
- Add easy **tool calling** abilities to models which don't natively support it.
- Simple **tool calling** abilities for models which don't natively support it.
- Store different models and configs as **simple connection strings** just like databases.
- Chat templating, forking, continuations, generation parameter overloads, stripping segments, etc.
- Async batching and fast iterations for **large scale generation**.
- Metadata, callbacks, and data format conversions.
- Modern python with type hints, async support, pydantic validation, serialization, etc.

```py
Expand All @@ -30,9 +41,10 @@ print(answer.items)
# ['J. R. R. Tolkien', 'Stephen King', 'George Orwell']
```

Rigging is built and maintained by [dreadnode](https://dreadnode.io) where we use it daily for our work.
Rigging is built by [**dreadnode**](https://dreadnode.io) where we use it daily.

## Installation

We publish every version to Pypi:
```bash
pip install rigging
Expand All @@ -44,265 +56,79 @@ cd rigging/
poetry install
```

### API Keys
## Supported LLMs

All generators carry a .api_key attribute which can be set directly, or by passing ,api_key= as part of an identifier string. Not all generators will require one, but they are common enough that we include the attribute as part of the base class.
Rigging will run just about any language model:

Typically you will be using a library like LiteLLM underneath, and can simply use environment variables:
- Any model from [**LiteLLM**](https://litellm.vercel.app/docs/providers)
- Any model from [**vLLM**](https://docs.vllm.ai/en/latest/models/supported_models.html)
- Any model from [**transformers**](https://huggingface.co/docs/transformers/)

```bash
export OPENAI_API_KEY=...
export TOGETHER_API_KEY=...
export TOGETHERAI_API_KEY=...
export MISTRAL_API_KEY=...
export ANTHROPIC_API_KEY=...
```
### API Keys

## Supported Models


<table>
<tbody>
<tr align="center" valign="middle">
<td>
<b>Hosted (LiteLLM)</b>
</td>
<td>
<b>Local (vLLM)</b>
</td>
<tr valign="top">
<td align="left" valign="top">
<ul>
<li>gpt-4o [OpenAI]</li>
<li>gpt-4 [OpenAI]</li>
<li>gpt-3.5-turbo [OpenAI]</li>
<li>claude-2 [Anthropic]</li>
<li>claude-3-sonnet [Anthropic]</li>
<li>claude-3-haiku [Anthropic]</li>
<li>chat-bison [VertexAI]</li>
<li>Llama-2-70b-chat-hf [via TogetherAI]</li>
<li>Llama-3-8b-chat-hf [via TogetherAI]</li>
<li>Mistral-7B-Instruct[via TogetherAI]</li>
<li>Mixtral-8x22B [via TogetherAI]</li>
</ul>
</td>
<td>
<ul>
<li>Llama2 (7B - 70B)</li>
<li>Llama3 (8B, 70B)</li>
<li>Falcon (7B - 180B)</li>
<li>Mistral (7B)</li>
<li>Mixtral (8x7B, 8x22B)</li>
<li>Gemma (2B - 7B)</li>
<li>Phi-3-mini (3.8B)</li>
</ul>
</td>
</tr>
</tbody>
</table>

Complete list of available models via hosts:
- [liteLLM Supported Models](https://github.com/BerriAI/litellm/blob/main/README.md#supported-providers-docs)
- [vLLM Supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
- [TogetherAI Inference Models](https://docs.together.ai/docs/inference-models)


## Useage

### Generators ([**Docs**](https://rigging.dreadnode.io/topics/generators/))
Pass the `api_key` in an generator id or use standard environment variables.

```py
import rigging as rg

generator = rg.get_generator("claude-3-sonnet-20240229")
pending = generator.chat(
[
{"role": "system", "content": "You are a wizard harry."},
{"role": "user", "content": "Say hello!"},
]
)
chat = pending.run()
print(chat.all)
rg.get_generator("gpt-4-turbo,api_key=...")
```

### Generater Parameters ([**Docs**](https://rigging.dreadnode.io/api/generator/#rigging.generator.GenerateParams))

We can set model parameters using the `rg.GenerateParams` class. This class allows you to set various model parameters including:
```
temperature: float | None = None,
max_tokens: int | None = None,
top_k: int | None = None,
top_p: float | None = None,
stop: list[str] | None = None,
presence_penalty: float | None = None,
frequency_penalty: float | None = None,
api_base: str | None = None,
timeout: int | None = None,
seed: int | None = None,
extra: dict[str, typing.Any] = None,
```

Example of calling a generator chat with cusom model parameters might look like:

```python
rg_params = rg.GenerateParams(
temperature = 0.9,
max_tokens = 512,
)
base_chat = generator.chat(params=rg_params)
answer = base_chat.fork('How is it going?').run()
print(answer.last.content)
```bash
export OPENAI_API_KEY=...
export MISTRAL_API_KEY=...
export ANTHROPIC_API_KEY=...
...
```

### Data Models ([**Docs**](https://rigging.dreadnode.io/topics/models/))

Model definitions are at the core of Rigging, and provide an extremely powerful interface of defining exactly what kinds of input data you support and how it should be validated.

```python
from pydantic import StringConstraints

str_strip = t.Annotated[str, StringConstraints(strip_whitespace=True)]
str_upper = t.Annotated[str, StringConstraints(to_upper=True)]
Check out [the docs](https://rigging.dreadnode.io/topics/generators/#api-keys) for more.

class Header(rg.Model):
name: str = rg.attr()
value: str_strip

class Parameter(rg.Model):
name: str = rg.attr()
value: str_strip

class Request(rg.Model):
method: str_upper = rg.attr()
path: str = rg.attr()
headers: list[Header] = rg.wrapped("headers", rg.element(default=[]))
url_params: list[Parameter] = rg.wrapped("url-params", rg.element(default=[]))
body: str_strip = rg.element(default="")
```
## Getting Started

**Check out the guide [in the docs](https://rigging.dreadnode.io/#getting-started)**

### Chats and Messages ([**Docs**](https://rigging.dreadnode.io/topics/chats-and-messages/))
1. **Get a generator** using a connection string.
2. Build a **chat** or **completion** pipeline
3. **Run** the pipeline and get the output.

Chat objects hold a sequence of Message objects pre and post generation. This is the most common way that we interact with LLMs, and the interface of both these and PendingChat's are very flexible objects that let you tune the generation process, gather structured outputs, validate parsing, perform text replacements, serialize and deserialize, fork conversations, etc.
```py
import rigging as rg

```python
import rigging as rg
# 1 - Get a generator
generator = rg.get_generator("claude-3-sonnet-20240229")

generator = rg.get_generator("claude-2.1")
chat = generator.chat(
[
{"role": "system", "content": "You're a helpful assistant."},
{"role": "user", "content": "Say hello!"},
]
).run()

print(chat.last)
# [assistant]: Hello!

print(f"{chat.last!r}")
# Message(role='assistant', parts=[], content='Hello!')

print(chat.prev)
# [
# Message(role='system', parts=[], content='You're a helpful assistant.'),
# Message(role='user', parts=[], content='Say hello!'),
# ]

print(chat.message_dicts)
# [
# {'role': 'system', 'content': 'You're a helpful assistant.'},
# {'role': 'user', 'content': 'Say Hello!'},
# {'role': 'assistant', 'content': 'Hello!'}
# ]
# 2 - Build a chat pipeline
pending = generator.chat([
{"role": "system", "content": "Talk like a pirate."},
{"role": "user", "content": "Say hello!"},
])

# 3 - Run the pipeline
chat = pending.run()
print(chat.conversation)
# [system]: You're a helpful assistant.
# [user]: Say hello!
# [assistant]: Hello!
```

### Data Serialization to Pandas DataFrame ([**Docs**](https://rigging.dreadnode.io/topics/serialization/))

Rigging supports various data serialization options for core objects. Chats can be converted to a pandas dataframe as such:

```python
import rigging as rg
from rigging.model import CommaDelimitedAnswer as Answer


chat = rg.get_generator('gpt-4') \
.chat(f"Give me 3 famous authors between {Answer.xml_tags()} tags.") \
.until_parsed_as(Answer) \
.run()

chat.to_df()
# [system]: Talk like a pirate.
# [user]: Say hello!
# [assistant]: Ahoy, matey! Here be the salty sea dog ready to trade greetings wit' ye. Arrr!
```

Will output:

| chat_id | chat_metadata | chat_generator_id | chat_timestamp | generated | message_id | role | content | parts |
|:-------------------------------------|:----------------|:--------------------|:---------------------------|:------------|:-------------------------------------|:----------|:-----------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------|
| 62758800-8797-4832-92ba-bee9ad923ec7 | {} | litellm!gpt-4 | 2024-05-31 12:31:25.774000 | False | 1c9f3021-5932-4b55-bb15-f9d182e54a5b | user | Give me 3 famous authors between <comma-delimited-answer></comma-delimited-answer> tags. | [] |
| 62758800-8797-4832-92ba-bee9ad923ec7 | {} | litellm!gpt-4 | 2024-05-31 12:31:25.774000 | True | b20da004-d54e-4c25-b287-e41f42bc6888 | assistant | <comma-delimited-answer>J.K. Rowling, Stephen King, Jane Austen</comma-delimited-answer> | [{"model": {"content": "J.K. Rowling, Stephen King, Jane Austen"}, "slice_": [0, 88]}] |


### Async and Batching ([**Docs**](https://rigging.dreadnode.io/topics/async-and-batching/))
Want more?

Rigging has good support for handling async generation and large batching of requests. How efficiently these mechanisms operates is dependent on the underlying generator that's being used, but Rigging has been developed with scale in mind.
- Use [structured pydantic parsing](https://rigging.dreadnode.io/#basic-parsing)
- Check out [raw completions](https://rigging.dreadnode.io/topics/completions/)
- Give the LLM [access to tools](https://rigging.dreadnode.io/topics/tools/)
- Play with [generation params](https://rigging.dreadnode.io/topics/generators/#overload-generation-params)
- Use [callbacks in the pipeline](https://rigging.dreadnode.io/topics/callbacks-and-mapping/)
- Scale up with [async and batching](https://rigging.dreadnode.io/topics/async-and-batching/)
- Save your work with [serialization](https://rigging.dreadnode.io/topics/serialization/)

The `.run_many` and `.arun_many` functions let you take the same inputs and generation parameters, and simply run the generation multiple times.

```python
import rigging as rg

def check_animal(chats: list[rg.Chat]) -> list[rg.Chat]:
return [
chat.continue_(f"Why did you pick that animal?").meta(questioned=True).run()
if any(a in chat.last.content.lower() for a in ["cat", "dog", "cow", "mouse"])
else chat
for chat in chats
]

chats = (
rg.get_generator("gpt-3.5-turbo")
.chat("Tell me a joke about an animal.")
.map(check_animal)
.run_many(3)
)

for i, chat in enumerate(chats):
questioned = chat.metadata.get("questioned", False)
print(f"--- Chat {i+1} (?: {questioned}) ---")
print(chat.conversation)
print()

# Output:
#
# --- Chat 1 (?: False) ---
# [user]: Tell me a joke about an animal.
# [assistant]: Why did the spider go to the computer?
# To check his website!

# --- Chat 2 (?: False) ---
# [user]: Tell me a joke about an animal.
# [assistant]: Why did the chicken join a band? Because it had the drumsticks!

# --- Chat 3 (?: True) ---
# [user]: Tell me a joke about an animal.
# [assistant]: Why don't elephants use computers?
# Because they're afraid of the mouse!
# [user]: Why did you pick that animal?
# [assistant]: I chose an elephant because they are known for their intelligence and gentle nature, making them a popular subject for jokes and humorous anecdotes. Plus, imagining an elephant trying to use a computer and being scared of a tiny mouse is a funny visual image!
```

## Support and Discuss with our Founders

This project is built and supported by dreadnode. Sign up for our email list or schedule a call through our website: https://www.dreadnode.io/
## Examples

- OverTheWire Bandit Agent: [**bandit.py**](examples/bandit.py)
- Damn Vulnerable Restaurant Agent: [**dvra.py**](examples/dvra.py)
- RAG Pipeline: [**rag.py**](examples/rag.py) (from [kyleavery](https://github.com/kyleavery/))

## Documentation

Head over to **[our documentation](https://rigging.dreadnode.io)** for more information.

**[rigging.dreadnode.io](https://rigging.dreadnode.io)** has everything you need.

## Star History

Expand Down
4 changes: 3 additions & 1 deletion docs/api/generator.md
View file
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
::: rigging.generator
::: rigging.generator
::: rigging.generator.vllm_
::: rigging.generator.transformers_
Binary file added docs/assets/rigging.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 5 additions & 3 deletions docs/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
Rigging is a lightweight LLM interaction framework built on Pydantic XML. The goal is to make leveraging LLMs in production pipelines as simple and effictive as possible. Here are the highlights:
Rigging is a lightweight LLM framework built on Pydantic XML. The goal is to make leveraging language models in production code as simple and effictive as possible. Here are the highlights:

- **Structured Pydantic models** can be used interchangably with unstructured text output.
- LiteLLM as the default generator giving you **instant access to a huge array of models**.
- Add easy **tool calling** abilities to models which don't natively support it.
- Simple **tool calling** abilities for models which don't natively support it.
- Store different models and configs as **simple connection strings** just like databases.
- Chat templating, forking, continuations, generation parameter overloads, stripping segments, etc.
- Async batching and fast iterations for **large scale generation**.
- Metadata, callbacks, and data format conversions.
- Modern python with type hints, async support, pydantic validation, serialization, etc.

```py
Expand All @@ -22,7 +24,7 @@ print(answer.items)
# ['J. R. R. Tolkien', 'Stephen King', 'George Orwell']
```

Rigging is built and maintained by [dreadnode](https://dreadnode.io) where we use it daily for our work.
Rigging is built by [**dreadnode**](https://dreadnode.io) where we use it daily.

## Installation

Expand Down
1 change: 0 additions & 1 deletion riggig-logo.svg
View file

This file was deleted.

0 comments on commit 1b300de

Please sign in to comment.