This is the first ever implementation of a joint Transformer + Mamba + LSTM architecture. The flow is the following: mamba -> transformer -> lstm in a loop. Perhaps with more iteration on model design, we can find a better architecture but this architecture is the future. This model is ready for training right now. If you're interested in training this model or working on entirely new model architectures, join the discord now!
$ pip3 install -U vortex-fusion
import torch
from vortex_fusion import VortexFusion
# Generate random input tensor
x = torch.randint(0, 10000, (1, 10))
# Create an instance of the VortexFusion model with dimension 512
model = VortexFusion(dim=512)
# Pass the input tensor through the model to get the output
output = model(x)
# Print the shape of the output tensor
print(output.shape)MIT
- Train this on the same dataset as Llama. Create one script that trains this model on a massive dataset to experiment with performance metrics.