moose-mini

Lots of code borrowed from Andrej Karpathy, Umar Jamil and Evin Tunador.

import torch
from huggingface_hub import hf_hub_download

moose = hf_hub_download(repo_id="namanbnsl/moose-mini", filename="model.py")
weights = hf_hub_download(repo_id="namanbnsl/moose-mini", filename="model.pth")
exec(open(moose).read())

params = ModelArgs()
model = Moose(params)
model.load_state_dict(torch.load(weights))
model.to(params.device)

print(model.generate("Once upon a time, there was a little car named Beep."))

• Only trained on 100m tokens
• Uses llama architecture

Github

huggingface 🤗