RsLLM Home

RsLLM (Rust LLM Stream Analyzer and Content Generator) is a powerful and versatile toolkit designed for AI-driven content creation and stream analysis on MacOS devices equipped with M1/M2/M3 ARM GPUs. It leverages the Candle framework from Huggingface to enable efficient interaction with large language models (LLMs) directly within a Rust environment, bypassing the need for external dependencies and Python servers.

Features

• Local LLM Focus: RsLLM utilizes Candle's Rust-based LLMs, Mistral and Gemma, optimized for Metal GPUs on MacOS, ensuring high performance and low latency in AI model interactions.

• Comprehensive AI Analyzer: Embeds a sophisticated AI analyzer capable of processing inputs and generating outputs across text, voice, speech, and images, facilitating a seamless flow of AI-generated content.

• Voice and Speech Integration (Planned): Plans to incorporate Whisper for voice-driven interactions, allowing users to communicate with the toolkit using voice commands and receive streaming text inputs in response.

• Image Generation and NDI Output (Beta): Supports generating images from text descriptions and outputting through NDI for a wide range of applications, including real-time content creation and broadcasting.

• TTS Support: Implements TTS using MetaVoice (default), OpenAI TTS API, and Mimic3 TTS API, offering various options for speech generation.

• Twitch Chat Interactive AI: Integrated Twitch chat for real-time AI interactions, enabling users to engage with the toolkit through chat commands and receive AI-generated responses.

Installation

1. Ensure Rust and Cargo are installed. Follow the official installation guide.

2. Clone the RsLLM repository:

   git clone https://github.com/groovybits/rsllm.git
   

3. Navigate to the project directory:

   cd rsllm
   

4. Compile with Metal GPU support and NDI SDK support:

   ./scripts/compile.sh
   

Configuration

• Copy .env.example to .env and customize the settings, including the OpenAI API key if intending to use external AI models.

Usage

RsLLM facilitates a wide range of AI-driven operations, from generating text-based content to analyzing network streams and processing visual and audio inputs. Usage examples can be found in the ./scripts directory.

Dependencies

RsLLM relies on the following key dependencies:

• Candle: Rust-native LLMs optimized for Metal GPUs on MacOS.
• OpenAI API: Allows leveraging external AI models when necessary.
• NDI SDK: Enables high-quality, low-latency video streaming over IP networks.
• Whisper: Speech-to-text functionality for voice-driven interactions (planned).
• MetaVoice, OpenAI TTS API, Mimic3 TTS API: Text-to-speech capabilities.
• Twitch API: Integration with Twitch chat for real-time AI interactions.

Contributing

Contributions to RsLLM are warmly welcomed, especially in areas such as feature development, performance optimization, and documentation. Your expertise can significantly enhance the toolkit's capabilities and user experience.

License

RsLLM is licensed under the MIT License. See the LICENSE file for detailed information.

Author

RsLLM is developed by Chris Kennedy, who leads the development of innovative AI solutions with the MacOS Metal GPU Rust LLM Toolkit.

For more information, visit the RsLLM GitHub repository and the RsLLM documentation.