Nvidia Switchyard and Nemotron 3.5 Lightning enable efficient local AI model routing and customization

Nvidia’s Switchyard and Nemotron 3.5 Lightning provide a framework for routing AI requests to specialized, local models, reducing reliance on expensive, general-purpose cloud AI. Switchyard acts as a proxy and library that routes traffic between OpenAI and Anthropic APIs to local models, offering protocol translation and multi-backend routing. This approach allows organizations to maintain control over data and reduce costs by offloading tasks to smaller, customized models. The 30-billion parameter Nemotron 3.5 Lightning model is designed for these agentic tasks, offering high performance and ease of customization through Low-Rank Adaptation (LoRA). By implementing this architecture, companies can build a data flywheel that captures internal knowledge and improves model performance over time. This system enables organizations to move away from the high costs of sending every request to a large, general-purpose model, instead utilizing a tiered approach where only complex tasks are escalated to frontier models. The architecture emphasizes observability, allowing IT teams to track resource usage and model performance, which is critical for optimizing AI deployments and ensuring cost-effective operations.

Switchyard is a Rust-based proxy that routes LLM traffic between different AI models and APIs. Nemotron 3.5 Lightning is a 30-billion parameter model optimized for agentic tasks and local deployment.

The architecture uses LoRA adapters to customize smaller models without the need for full retraining. Organizations can reduce cloud AI costs by routing routine tasks to smaller, local models.

The data flywheel approach captures internal usage patterns to improve model accuracy and efficiency. Observability features in the framework allow IT teams to monitor resource usage and model performance.