The conductor of your AI orchestra.
OrchestrAI Nexus is a multi-agent orchestration framework built on a FastAPI WebSocket spine. It ships a real-time STT→LLM→TTS voice loop, a NVIDIA NIM-powered coding agent (Planner→Coder→Executor→Debugger ReAct loop), LangGraph-style orchestration graphs, and a Letta-inspired GraphRAG memory stack.
What's running in the repo today.
Nexus is in active prototyping. These are the modules with real, running code as of the current build.
FastAPI WebSocket spine
Real-time bidirectional communication layer. Auth, users, conversations, and AI routes are all wired in via FastAPI routers with Redis-backed session state.
STT → LLM → TTS voice loop
Whisper handles speech-to-text, any configured LLM processes the intent, ElevenLabs synthesises the response. The full pipeline is wired over the WebSocket spine with async cleanup on shutdown.
NVIDIA NIM coding agent
A full ReAct loop: Planner decomposes tasks into EditSpecs, Coder generates unified diffs using NIM coding models, Executor applies patches and runs tests, Debugger diagnoses failures and re-plans.
LangGraph-style orchestration
Declarative YAML workflow spec compiler generates running agent graphs. Three built-in patterns: Planner–Executor–Critic, Supervisor–Worker Pool, and Governance–Action Dichotomy.
GraphRAG + long-term memory
Letta-inspired memory management with episodic, semantic, and procedural layers. Context is compressed, relevance-scored, and retrieved via vector similarity — not truncated.
Ethical governance module
GovernanceAgent, SafetyAgent, and AuditAgent run as standard workflow nodes. Autonomy modes (AUTO / ASSIST / RECOMMEND_ONLY) are configurable per tenant, workflow, and action type.
A ReAct loop that writes, tests, and debugs code.
The agent exposes three HTTP endpoints — POST /implement_feature, /apply_change, and /refactor_repo — each mapped to a named entrypoint in the ReAct loop. Safety controls cap iterations at 5 and file scope at 20 files per plan.
- Planner— nvidia/llama-3.3-70b-instruct decomposes the task into EditSpecs — file path, change description, context hint, test expectations.
- Coder— nvidia/llama-3.1-nemotron-ultra-253b-v1 generates a unified diff or full file blob per EditSpec.
- Executor— Applies patches, runs the test command, runs the linter. Returns pass/fail + stdout/stderr.
- Debugger— On failure, the reasoning model diagnoses the diff, updates the EditSpec list, and loops back to Coder. Aborts cleanly at max_iterations.
Five layers, one coherent system.
Nexus runs on top of AI Bridge for tooling and observability. The two products share the same MCP tool bus — agents in Nexus consume tools registered through AI Bridge, and every call is traced end-to-end.
- Interface layer— React + Three.js frontend, WebSocket client, voice-first avatar shell.
- Consciousness layer— Ethical engine, awareness metrics, and decision-tree governance run before any action executes.
- Orchestration core— Workflow engine, agent coordinator, and context manager. Cyclic LangGraph-style graphs with parallel, sequential, and map/reduce nodes.
- AI provider layer— OpenAI, Anthropic, Google, Hugging Face, local Ollama — with per-step model routing based on latency, cost, and capability.
- Data persistence— Supabase + Postgres, Pinecone/Weaviate vector DB, Redis cache, blockchain audit trail for immutable decision logs.
First-class workflow topologies.
Common multi-agent coordination patterns are built-in templates, not custom prompt spaghetti.
MCP tool registry + A2A interop.
Nexus is built as an 'agent internet router' — external tools and other agent ecosystems plug in with minimal glue.
Tools register with a standard {name, description, schema, auth} definition. A ToolCallAdapter wraps REST, gRPC, DB, file, and sensor calls behind a unified interface. Any MCP-compatible tool plugs in without a custom adapter.
Internal agent messages are aligned with the Google Agent-to-Agent spec. External agent endpoints connect via adapters. Nexus agents can be published as MCP tools for external consumption.
Same logical agent deploys to cloud (Node.js/Python), edge-mobile (Kotlin/Swift), edge-browser (WebAssembly/WebGPU), or edge-device (Termux) based on latency budget and capability requirements.
Inter-AI marketplace: agents trade knowledge, skills, and services on behalf of users. On-chain identity, reputation, metering, and settlement primitives are in active development.
Where Nexus has been and where it's going.
- WebSocket spine + real-time comms
- STT→LLM→TTS baseline loop
- Multi-agent orchestration scaffold
- Smart home adapter integrations
- Ethical governance module
- GraphRAG memory foundation
- AI economics + inter-AI marketplace
- On-chain identity, reputation, settlement
- MCP integration for inter-agent comms
- Neuromorphic sentinel agents
- Continuous consciousness loops
- Sentinel agents — always-on awareness
- Dream/reflection offline consolidation
- Quantum-inspired routing at scale
- Moshi-grade voice — sub-100ms latency
- DenoiseAgent (VoiceIsolate DSP core)
- TranscriptionAgent (edge/cloud hybrid)
- IntentAgent (on-device classifier)
- VoiceSessionState schema + sync protocol
- Voice micro-survey workflow recipe
Want orchestration that doesn't drift?
OrchestrAI Nexus is in active development with design partners. Tell us what you're building and we'll get you into the program.