Skip to content
Back to search

nexus

Govern & Secure your AI

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/nexus-router--nexus/badge.svg)](/listings/nexus-router--nexus)

HTML

<a href="/listings/nexus-router--nexus"><img src="/api/listings/nexus-router--nexus/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

Quick Install (Linux/Windows (WSL)/macOS)

curl -fsSL https://nexusrouter.com/install | bash

Docker

Pull the latest image:

docker pull ghcr.io/grafbase/nexus:latest

Or use the stable version:

docker pull ghcr.io/grafbase/nexus:stable

Or use a specific version:

docker pull ghcr.io/grafbase/nexus:X.Y.Z

Build from Source

git clone https://github.com/grafbase/nexus
cd nexus
cargo build --release

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 1,990 lines · 1 finding marked

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. <picture>
  3. <source width="600" height="244" srcset="https://github.com/user-attachments/assets/9ca64651-b601-45e3-8ba1-f7bfb91625ab" media="(prefers-color-scheme: dark)">
  4. <source width="600" height="244" srcset="https://github.com/user-attachments/assets/5ee33450-f9ee-4e47-b0ed-0d302110c4ce"" media="(prefers-color-scheme: light)">
  5. <img src="https://github.com/user-attachments/assets/9ca64651-b601-45e3-8ba1-f7bfb91625ab" alt="Nexus logo">
  6. </picture>
  7. </div>
  8. <p align="center">
  9. Plug in all your MCP servers, APIs, and LLM providers. Route everything through a unified endpoint. <br />
  10. Aggregate, govern, and control your AI stack.
  11. </p>
  12. ## Features
  13. - **MCP Server Aggregation**: Connect multiple MCP servers (STDIO, SSE, HTTP) through a single endpoint
  14. - **LLM Provider Routing**: Unified interface for OpenAI, Anthropic, Google, and AWS Bedrock LLM providers with full tool calling support
  15. - **Context-Aware Tool Search**: Intelligent fuzzy search across all connected tools using natural language queries
  16. - **Protocol Support**: Supports STDIO (subprocess), SSE (Server-Sent Events), and streamable HTTP MCP servers
  17. - **Flexible Configuration**: TOML-based configuration with environment variable substitution
  18. - **Security**: Built-in CORS, CSRF protection, OAuth2, and TLS support
  19. - **Rate Limiting**: Multi-level rate limiting with in-memory or Redis backends
  20. - **Docker Ready**: Available as a container image with minimal configuration needed
  21. ## Installation
  22. ### Quick Install (Linux/Windows (WSL)/macOS)
  23. ```bash
  24. curl -fsSL https://nexusrouter.com/install | bash
    • Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
    • Line 30 on GitHub
  25. ```
  26. ### Docker
  27. Pull the latest image:
  28. ```bash
  29. docker pull ghcr.io/grafbase/nexus:latest
  30. ```
  31. Or use the stable version:
  32. ```bash
  33. docker pull ghcr.io/grafbase/nexus:stable
  34. ```
  35. Or use a specific version:
  36. ```bash
  37. docker pull ghcr.io/grafbase/nexus:X.Y.Z
  38. ```
  39. ### Build from Source
  40. ```bash
  41. git clone https://github.com/grafbase/nexus
  42. cd nexus
  43. cargo build --release
  44. ```
  45. ## Running Nexus
  46. ### Using the Binary
  47. ```bash
  48. nexus
  49. ```
  50. ### Using Docker
  51. ```bash
  52. docker run -p 8000:8000 -v /path/to/config:/etc/nexus.toml ghcr.io/grafbase/nexus:latest
  53. ```
  54. ### Docker Compose Example
  55. ```yaml
  56. services:
  57. nexus:
  58. image: ghcr.io/grafbase/nexus:latest
  59. ports:
  60. - "8000:8000"
  61. volumes:
  62. - ./nexus.toml:/etc/nexus.toml
  63. environment:
  64. - GITHUB_TOKEN=${GITHUB_TOKEN}
  65. healthcheck:
  66. test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
  67. interval: 30s
  68. timeout: 10s
  69. retries: 3
  70. ```
  71. ## Configuration
  72. Create a `nexus.toml` file to configure Nexus:
  73. ```toml
  74. # LLM Provider configuration
  75. [llm.providers.openai]
  76. type = "openai"
  77. api_key = "{{ env.OPENAI_API_KEY }}"
  78. forward_token = true
  79. # Model configuration (at least one model required per provider)
  80. [llm.providers.openai.models.gpt-4]
  81. [llm.providers.openai.models.gpt-3-5-turbo]
  82. [llm.providers.anthropic]
  83. type = "anthropic"
  84. api_key = "{{ env.ANTHROPIC_API_KEY }}"
  85. [llm.providers.anthropic.models.claude-3-5-sonnet-20241022]
  86. # MCP Server configuration
  87. [mcp.servers.github]
  88. url = "https://api.githubcopilot.com/mcp/"
  89. auth.token = "{{ env.GITHUB_TOKEN }}"
  90. [mcp.servers.filesystem]
  91. cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/home/YOUR_USERNAME/Desktop"]
  92. [mcp.servers.python_server]
  93. cmd = ["python", "-m", "mcp_server"]
  94. env = { PYTHONPATH = "/opt/mcp" }
  95. cwd = "/workspace"
  96. ```
  97. ### Configuration Options
  98. #### Server Configuration
  99. - `server.listen_address`: The address and port Nexus will listen on (default: `127.0.0.1:8000`)
  100. - `server.health.enabled`: Enable health endpoint (default: `true`)
  101. - `server.health.path`: Health check endpoint path (default: `/health`)
  102. #### LLM Configuration
  103. - `llm.enabled`: Enable LLM functionality (default: `true`)
  104. - `llm.protocols.openai.enabled`: Enable OpenAI protocol endpoint (default: `true`)
  105. - `llm.protocols.openai.path`: OpenAI endpoint path (default: `/llm/openai`)
  106. - `llm.protocols.anthropic.enabled`: Enable Anthropic protocol endpoint (default: `false`)
  107. - `llm.protocols.anthropic.path`: Anthropic endpoint path (default: `/llm/anthropic`)
  108. For detailed LLM provider configuration, see the LLM Provider Configuration section below.
  109. #### MCP Configuration
  110. - `mcp.enabled`: Enable MCP functionality (default: `true`)
  111. - `mcp.path`: MCP endpoint path (default: `/mcp`)
  112. - `mcp.enable_structured_content`: Control MCP search tool response format (default: `true`)
  113. - When `true`: Uses modern `structuredContent` field for better performance and type safety
  114. - When `false`: Uses legacy `content` field with `Content::json` objects for compatibility with older MCP clients
  115. #### MCP Server Types
  116. 1. **STDIO Servers**: Launch local processes that communicate via standard input/output
  117. ```toml
  118. [mcp.servers.my_tool]
  119. cmd = ["path/to/executable", "--arg1", "--arg2"]
  120. # Optional: Set environment variables
  121. env = { DEBUG = "1", API_KEY = "{{ env.MY_API_KEY }}" }
  122. # Optional: Set working directory
  123. cwd = "/path/to/working/directory"
  124. # Optional: Configure stderr handling (default: "null")
  125. stderr = "inherit" # Show in console
  126. # or
  127. stderr = { file = "/var/log/mcp/server.log" } # Log to file
  128. ```
  129. **Note**: STDIO servers must output valid JSON-RPC messages on stdout. The `cmd` array must have at least one element (the executable).
  130. 2. **SSE Servers**: Connect to Server-Sent Events endpoints
  131. ```toml
  132. [mcp.servers.my_sse_server]
  133. protocol = "sse"
  134. url = "http://example.com/sse"
  135. message_url = "http://example.com/messages" # Optional
  136. ```
  137. 3. **HTTP Servers**: Connect to streamable HTTP endpoints
  138. ```toml
  139. [mcp.servers.my_http_server]
  140. protocol = "streamable-http"
  141. url = "https://api.example.com/mcp"
  142. ```
  143. For remote MCP servers, if you omit the protocol Nexus will first try streamable HTTP and then SSE.
  144. #### Authentication
  145. Add service token authentication to any server:
  146. ```toml
  147. [mcp.servers.my_server.auth]
  148. token = "your-token-here"
  149. # Or use environment variables
  150. token = "{{ env.MY_API_TOKEN }}"
  151. ```
  152. If you enable OAuth2 authentication to your server, and your downstream servers all use the same authentication server, you can configure Nexus to forward the request access token to the downstream server.
  153. ```toml
  154. [mcp.servers.my_server.auth]
  155. type = "forward"
  156. ```
  157. #### Header Insertion for MCP Servers
  158. Nexus supports inserting static headers when making requests to MCP servers. Headers can be configured globally (for all MCP servers) or per-server.
  159. **Note**: MCP currently only supports header insertion with static values. Headers from incoming requests are not forwarded.
  160. ##### Global MCP Headers
  161. Configure headers that apply to all MCP servers:
  162. ```toml
  163. # Global headers for all MCP servers
  164. [[mcp.headers]]
  165. rule = "insert"
  166. name = "X-Application"
  167. value = "nexus-router"
  168. [[mcp.headers]]
  169. rule = "insert"
  170. name = "X-API-Version"
  171. value = "v1"
  172. ```
  173. ##### Server-Specific Headers
  174. Configure headers for individual HTTP-based MCP servers:
  175. ```toml
  176. [mcp.servers.my_server]
  177. url = "https://api.example.com/mcp"
  178. # Insert headers for this specific server
  179. [[mcp.servers.my_server.headers]]
  180. rule = "insert"
  181. name = "X-API-Key"
  182. value = "{{ env.MY_API_KEY }}" # Environment variable substitution
  183. [[mcp.servers.my_server.headers]]
  184. rule = "insert"
  185. name = "X-Service-Name"
  186. value = "my-service"
  187. ```
  188. ##### MCP Header Features
  189. - **Static Values Only**: Headers are set at client initialization time with static values
  190. - **Environment Variables**: Use `{{ env.VAR_NAME }}` syntax for environment variable substitution
  191. - **HTTP Servers Only**: Headers only apply to HTTP-based MCP servers (not STDIO servers)
  192. - **Insert Rule**: Currently only the `insert` rule is supported for MCP
  193. #### MCP Access Control
  194. Restrict access to MCP servers and tools based on user groups:
  195. ```toml
  196. # Server-level access control
  197. [mcp.servers.premium_tools]
  198. cmd = ["premium-server"]
  199. allow = ["premium", "enterprise"] # Only these groups can access
  200. deny = ["suspended"] # Block specific groups
  201. # Tool-level override (more specific than server-level)
  202. [mcp.servers.premium_tools.tools.expensive_feature]
  203. allow = ["enterprise"] # Only enterprise can use this tool
  204. [mcp.servers.premium_tools.tools.deprecated_tool]
  205. allow = [] # Empty allow list blocks all access (no client ID needed)
  206. ```
  207. Access control rules:
  208. - If `allow` is set, only listed groups can access (requires client identification)
  209. - If `deny` is set, listed groups are blocked (requires client identification)
  210. - Empty `allow = []` blocks all access without requiring client identification
  211. - Tool-level rules override server-level rules
  212. - Deny takes priority over allow
  213. #### OAuth2 Authentication
  214. Configure OAuth2 authentication to protect your Nexus endpoints:
  215. ```toml
  216. [server.oauth]
  217. url = "https://your-oauth-provider.com/.well-known/jwks.json"
  218. poll_interval = "5m"
  219. expected_issuer = "https://your-oauth-provider.com"
  220. expected_audience = "your-service-audience"
  221. [server.oauth.protected_resource]
  222. resource = "https://your-nexus-instance.com"
  223. authorization_servers = ["https://your-oauth-provider.com"]
  224. ```
  225. OAuth2 configuration options:
  226. - `url`: JWKs endpoint URL for token validation
  227. - `poll_interval`: How often to refresh JWKs (optional, default: no polling)
  228. - `expected_issuer`: Expected `iss` claim in JWT tokens (optional)
  229. - `expected_audience`: Expected `aud` claim in JWT tokens (optional)
  230. - `protected_resource.resource`: URL of this protected resource
  231. - `protected_resource.authorization_servers`: List of authorization server URLs
  232. When OAuth2 is enabled, all endpoints except `/health` and `/.well-known/oauth-protected-resource` require valid JWT tokens in the `Authorization: Bearer <token>` header.
  233. #### Rate Limiting
  234. Nexus supports rate limiting to prevent abuse and ensure fair resource usage:
  235. ```toml
  236. # Global rate limiting configuration
  237. [server.rate_limits]
  238. enabled = true
  239. # Storage backend configuration
  240. [server.rate_limits.storage]
  241. type = "memory" # or "redis" for distributed rate limiting
  242. # For Redis backend:
  243. # url = "redis://localhost:6379"
  244. # key_prefix = "nexus:rate_limit:"
  245. # Global rate limit (applies to all requests)
  246. [server.rate_limits.global]
  247. limit = 1000
  248. interval = "60s"
  249. # Per-IP rate limit
  250. [server.rate_limits.per_ip]
  251. limit = 100
  252. interval = "60s"
  253. # Per-MCP server rate limits
  254. [mcp.servers.my_server.rate_limits]
  255. limit = 50
  256. interval = "60s"
  257. # Tool-specific rate limits (override server defaults)
  258. [mcp.servers.my_server.rate_limits.tools]
  259. expensive_tool = { limit = 10, interval = "60s" }
  260. cheap_tool = { limit = 100, interval = "60s" }
  261. ```
  262. **Rate Limiting Features:**
  263. - **Multiple levels**: Global, per-IP, per-server, and per-tool limits
  264. - **Storage backends**: In-memory (single instance) or Redis (distributed)
  265. - **Flexible intervals**: Configure time windows for each limit
  266. - **Tool-specific overrides**: Set different limits for expensive operations
  267. **Redis Backend Configuration:**
  268. ```toml
  269. [server.rate_limits.storage]
  270. type = "redis"
  271. url = "redis://localhost:6379"
  272. key_prefix = "nexus:rate_limit:"
  273. response_timeout = "1s"
  274. connection_timeout = "5s"
  275. # Connection pool settings
  276. [server.rate_limits.storage.pool]
  277. max_size = 16
  278. min_idle = 0
  279. timeout_create = "5s"
  280. timeout_wait = "5s"
  281. timeout_recycle = "300s"
  282. # TLS configuration for Redis (optional)
  283. [server.rate_limits.storage.tls]
  284. enabled = true
  285. ca_cert_path = "/path/to/ca.crt"
  286. client_cert_path = "/path/to/client.crt" # For mutual TLS
  287. client_key_path = "/path/to/client.key"
  288. # insecure = true # WARNING: Only for development/testing, skips certificate validation
  289. ```
  290. **Note**: When configuring tool-specific rate limits, Nexus will warn if you reference tools that don't exist.
  291. #### LLM Token Rate Limiting
  292. Nexus provides token-based rate limiting for LLM providers to help control costs and prevent abuse. Unlike request-based rate limits, token rate limits count an estimate of actual tokens consumed.
  293. ##### Prerequisites
  294. **IMPORTANT**: LLM rate limiting requires client identification to be enabled:
  295. ```toml
  296. [server.client_identification]
  297. enabled = true
  298. # Choose identification methods (at least one required)
  299. client_id.jwt_claim = "sub" # Extract ID from JWT 'sub' claim
  300. # or
  301. client_id.http_header = "X-Client-ID" # Extract ID from HTTP header
  302. # Optional: Limit groups per user (at most one allowed)
  303. group_id.jwt_claim = "groups" # JWT claim containing user's group
  304. # or
  305. group_id.http_header = "X-Group-ID" # Extract ID from HTTP header
  306. # You must provide a list of allowed groups
  307. [server.client_identification.validation]
  308. group_values = ["free", "pro", "max"]
  309. ```
  310. Without client identification, rate limits cannot be enforced and requests will fail with a configuration error.
  311. ##### Configuration Hierarchy
  312. Token rate limits can be configured at four levels, from most to least specific:
  313. 1. **Model per user + group**: Specific model for specific each user in a group
  314. 2. **Model per user**: Specific model for each user
  315. 3. **Provider per user + group**: All models from provider for each user in a group
  316. 4. **Provider per user**: All models from provider for each user
  317. The most specific applicable limit is always used.
  318. ##### Basic Configuration
  319. ```toml
  320. # Provider-level default rate limit (applies to all models)
  321. [llm.providers.openai.rate_limits.per_user]
  322. input_token_limit = 100000 # 100K input tokens
  323. interval = "1m" # Per minute
  324. # Model-specific rate limit (overrides provider default)
  325. [llm.providers.openai.models.gpt-4.rate_limits.per_user]
  326. input_token_limit = 50000 # More restrictive for expensive model
  327. interval = "30s"
  328. ```
  329. ##### Group-Based Rate Limits
  330. Configure different limits for user groups (requires `group_id` and `group_values` in client identification):
  331. ```toml
  332. # Provider-level group limits
  333. [llm.providers.openai.rate_limits.per_user.groups]
  334. free = { input_token_limit = 10000, interval = "60s" }
  335. pro = { input_token_limit = 100000, interval = "60s" }
  336. enterprise = { input_token_limit = 1000000, interval = "60s" }
  337. # Model-specific group limits (override provider groups)
  338. [llm.providers.openai.models.gpt-4.rate_limits.per_user.groups]
  339. free = { input_token_limit = 5000, interval = "60s" }
  340. pro = { input_token_limit = 50000, interval = "60s" }
  341. enterprise = { input_token_limit = 500000, interval = "60s" }
  342. ```
  343. The limits are per user, but you can define different limits if the user is part of a specific group. If the user does not belong to any group, they will be assigned to the per-user limits.
  344. ##### Complete Example
  345. ```toml
  346. # Client identification (REQUIRED for rate limiting)
  347. [server.client_identification]
  348. enabled = true
  349. client_id.jwt_claim = "sub"
  350. group_id.jwt_claim = "subscription_tier"
  351. [server.client_identification.validation]
  352. group_values = ["free", "pro", "enterprise"]
  353. # OpenAI provider with rate limiting
  354. [llm.providers.openai]
  355. type = "openai"
  356. api_key = "{{ env.OPENAI_API_KEY }}"
  357. # Provider-level defaults
  358. [llm.providers.openai.rate_limits.per_user]
  359. input_token_limit = 100000
  360. interval = "60s"
  361. [llm.providers.openai.rate_limits.per_user.groups]
  362. free = { input_token_limit = 10000, interval = "60s" }
  363. pro = { input_token_limit = 100000, interval = "60s" }
  364. # GPT-4 specific limits (more restrictive)
  365. [llm.providers.openai.models.gpt-4]
  366. [llm.providers.openai.models.gpt-4.rate_limits.per_user]
  367. input_token_limit = 50000
  368. interval = "60s"
  369. [llm.providers.openai.models.gpt-4.rate_limits.per_user.groups]
  370. free = { input_token_limit = 5000, interval = "60s" }
  371. pro = { input_token_limit = 50000, interval = "60s" }
  372. # GPT-3.5 uses provider defaults
  373. [llm.providers.openai.models.gpt-3-5-turbo]
  374. ```
  375. ##### How Token Counting Works
  376. 1. **Input Tokens Only**: Rate limiting is based solely on input tokens counted from the request's messages and system prompts
  377. 2. **No Output Buffering**: Output tokens and `max_tokens` parameter are NOT considered in rate limit calculations
  378. 3. **Pre-check**: Input tokens are checked against rate limits before processing
  379. 4. **Token Accumulation**: Uses a sliding window algorithm to track usage over time
  380. Note: The rate limiting is designed to be predictable and based only on what the client sends, not on variable output sizes.
  381. ##### Rate Limit Response
  382. When rate limited, the server returns a 429 status code. No Retry-After headers are sent to maintain consistency with downstream LLM provider behavior.
  383. ##### Error Responses
  384. When rate limits are exceeded:
  385. ```json
  386. {
  387. "error": {
  388. "message": "Rate limit exceeded: Token rate limit exceeded. Please try again later.",
  389. "type": "rate_limit_error",
  390. "code": 429
  391. }
  392. }
  393. ```
  394. ##### Important Notes
  395. - **Per-User Limits**: All limits are per individual user/client ID
  396. - **No Shared Pools**: Currently, there are no shared/global token pools
  397. - **Streaming Support**: Token counting works with both regular and streaming responses
  398. - **Provider Agnostic**: Works consistently across all LLM providers
  399. - **Validation**: Configuration is validated at startup; invalid group names will cause errors
  400. #### TLS Configuration
  401. Configure TLS for downstream connections:
  402. ```toml
  403. [mcp.servers.my_server.tls]
  404. verify_certs = true
  405. accept_invalid_hostnames = false
  406. root_ca_cert_path = "/path/to/ca.pem"
  407. client_cert_path = "/path/to/client.pem"
  408. client_key_path = "/path/to/client.key"
  409. ```
  410. ### LLM Provider Configuration
  411. Nexus provides a unified interface for multiple LLM providers, allowing you to route chat completions through various services with a consistent API.
  412. #### Enabling LLM Routing
  413. ```toml
  414. [llm]
  415. enabled = true # Enable LLM functionality (default: true)
  416. # OpenAI protocol endpoint configuration
  417. [llm.protocols.openai]
  418. enabled = true # Enable OpenAI protocol (default: true)
  419. path = "/llm" # Custom path (default: "/llm/openai")
  420. # Anthropic protocol endpoint configuration
  421. [llm.protocols.anthropic]
  422. enabled = true # Enable Anthropic protocol (default: false)
  423. path = "/claude" # Custom path (default: "/llm/anthropic")
  424. ```
  425. #### Supported Providers
  426. Nexus currently supports four major LLM providers with full tool calling capabilities:
  427. 1. **OpenAI** (including OpenAI-compatible APIs) - Full tool calling and parallel execution support
  428. 2. **Anthropic** (Claude models) - Tool calling with function definitions and tool choice controls
  429. 3. **Google** (Gemini models) - Function calling with parameter schemas and tool selection
  430. 4. **AWS Bedrock** (Multiple model families via AWS) - Tool calling support across all supported model families
  431. #### Provider Configuration
  432. Configure one or more LLM providers in your `nexus.toml`:
  433. ##### OpenAI Provider
  434. ```toml
  435. [llm.providers.openai]
  436. type = "openai"
  437. api_key = "{{ env.OPENAI_API_KEY }}"
  438. # Optional: Use a custom base URL (for Azure OpenAI, proxies, or compatible APIs)
  439. base_url = "https://api.openai.com/v1" # Default
  440. # Model Configuration (REQUIRED - at least one model must be configured)
  441. [llm.providers.openai.models.gpt-4]
  442. # Optional: Rename the model for your users
  443. # rename = "smart-model" # Users will see "openai/smart-model"
  444. [llm.providers.openai.models.gpt-3-5-turbo]
  445. # Models without rename use their original ID
  446. ```
  447. ##### Anthropic Provider
  448. ```toml
  449. [llm.providers.anthropic]
  450. type = "anthropic"
  451. api_key = "{{ env.ANTHROPIC_API_KEY }}"
  452. # Optional: Use a custom base URL
  453. base_url = "https://api.anthropic.com/v1" # Default
  454. # Model Configuration (REQUIRED - at least one model must be configured)
  455. [llm.providers.anthropic.models.claude-3-opus-20240229]
  456. [llm.providers.anthropic.models.claude-3-5-sonnet-20241022]
  457. ```
  458. ##### Google Provider
  459. ```toml
  460. [llm.providers.google]
  461. type = "google"
  462. api_key = "{{ env.GOOGLE_API_KEY }}"
  463. # Optional: Use a custom base URL
  464. base_url = "https://generativelanguage.googleapis.com/v1beta" # Default
  465. # Model Configuration (REQUIRED - at least one model must be configured)
  466. # Note: Model IDs with dots must be quoted in TOML
  467. [llm.providers.google.models."gemini-1.5-flash"]
  468. [llm.providers.google.models.gemini-pro]
  469. ```
  470. ##### AWS Bedrock Provider
  471. ```toml
  472. [llm.providers.bedrock]
  473. type = "bedrock"
  474. # Optional: AWS profile to use (defaults to environment settings)
  475. profile = "{{ env.AWS_PROFILE }}"
  476. # Optional: AWS region (defaults to environment or us-east-1)
  477. region = "us-west-2"
  478. # Model Configuration (REQUIRED - at least one model must be configured)
  479. # Bedrock uses model IDs with dots, so they must be quoted
  480. [llm.providers.bedrock.models."anthropic.claude-3-5-sonnet-20241022-v2:0"]
  481. [llm.providers.bedrock.models."anthropic.claude-3-opus-20240229-v1:0"]
  482. [llm.providers.bedrock.models."amazon.nova-micro-v1:0"]
  483. [llm.providers.bedrock.models."meta.llama3-8b-instruct-v1:0"]
  484. [llm.providers.bedrock.models."ai21.jamba-1.5-mini-v1:0"]
  485. # Rename models for simpler access
  486. [llm.providers.bedrock.models.claude-haiku]
  487. rename = "anthropic.claude-3-5-haiku-20241022-v1:0" # Users will access as "bedrock/claude-haiku"
  488. [llm.providers.bedrock.models.jamba-mini]
  489. rename = "ai21.jamba-1.5-mini-v1:0" # Users will access as "bedrock/jamba-mini"
  490. ```
  491. AWS Bedrock provides access to multiple foundation models through a single API. Key features:
  492. - **Unified Access**: Use models from Anthropic, Amazon, Meta, Cohere, and more through one interface
  493. - **AWS Integration**: Leverages AWS credentials and IAM for authentication
  494. - **Regional Availability**: Models may vary by AWS region
  495. - **Native Streaming**: Full streaming support for all compatible models
  496. **Authentication**: Bedrock uses standard AWS credential chain:
  497. 1. Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
  498. 2. AWS profile (specified in config or via `AWS_PROFILE` environment variable)
  499. 3. IAM role (when running on EC2/ECS/Lambda)
  500. 4. AWS SSO credentials
  501. **Supported Model Families**:
  502. - **AI21 Jamba**: Jamba 1.5 Mini and Large models with 256K context window
  503. - **Anthropic Claude**: All Claude 3 models (Opus, Sonnet, Haiku) and Claude Instant
  504. - **Amazon Nova**: Nova Micro, Lite, Pro models
  505. - **Amazon Titan**: Titan Text and Embeddings models
  506. - **Meta Llama**: Llama 2 and Llama 3 models
  507. - **Cohere Command**: Command and Command R models
  508. - **DeepSeek**: DeepSeek R1 reasoning models
  509. - **Mistral**: Mistral 7B and Mixtral models
  510. **Model ID Format**: Bedrock model IDs follow the pattern `provider.model-name-version:revision`, for example:
  511. - `ai21.jamba-1.5-mini-v1:0`
  512. - `anthropic.claude-3-5-sonnet-20241022-v2:0`
  513. - `amazon.nova-micro-v1:0`
  514. - `meta.llama3-8b-instruct-v1:0`
  515. #### Model Configuration
  516. Nexus automatically discovers models from every provider. At startup the server performs an initial discovery; a background task then refreshes provider inventories every five minutes and publishes the results through a watch channel for lock-free lookups.
  517. ##### Automatic Discovery (default)
  518. Simply configure a provider—no model list is required. Nexus fetches the provider's models and exposes them as bare model names:
  519. ```toml
  520. [llm.providers.openai]
  521. type = "openai"
  522. api_key = "{{ env.OPENAI_API_KEY }}"
  523. ```
  524. Discovered models appear in `/v1/models` (e.g., `gpt-4`, `gpt-4o-mini`). Requests using these bare names route automatically.
  525. ##### Restrict Discovery with `model_filter`
  526. Use `model_filter` to limit which discovered models are exposed. The regex is case-insensitive, must not be empty, and follows the [regex crate syntax](https://docs.rs/regex/1.11.3/regex/).
  527. ```toml
  528. [llm.providers.openai]
  529. type = "openai"
  530. api_key = "{{ env.OPENAI_API_KEY }}"
  531. model_filter = "^gpt-4" # Only expose GPT-4-family models from discovery
  532. [llm.providers.anthropic]
  533. type = "anthropic"
  534. api_key = "{{ env.ANTHROPIC_API_KEY }}"
  535. model_filter = "^claude-"
  536. ```
  537. If discovery fails for any provider at startup, Nexus exits so you can fix the configuration. Refresh failures log errors but keep the last successful snapshot.
  538. ##### Combine Discovery with Explicit Models
  539. Explicit models remain useful for renames, rate limits, and legacy prefixes. They always appear with `provider/model` even when a filter would exclude them.
  540. ```toml
  541. [llm.providers.openai]
  542. type = "openai"
  543. api_key = "{{ env.OPENAI_API_KEY }}"
  544. model_filter = "^gpt-4"
  545. [llm.providers.openai.models.gpt-3-5-turbo]
  546. rename = "fast-model"
  547. [llm.providers.openai.models.gpt-4]
  548. [llm.providers.openai.models.gpt-4.rate_limits.per_user]
  549. input_token_limit = 50000
  550. interval = "60s"
  551. ```
  552. **Discovery guarantees:**
  553. - Provider order in the TOML determines duplicate resolution (first provider wins); skipped duplicates emit warn-level logs
  554. - `/v1/models` lists bare discovered models first, followed by explicit `provider/model` entries
  555. - The watch channel keeps request handlers in sync without locking
  556. **How model resolution works:**
  557. 1. If the model name contains `/`, route to the specified provider (`openai/gpt-4`)
  558. 2. Otherwise look up the bare name in the discovered model map (`gpt-4` → provider)
  559. 3. Return "model not found" if the name is missing from both
  560. **Accessing models:**
  561. - **Discovered models**: Use bare names (e.g., `gpt-4`, `claude-3-opus`)
  562. - **Explicit models**: Use `provider/model` format (e.g., `openai/fast-model`)
  563. - Both formats share rate limits and telemetry accounting
  564. ##### Model Renaming
  565. You can rename models to provide custom identifiers for your users:
  566. ```toml
  567. [llm.providers.openai.models.gpt-4]
  568. rename = "smart-model" # Users will access this as "openai/smart-model"
  569. [llm.providers.openai.models.gpt-3-5-turbo]
  570. rename = "fast-model" # Users will access this as "openai/fast-model"
  571. ```
  572. This is useful for:
  573. - Creating business-friendly model names
  574. - Abstracting away provider-specific model names
  575. - Providing consistent naming across different providers
  576. ##### TOML Syntax for Model IDs
  577. Model IDs that contain dots must be quoted in TOML:
  578. ```toml
  579. # Correct - dots in model IDs require quotes
  580. [llm.providers.google.models."gemini-1.5-flash"]
  581. [llm.providers.google.models."gemini-1.5-pro"]
  582. # Also correct - no dots, no quotes needed
  583. [llm.providers.google.models.gemini-pro]
  584. ```
  585. #### Multiple Provider Configuration
  586. You can configure multiple instances of the same provider type with different names:
  587. ```toml
  588. # Primary OpenAI account
  589. [llm.providers.openai_primary]
  590. type = "openai"
  591. api_key = "{{ env.OPENAI_PRIMARY_KEY }}"
  592. [llm.providers.openai_primary.models.gpt-4]
  593. [llm.providers.openai_primary.models.gpt-3-5-turbo]
  594. # Secondary OpenAI account or Azure OpenAI
  595. [llm.providers.openai_secondary]
  596. type = "openai"
  597. api_key = "{{ env.OPENAI_SECONDARY_KEY }}"
  598. base_url = "https://my-azure-instance.openai.azure.com/v1"
  599. [llm.providers.openai_secondary.models.gpt-4]
  600. rename = "azure-gpt-4" # Distinguish from primary account
  601. # Anthropic
  602. [llm.providers.claude]
  603. type = "anthropic"
  604. api_key = "{{ env.ANTHROPIC_API_KEY }}"
  605. [llm.providers.claude.models.claude-3-opus-20240229]
  606. # Google Gemini
  607. [llm.providers.gemini]
  608. type = "google"
  609. api_key = "{{ env.GOOGLE_API_KEY }}"
  610. [llm.providers.gemini.models."gemini-1.5-flash"]
  611. ```
  612. #### Token Forwarding
  613. Nexus supports token forwarding, allowing users to provide their own API keys at request time instead of using the configured keys. This feature is opt-in and disabled by default.
  614. ##### Configuring Token Forwarding
  615. Enable token forwarding for any provider by setting `forward_token = true`:
  616. ```toml
  617. [llm.providers.openai]
  618. type = "openai"
  619. api_key = "{{ env.OPENAI_API_KEY }}" # Fallback key (optional with token forwarding)
  620. forward_token = true # Enable token forwarding for this provider
  621. [llm.providers.openai.models.gpt-4]
  622. [llm.providers.openai.models.gpt-3-5-turbo]
  623. [llm.providers.anthropic]
  624. type = "anthropic"
  625. # No api_key required when token forwarding is enabled
  626. forward_token = true
  627. [llm.providers.anthropic.models.claude-3-5-sonnet-20241022]
  628. [llm.providers.google]
  629. type = "google"
  630. api_key = "{{ env.GOOGLE_API_KEY }}"
  631. forward_token = false # Explicitly disabled (default)
  632. [llm.providers.google.models."gemini-1.5-flash"]
  633. ```
  634. ##### Using Token Forwarding
  635. When token forwarding is enabled for a provider, users can pass their own API key using the `X-Provider-API-Key` header:
  636. ```bash
  637. # Using your own OpenAI key
  638. curl -X POST http://localhost:8000/llm/v1/chat/completions \
  639. -H "Content-Type: application/json" \
  640. -H "X-Provider-API-Key: sk-your-openai-key" \
  641. -d '{
  642. "model": "openai/gpt-4",
  643. "messages": [{"role": "user", "content": "Hello"}]
  644. }'
  645. # Using your own Anthropic key
  646. curl -X POST http://localhost:8000/llm/v1/chat/completions \
  647. -H "Content-Type: application/json" \
  648. -H "X-Provider-API-Key: sk-ant-your-anthropic-key" \
  649. -d '{
  650. "model": "anthropic/claude-3-opus-20240229",
  651. "messages": [{"role": "user", "content": "Hello"}]
  652. }'
  653. ```
  654. ##### Token Forwarding Behavior
  655. - **When token forwarding is enabled (`forward_token = true`)**:
  656. - User-provided keys (via header) take priority
  657. - Falls back to configured key if no header is provided
  658. - Returns 401 error if neither key is available
  659. - **When token forwarding is disabled (`forward_token = false`, default)**:
  660. - Always uses the configured API key
  661. - Ignores the `X-Provider-API-Key` header
  662. - Returns 401 error if no configured key exists
  663. ##### Security Considerations
  664. - **OAuth2 Integration**: When OAuth2 is enabled, users must still authenticate with Nexus even when using token forwarding
  665. - **Key Validation**: API keys are validated by the provider's API
  666. - **No Logging**: User-provided keys are never logged
  667. - **HTTPS Recommended**: Always use HTTPS in production to protect API keys in transit
  668. ##### Provider Limitations
  669. - **AWS Bedrock**: Token forwarding is **not supported** for Bedrock providers. Bedrock uses AWS IAM credentials and request signing, which cannot be provided via simple API key headers. You must configure AWS credentials at the provider level (via environment variables, AWS profile, or explicit credentials in configuration).
  670. #### Header Transformation for LLM Providers
  671. Nexus supports header transformation for LLM providers, allowing you to forward, insert, remove, or rename headers when making requests to LLM APIs.
  672. ##### Provider-Level Headers
  673. Configure header rules at the provider level:
  674. ```toml
  675. [llm.providers.openai]
  676. type = "openai"
  677. api_key = "{{ env.OPENAI_API_KEY }}"
  678. # Forward headers from incoming requests
  679. [[llm.providers.openai.headers]]
  680. rule = "forward"
  681. name = "X-Request-ID"
  682. # Forward with a default value if not present
  683. [[llm.providers.openai.headers]]
  684. rule = "forward"
  685. name = "X-Trace-ID"
  686. default = "generated-trace-id"
  687. # Forward and rename
  688. [[llm.providers.openai.headers]]
  689. rule = "forward"
  690. name = "X-Custom-Header"
  691. rename = "X-OpenAI-Custom"
  692. # Insert static headers
  693. [[llm.providers.openai.headers]]
  694. rule = "insert"
  695. name = "X-OpenAI-Beta"
  696. value = "assistants=v2"
  697. # Remove headers
  698. [[llm.providers.openai.headers]]
  699. rule = "remove"
  700. name = "X-Internal-Secret"
  701. # Pattern-based forwarding (regex)
  702. [[llm.providers.openai.headers]]
  703. rule = "forward"
  704. pattern = "X-Debug-.*"
  705. # Pattern-based removal
  706. [[llm.providers.openai.headers]]
  707. rule = "remove"
  708. pattern = "X-Internal-.*"
  709. # Rename and duplicate (keeps original and adds renamed copy)
  710. [[llm.providers.openai.headers]]
  711. rule = "rename_duplicate"
  712. name = "X-User-ID"
  713. rename = "X-OpenAI-User"
  714. ```
  715. ##### Model-Level Headers
  716. Configure headers for specific models (overrides provider-level rules):
  717. ```toml
  718. [llm.providers.openai.models.gpt-4]
  719. # Model-specific headers override provider headers
  720. [[llm.providers.openai.models.gpt-4.headers]]
  721. rule = "insert"
  722. name = "X-Model-Config"
  723. value = "premium"
  724. [[llm.providers.openai.models.gpt-4.headers]]
  725. rule = "forward"
  726. pattern = "X-Premium-.*"
  727. ```
  728. ##### Header Rule Types
  729. 1. **forward**: Pass headers from incoming requests to the LLM provider
  730. - `name`: Single header name to forward
  731. - `pattern`: Regex pattern to match multiple headers
  732. - `default`: Optional default value if header is missing
  733. - `rename`: Optional new name for the forwarded header
  734. 2. **insert**: Add static headers to requests
  735. - `name`: Header name
  736. - `value`: Static value (supports `{{ env.VAR }}` substitution)
  737. 3. **remove**: Remove headers before sending to provider
  738. - `name`: Single header name to remove
  739. - `pattern`: Regex pattern to match headers to remove
  740. 4. **rename_duplicate**: Forward header with both original and new name
  741. - `name`: Original header name
  742. - `rename`: New header name for the duplicate
  743. - `default`: Optional default if header is missing
  744. ##### Important Notes
  745. - **AWS Bedrock**: Does not support custom headers due to SigV4 signing requirements
  746. - **Priority**: Model-level rules override provider-level rules
  747. - **Token Forwarding**: The `X-Provider-API-Key` header is handled separately for token forwarding
  748. - **Pattern Matching**: Patterns are case-insensitive regex expressions
  749. #### Using the LLM API
  750. Once configured, you can interact with LLM providers through Nexus's unified API:
  751. ##### List Available Models
  752. ```bash
  753. curl http://localhost:8000/llm/models
  754. ```
  755. Response:
  756. ```json
  757. {
  758. "object": "list",
  759. "data": [
  760. {
  761. "id": "openai_primary/gpt-4-turbo",
  762. "object": "model",
  763. "created": 1677651200,
  764. "owned_by": "openai"
  765. },
  766. {
  767. "id": "claude/claude-3-5-sonnet-20241022",
  768. "object": "model",
  769. "created": 1709164800,
  770. "owned_by": "anthropic"
  771. },
  772. {
  773. "id": "gemini/gemini-1.5-pro",
  774. "object": "model",
  775. "created": 1710000000,
  776. "owned_by": "google"
  777. }
  778. ]
  779. }
  780. ```
  781. ##### Tool Calling (Function Calling)
  782. Nexus supports advanced tool calling across all LLM providers, allowing models to invoke external functions with structured parameters. All providers use the standardized OpenAI tool calling format for consistency.
  783. **Basic Tool Calling Example:**
  784. ```bash
  785. curl -X POST http://localhost:8000/llm/chat/completions \
  786. -H "Content-Type: application/json" \
  787. -d '{
  788. "model": "anthropic/claude-3-5-sonnet-20241022",
  789. "messages": [
  790. {"role": "user", "content": "What'\''s the weather in San Francisco?"}
  791. ],
  792. "tools": [{
  793. "type": "function",
  794. "function": {
  795. "name": "get_weather",
  796. "description": "Get current weather for a location",
  797. "parameters": {
  798. "type": "object",
  799. "properties": {
  800. "location": {"type": "string", "description": "City and state"},
  801. "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
  802. },
  803. "required": ["location"]
  804. }
  805. }
  806. }],
  807. "tool_choice": "auto"
  808. }'
  809. ```
  810. **Tool Choice Options:**
  811. - `"auto"` (default): Model decides whether to call tools
  812. - `"none"`: Model won't call any tools
  813. - `"required"`: Model must call at least one tool
  814. - `{"type": "function", "function": {"name": "specific_function"}}`: Force a specific tool
  815. **Parallel Tool Calls:**
  816. When supported by the provider, models can call multiple tools simultaneously:
  817. ```json
  818. {
  819. "model": "openai/gpt-4",
  820. "messages": [{"role": "user", "content": "Get weather for NYC and LA"}],
  821. "tools": [/* tool definitions */],
  822. "parallel_tool_calls": true
  823. }
  824. ```
  825. **Tool Conversation Flow:**
  826. Tool calling creates a multi-turn conversation:
  827. 1. **User message** → asks a question requiring tool use
  828. 2. **Assistant message** → responds with tool calls (no content)
  829. 3. **Tool messages** → provide results from tool execution
  830. 4. **Assistant message** → synthesizes final response
  831. ```json
  832. {
  833. "messages": [
  834. {"role": "user", "content": "What's the weather in Paris?"},
  835. {
  836. "role": "assistant",
  837. "tool_calls": [{
  838. "id": "call_123",
  839. "type": "function",
  840. "function": {"name": "get_weather", "arguments": "{\"location\": \"Paris\"}"}
  841. }]
  842. },
  843. {
  844. "role": "tool",
  845. "tool_call_id": "call_123",
  846. "content": "Weather in Paris: 22°C, sunny"
  847. },
  848. {
  849. "role": "assistant",
  850. "content": "The weather in Paris is currently 22°C and sunny!"
  851. }
  852. ]
  853. }
  854. ```
  855. **Provider-Specific Tool Support:**
  856. All providers support the standardized format, but have different capabilities:
  857. - **OpenAI**: Full support including parallel calls and streaming tool calls
  858. - **Anthropic**: Tool calling with robust function definitions and tool choice controls
  859. - **Google**: Function calling with JSON schema validation and tool selection
  860. - **AWS Bedrock**: Tool calling support varies by model family (Claude, Nova, etc.)
  861. **Streaming Tool Calls:**
  862. Tool calls can be streamed just like regular responses:
  863. ```bash
  864. curl -X POST http://localhost:8000/llm/chat/completions \
  865. -H "Content-Type: application/json" \
  866. -d '{
  867. "model": "openai/gpt-4",
  868. "messages": [{"role": "user", "content": "Search for Python tutorials"}],
  869. "tools": [/* tool definitions */],
  870. "stream": true
  871. }'
  872. ```
  873. The stream will include tool call chunks as they're generated, followed by the tool execution results.
  874. ##### Chat Completions
  875. Send a chat completion request using the OpenAI-compatible format:
  876. ```bash
  877. curl -X POST http://localhost:8000/llm/chat/completions \
  878. -H "Content-Type: application/json" \
  879. -d '{
  880. "model": "openai_primary/gpt-4-turbo",
  881. "messages": [
  882. {"role": "system", "content": "You are a helpful assistant."},
  883. {"role": "user", "content": "Hello, how are you?"}
  884. ],
  885. "temperature": 0.7,
  886. "max_tokens": 150
  887. }'
  888. ```
  889. The model name format is `<provider_name>/<model_id>`. Nexus automatically routes the request to the appropriate provider and transforms the request/response as needed.
  890. ##### Streaming Responses
  891. Nexus supports streaming responses for all LLM providers using Server-Sent Events (SSE):
  892. ```bash
  893. curl -X POST http://localhost:8000/llm/chat/completions \
  894. -H "Content-Type: application/json" \
  895. -d '{
  896. "model": "anthropic/claude-3-5-sonnet-20241022",
  897. "messages": [
  898. {"role": "user", "content": "Write a short poem"}
  899. ],
  900. "stream": true,
  901. "max_tokens": 100
  902. }'
  903. ```
  904. When `stream: true` is set, the response will be streamed as Server-Sent Events with the following format:
  905. ```
  906. data: {"id":"msg_123","object":"chat.completion.chunk","created":1234567890,"model":"anthropic/claude-3-5-sonnet-20241022","choices":[{"index":0,"delta":{"role":"assistant","content":"Here"}}]}
  907. data: {"id":"msg_123","object":"chat.completion.chunk","created":1234567890,"model":"anthropic/claude-3-5-sonnet-20241022","choices":[{"index":0,"delta":{"content":" is"}}]}
  908. data: {"id":"msg_123","object":"chat.completion.chunk","created":1234567890,"model":"anthropic/claude-3-5-sonnet-20241022","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":25,"total_tokens":35}}
  909. data: [DONE]
  910. ```
  911. Streaming is supported for all providers (OpenAI, Anthropic, Google) and provides:
  912. - Real-time token delivery as they're generated
  913. - Consistent chunk format across all providers
  914. - Usage statistics in the final chunk
  915. - Standard SSE format compatible with OpenAI SDKs
  916. #### Provider-Specific Considerations
  917. ##### OpenAI
  918. - Supports all standard OpenAI models (GPT-3.5, GPT-4, etc.)
  919. - Compatible with Azure OpenAI endpoints
  920. - **Tool Calling**: Full support including parallel tool calls and streaming tool calls
  921. - **Function Definitions**: JSON schema support for parameters
  922. - **Tool Choice**: Supports all tool choice modes including specific function forcing
  923. - Supports streaming responses with Server-Sent Events (SSE)
  924. ##### Anthropic
  925. - System messages are automatically extracted and placed in the `system` field
  926. - Messages must alternate between user and assistant roles
  927. - Requires explicit `max_tokens` parameter (defaults to 4096 if not specified)
  928. - **Tool Calling**: Robust tool calling with function definitions and tool choice controls
  929. - **Tool Use**: Supports tool_use blocks with structured parameter validation
  930. - **Streaming Tools**: Tool calls can be streamed incrementally
  931. - Supports all Claude models (Opus, Sonnet, Haiku)
  932. - Supports streaming responses with Server-Sent Events (SSE)
  933. ##### Google
  934. - Assistant role is automatically mapped to "model" role
  935. - System messages are placed in the `systemInstruction` field
  936. - **Function Calling**: Native function calling with JSON schema validation
  937. - **Tool Selection**: Supports automatic tool selection and forced function calls
  938. - **Parameter Validation**: Strict parameter schema enforcement
  939. - Returns appropriate safety ratings when available
  940. - Supports streaming responses with Server-Sent Events (SSE)
  941. ##### AWS Bedrock
  942. - Uses the Bedrock conversation API for seamless integration
  943. - We have tested and verified Anthropic Claude, Amazon Nova, Meta Llama, Cohere Command, Mistral, and DeepSeek models
  944. - Other models may work, but we have not tested them. Please report any issues or successes.
  945. - **Tool Calling**: Support varies by model family:
  946. - **Anthropic Claude**: Full tool calling support through Bedrock
  947. - **Amazon Nova**: Native tool calling capabilities
  948. - **Meta Llama**: Function calling has issues, avoid using
  949. - **Cohere Command**: Tool use support
  950. - **Mistral**: Tools work well with Bedrock
  951. - **DeepSeek**: Tools do not work with Bedrock
  952. - Uses AWS SDK for authentication and request signing
  953. - Supports all Bedrock features including streaming and model-specific parameters
  954. - Regional endpoint selection based on configuration or AWS defaults
  955. - Model availability varies by AWS region
  956. #### Using Nexus with LLM SDKs
  957. Nexus provides an OpenAI-compatible API, making it easy to use with existing LLM SDKs and libraries. Simply point the SDK to your Nexus instance instead of the provider's API.
  958. ##### OpenAI SDK (Python)
  959. ```python
  960. from openai import OpenAI
  961. # Point to your Nexus instance
  962. client = OpenAI(
  963. base_url="http://localhost:8000/llm",
  964. api_key="your-service-token" # Use a JWT token if OAuth2 is enabled, or any string if not
  965. )
  966. # Use any configured provider/model
  967. response = client.chat.completions.create(
  968. model="anthropic/claude-3-5-sonnet-20241022",
  969. messages=[
  970. {"role": "user", "content": "Hello!"}
  971. ]
  972. )
  973. # Streaming works seamlessly
  974. stream = client.chat.completions.create(
  975. model="openai/gpt-4-turbo",
  976. messages=[
  977. {"role": "user", "content": "Write a poem"}
  978. ],
  979. stream=True
  980. )
  981. for chunk in stream:
  982. if chunk.choices[0].delta.content:
  983. print(chunk.choices[0].delta.content, end="")
  984. ```
  985. ##### OpenAI SDK (Node.js/TypeScript)
  986. ```typescript
  987. import OpenAI from 'openai';
  988. // Configure to use Nexus
  989. const openai = new OpenAI({
  990. baseURL: 'http://localhost:8000/llm',
  991. apiKey: 'your-service-token', // Use a JWT token if OAuth2 is enabled, or any string if not
  992. });
  993. // Use any provider through Nexus
  994. const response = await openai.chat.completions.create({
  995. model: 'google/gemini-1.5-pro',
  996. messages: [
  997. { role: 'user', content: 'Explain quantum computing' }
  998. ],
  999. });
  1000. // Streaming with any provider
  1001. const stream = await openai.chat.completions.create({
  1002. model: 'anthropic/claude-3-opus-20240229',
  1003. messages: [
  1004. { role: 'user', content: 'Write a story' }
  1005. ],
  1006. stream: true,
  1007. });
  1008. for await (const chunk of stream) {
  1009. process.stdout.write(chunk.choices[0]?.delta?.content || '');
  1010. }
  1011. ```
  1012. ##### LangChain Integration
  1013. ```python
  1014. from langchain_openai import ChatOpenAI
  1015. # Use Nexus as the LLM provider
  1016. llm = ChatOpenAI(
  1017. base_url="http://localhost:8000/llm",
  1018. api_key="your-service-token", # Use a JWT token if OAuth2 is enabled
  1019. model="openai/gpt-4-turbo"
  1020. )
  1021. # Works with any configured provider
  1022. claude = ChatOpenAI(
  1023. base_url="http://localhost:8000/llm",
  1024. api_key="your-service-token", # Use a JWT token if OAuth2 is enabled
  1025. model="anthropic/claude-3-5-sonnet-20241022"
  1026. )
  1027. ```
  1028. ##### cURL with jq for Command Line
  1029. ```bash
  1030. # Regular completion (with OAuth2 authentication if enabled)
  1031. curl -s http://localhost:8000/llm/chat/completions \
  1032. -H "Content-Type: application/json" \
  1033. -H "Authorization: Bearer your-jwt-token" \
  1034. -d '{
  1035. "model": "openai/gpt-4",
  1036. "messages": [{"role": "user", "content": "Hello"}]
  1037. }' | jq -r '.choices[0].message.content'
  1038. # Streaming with SSE parsing
  1039. curl -s http://localhost:8000/llm/chat/completions \
  1040. -H "Content-Type: application/json" \
  1041. -H "Authorization: Bearer your-jwt-token" \
  1042. -d '{
  1043. "model": "anthropic/claude-3-5-sonnet-20241022",
  1044. "messages": [{"role": "user", "content": "Write a haiku"}],
  1045. "stream": true
  1046. }' | grep "^data: " | sed 's/^data: //' | jq -r 'select(.choices != null) | .choices[0].delta.content // empty'
  1047. ```
  1048. ##### Authentication with OAuth2
  1049. When OAuth2 is enabled in your Nexus configuration, you must provide a valid JWT token:
  1050. ```python
  1051. # With OAuth2 enabled
  1052. client = OpenAI(
  1053. base_url="http://localhost:8000/llm",
  1054. api_key="eyJhbGciOiJSUzI1NiIs..." # Your JWT token
  1055. )
  1056. ```
  1057. Without OAuth2, the `api_key` field is still required by most SDKs but can be any non-empty string:
  1058. ```python
  1059. # Without OAuth2
  1060. client = OpenAI(
  1061. base_url="http://localhost:8000/llm",
  1062. api_key="dummy" # Any non-empty string works
  1063. )
  1064. ```
  1065. #### Error Handling
  1066. Nexus provides consistent error responses across all providers:
  1067. - **400 Bad Request**: Invalid request format or parameters
  1068. - **401 Unauthorized**: Missing or invalid API key
  1069. - **429 Too Many Requests**: Rate limit exceeded
  1070. - **500 Internal Server Error**: Provider API error or network issues
  1071. Example error response:
  1072. ```json
  1073. {
  1074. "error": {
  1075. "message": "Invalid model format: expected 'provider/model', got 'invalid-format'",
  1076. "type": "invalid_request_error",
  1077. "code": 400
  1078. }
  1079. }
  1080. ```
  1081. ### Telemetry
  1082. Nexus provides comprehensive observability through OpenTelemetry metrics, distributed tracing, and logs export. All telemetry features have zero overhead when the `[telemetry]` section is absent from configuration.
  1083. **Telemetry activation is controlled by exporters configuration:**
  1084. - **Metrics**: Enabled when OTLP exporters are configured
  1085. - **Tracing**: Enabled when OTLP exporters are configured
  1086. - **Logs**: Enabled when OTLP exporters are configured
  1087. Each telemetry signal (metrics, tracing, logs) can override the global exporters configuration with their own specific exporters.
  1088. #### Metrics Configuration
  1089. ```toml
  1090. [telemetry]
  1091. service_name = "nexus-production" # Optional, defaults to "nexus"
  1092. # Resource attributes for all telemetry (optional)
  1093. [telemetry.resource_attributes]
  1094. environment = "production"
  1095. region = "us-east-1"
  1096. # OTLP exporter for metrics
  1097. [telemetry.exporters.otlp]
  1098. enabled = true # Default: false
  1099. endpoint = "http://localhost:4317" # Default: http://localhost:4317
  1100. protocol = "grpc" # Default: grpc (options: grpc, http)
  1101. timeout = "60s" # Default: 60s
  1102. # Protocol-specific configuration
  1103. # For gRPC protocol (headers must be lowercase, cannot start with "grpc-"):
  1104. [telemetry.exporters.otlp.grpc.headers]
  1105. authorization = "Bearer {{ env.OTLP_TOKEN }}"
  1106. x-nexus-shard = "primary"
  1107. # TLS configuration for secure gRPC connections (all fields optional):
  1108. [telemetry.exporters.otlp.grpc.tls]
  1109. domain_name = "custom_name" # Domain name for TLS verification (SNI)
  1110. key = "/path/to/key.pem" # Path to client private key PEM file
  1111. cert = "/path/to/cert.pem" # Path to client certificate PEM file
  1112. ca = "/path/to/ca.pem" # Path to CA certificate PEM file
  1113. # For HTTP protocol (standard HTTP header names):
  1114. [telemetry.exporters.otlp.http.headers]
  1115. Authorization = "Bearer {{ env.OTLP_TOKEN }}"
  1116. X-Nexus-Shard = "primary"
  1117. # Batch export configuration (all optional with defaults)
  1118. [telemetry.exporters.otlp.batch_export]
  1119. scheduled_delay = "5s" # Default: 5s
  1120. max_queue_size = 2048 # Default: 2048
  1121. max_export_batch_size = 512 # Default: 512
  1122. max_concurrent_exports = 1 # Default: 1
  1123. ```
  1124. **See [Available Metrics](#available-metrics) for detailed metric definitions.**
  1125. #### Tracing Configuration
  1126. ```toml
  1127. [telemetry.tracing]
  1128. sampling = 0.15 # Sample 15% of requests (default: 0.15)
  1129. parent_based_sampler = false # Respect parent trace sampling decision (default: false)
  1130. # Collection limits (per span)
  1131. [telemetry.tracing.collect]
  1132. max_events_per_span = 128
  1133. max_attributes_per_span = 128
  1134. max_links_per_span = 128
  1135. max_attributes_per_event = 128
  1136. max_attributes_per_link = 128
  1137. # Trace context propagation formats
  1138. [telemetry.tracing.propagation]
  1139. trace_context = true # W3C Trace Context
  1140. aws_xray = false # AWS X-Ray format
  1141. # Optional: Override global OTLP exporter for traces
  1142. [telemetry.tracing.exporters.otlp]
  1143. enabled = true
  1144. endpoint = "http://localhost:4317"
  1145. protocol = "grpc" # or "http"
  1146. timeout = "60s"
  1147. # Protocol-specific headers
  1148. [telemetry.tracing.exporters.otlp.grpc.headers]
  1149. authorization = "Bearer {{ env.OTLP_TOKEN }}"
  1150. # TLS configuration (same format as global OTLP):
  1151. # [telemetry.tracing.exporters.otlp.grpc.tls]
  1152. # domain_name = "custom_name"
  1153. # key = "/path/to/key.pem"
  1154. # cert = "/path/to/cert.pem"
  1155. # ca = "/path/to/ca.pem"
  1156. # Or for HTTP:
  1157. # [telemetry.tracing.exporters.otlp.http.headers]
  1158. # Authorization = "Bearer {{ env.OTLP_TOKEN }}"
  1159. ```
  1160. **See [Distributed Tracing](#distributed-tracing) for span hierarchy and attributes.**
  1161. #### Logs Configuration
  1162. Nexus can export application logs to OpenTelemetry collectors, automatically correlating them with distributed traces. Logs are enabled when OTLP exporters are configured (either globally or specifically for logs).
  1163. ```toml
  1164. # Option 1: Use global OTLP exporter for all telemetry signals
  1165. # (logs will use the same endpoint as metrics and traces)
  1166. # Option 2: Configure a separate OTLP endpoint specifically for logs
  1167. [telemetry.logs.exporters.otlp]
  1168. enabled = true
  1169. endpoint = "http://logs-collector:4317" # Different endpoint for logs
  1170. protocol = "grpc" # or "http"
  1171. timeout = "30s"
  1172. # Protocol-specific headers
  1173. [telemetry.logs.exporters.otlp.grpc.headers]
  1174. authorization = "Bearer {{ env.OTLP_TOKEN }}"
  1175. # TLS configuration (same format as global OTLP):
  1176. # [telemetry.logs.exporters.otlp.grpc.tls]
  1177. # domain_name = "custom_name"
  1178. # key = "/path/to/key.pem"
  1179. # cert = "/path/to/cert.pem"
  1180. # ca = "/path/to/ca.pem"
  1181. # Or for HTTP:
  1182. # [telemetry.logs.exporters.otlp.http.headers]
  1183. # Authorization = "Bearer {{ env.OTLP_TOKEN }}"
  1184. # Logs are automatically correlated with active trace and span IDs
  1185. ```
  1186. When logs export is enabled, logs include:
  1187. - Automatic trace and span ID correlation for distributed debugging
  1188. - Log severity levels mapped from Rust log levels
  1189. - Source code location attributes (file, line, module)
  1190. - All logs from Nexus and its dependencies
  1191. **Note**: OpenTelemetry's own internal logs are filtered out to prevent recursion.
  1192. #### Available Metrics
  1193. All histograms also function as counters (count field tracks number of observations).
  1194. **HTTP Server Metrics:**
  1195. - `http.server.request.duration` (histogram)
  1196. - Attributes: `http.route`, `http.request.method`, `http.response.status_code`
  1197. **MCP Operation Metrics:**
  1198. - `mcp.tool.call.duration` (histogram)
  1199. - Attributes: `tool_name`, `tool_type` (builtin/downstream), `status` (success/error), `client.id`, `client.group`
  1200. - Additional for search: `keyword_count`, `result_count`
  1201. - Additional for execute: `server_name` (for downstream tools)
  1202. - Additional for errors: `error.type`:
  1203. - `parse_error` - Invalid JSON (-32700)
  1204. - `invalid_request` - Not a valid request (-32600)
  1205. - `method_not_found` - Method/tool does not exist (-32601)
  1206. - `invalid_params` - Invalid method parameters (-32602)
  1207. - `internal_error` - Internal server error (-32603)
  1208. - `rate_limit_exceeded` - Rate limit hit (-32000)
  1209. - `server_error` - Other server errors (-32001 to -32099)
  1210. - `unknown` - Any other error code
  1211. - `mcp.tools.list.duration` (histogram)
  1212. - Attributes: `method` (list_tools), `status`, `client.id`, `client.group`
  1213. - `mcp.prompt.request.duration` (histogram)
  1214. - Attributes: `method` (list_prompts/get_prompt), `status`, `client.id`, `client.group`
  1215. - Additional for errors: `error.type` (same values as above)
  1216. - `mcp.resource.request.duration` (histogram)
  1217. - Attributes: `method` (list_resources/read_resource), `status`, `client.id`, `client.group`
  1218. - Additional for errors: `error.type` (same values as above)
  1219. **LLM Operation Metrics:**
  1220. - `gen_ai.client.operation.duration` (histogram)
  1221. - Tracks the total duration of LLM chat completion operations
  1222. - Attributes:
  1223. - `gen_ai.system` (always "nexus.llm")
  1224. - `gen_ai.operation.name` (always "chat.completions")
  1225. - `gen_ai.request.model` (e.g., "openai/gpt-4")
  1226. - `gen_ai.response.finish_reason` (stop/length/tool_calls/content_filter)
  1227. - `client.id` (from x-client-id header)
  1228. - `client.group` (from x-client-group header)
  1229. - `error.type` (for failed requests):
  1230. - `invalid_request` - Malformed request
  1231. - `authentication_failed` - Invalid API key
  1232. - `insufficient_quota` - Quota exceeded
  1233. - `model_not_found` - Unknown model
  1234. - `rate_limit_exceeded` - Provider or token rate limit hit
  1235. - `streaming_not_supported` - Streaming unavailable for model
  1236. - `invalid_model_format` - Incorrect model name format
  1237. - `provider_not_found` - Unknown provider
  1238. - `internal_error` - Server error
  1239. - `provider_api_error` - Upstream provider error
  1240. - `connection_error` - Network failure
  1241. - `gen_ai.client.time_to_first_token` (histogram)
  1242. - Tracks time until first token in streaming responses
  1243. - Attributes: `gen_ai.system`, `gen_ai.operation.name`, `gen_ai.request.model`, `client.id`, `client.group`
  1244. - `gen_ai.client.input.token.usage` (counter)
  1245. - Cumulative input token consumption
  1246. - Attributes: `gen_ai.system`, `gen_ai.request.model`, `client.id`, `client.group`
  1247. - `gen_ai.client.output.token.usage` (counter)
  1248. - Cumulative output token consumption
  1249. - Attributes: `gen_ai.system`, `gen_ai.request.model`, `client.id`, `client.group`
  1250. - `gen_ai.client.total.token.usage` (counter)
  1251. - Cumulative total token consumption (input + output)
  1252. - Attributes: `gen_ai.system`, `gen_ai.request.model`, `client.id`, `client.group`
  1253. **Redis Storage Metrics (when rate limiting with Redis is enabled):**
  1254. - `redis.command.duration` (histogram)
  1255. - Tracks Redis command execution times
  1256. - Attributes:
  1257. - `operation` - The Redis operation type:
  1258. - `check_and_consume` - HTTP rate limit checking
  1259. - `check_and_consume_tokens` - Token-based rate limit checking
  1260. - `status` (success/error)
  1261. - `tokens` (for token operations) - Number of tokens checked
  1262. - `redis.pool.connections.in_use` (gauge)
  1263. - Current number of connections in use from the pool
  1264. - No additional attributes
  1265. - `redis.pool.connections.available` (gauge)
  1266. - Current number of available connections in the pool
  1267. - No additional attributes
  1268. ### Distributed Tracing
  1269. **Configuration**: See [Tracing Configuration](#tracing-configuration) in the Telemetry section.
  1270. Distributed tracing provides detailed insights into request flows across all components, helping you understand latency, identify bottlenecks, and debug issues in production.
  1271. #### Trace Context Propagation
  1272. Nexus supports multiple trace context propagation formats for incoming requests:
  1273. - **W3C Trace Context**: Standard `traceparent` and `tracestate` headers (enabled by default)
  1274. - **AWS X-Ray**: `X-Amzn-Trace-Id` header for AWS services integration
  1275. When enabled, Nexus will:
  1276. 1. Extract trace context from incoming HTTP requests
  1277. 2. Continue the trace with the parent context or start a new trace
  1278. 3. Create spans for all internal operations
  1279. **Note**: Nexus is a terminal node for traces - it does not propagate trace context to downstream MCP servers or LLM providers.
  1280. #### Span Hierarchy
  1281. Nexus creates a hierarchical span structure for each request:
  1282. ```
  1283. HTTP Request (root span)
  1284. ├── Redis Rate Limit Check (redis:check_and_consume:global/ip) - HTTP-level rate limits
  1285. ├── MCP Operation
  1286. │ ├── Redis Rate Limit Check (redis:check_and_consume:server/tool) - MCP-specific rate limits
  1287. │ ├── Tool Search (index:search)
  1288. │ └── Tool Execution
  1289. │ └── Downstream Operations (downstream:execute, downstream:call_tool)
  1290. └── LLM Operation (llm:chat_completion or llm:chat_completion_stream)
  1291. └── Token Rate Limit Check (redis:check_and_consume_tokens) - Token-based rate limits
  1292. ```
  1293. #### Span Attributes
  1294. Each span includes semantic attributes following OpenTelemetry conventions:
  1295. **HTTP Spans:**
  1296. - `http.request.method` - HTTP method (GET, POST, etc.)
  1297. - `http.route` - Matched route pattern
  1298. - `http.response.status_code` - Response status code
  1299. - `url.scheme` - URL scheme (http/https)
  1300. - `server.address` - Host header value
  1301. - `url.full` - Full request URL
  1302. - `client.id` - Client identifier (if configured)
  1303. - `client.group` - Client group (if configured)
  1304. - `error` - Set to "true" if request failed
  1305. **MCP Spans:**
  1306. - `mcp.method` - MCP method name (tools/list, tools/call, etc.)
  1307. - `mcp.tool.name` - Tool being called
  1308. - `mcp.tool.type` - Tool type (builtin/downstream)
  1309. - `mcp.transport` - Transport type (stdio/http)
  1310. - `mcp.auth_forwarded` - Whether auth was forwarded
  1311. - `client.id` - Client identifier (if configured)
  1312. - `client.group` - Client group (if configured)
  1313. - `mcp.search.keywords` - Search keywords (for search tool)
  1314. - `mcp.search.keyword_count` - Number of search keywords
  1315. - `mcp.execute.target_tool` - Tool being executed
  1316. - `mcp.execute.target_server` - Server hosting the tool
  1317. - `mcp.error.code` - Error code if operation failed
  1318. **LLM Spans:**
  1319. - `gen_ai.request.model` - Model identifier
  1320. - `gen_ai.request.max_tokens` - Max tokens requested
  1321. - `gen_ai.request.temperature` - Temperature parameter
  1322. - `gen_ai.request.has_tools` - Whether tools were provided
  1323. - `gen_ai.request.tool_count` - Number of tools provided
  1324. - `gen_ai.response.model` - Model used for response
  1325. - `gen_ai.response.finish_reason` - Completion reason
  1326. - `gen_ai.usage.input_tokens` - Input token count
  1327. - `gen_ai.usage.output_tokens` - Output token count
  1328. - `gen_ai.usage.total_tokens` - Total token count
  1329. - `llm.stream` - Whether streaming was used
  1330. - `llm.auth_forwarded` - Whether auth was forwarded
  1331. - `client.id` - Client identifier (if configured)
  1332. - `client.group` - Client group (if configured)
  1333. - `error.type` - Error type if operation failed
  1334. **Redis Spans:**
  1335. - `redis.operation` - Operation type (check_and_consume or check_and_consume_tokens)
  1336. - `rate_limit.scope` - Scope (global/ip/server/tool/token)
  1337. - `rate_limit.limit` - Request/token limit
  1338. - `rate_limit.interval_ms` - Time window in milliseconds
  1339. - `rate_limit.tokens` - Number of tokens (for token operations)
  1340. - `rate_limit.allowed` - Whether request was allowed
  1341. - `rate_limit.retry_after_ms` - Retry delay if rate limited
  1342. - `redis.pool.size` - Connection pool size
  1343. - `redis.pool.available` - Available connections
  1344. - `redis.pool.in_use` - Connections in use
  1345. - `client.address_hash` - Hashed IP for privacy (per-IP limits)
  1346. - `llm.provider` - Provider name (token limits)
  1347. - `llm.model` - Model name (token limits)
  1348. - `mcp.server` - Server name (per-server limits)
  1349. - `mcp.tool` - Tool name (per-tool limits)
  1350. #### Sampling
  1351. Nexus supports two sampling strategies:
  1352. **Fixed-Rate Sampling** (default): Randomly samples a percentage of traces based on the configured rate.
  1353. ```toml
  1354. [telemetry.tracing]
  1355. sampling = 0.15 # Sample 15% of requests (0.0 to 1.0)
  1356. ```
  1357. **Parent-Based Sampling**: When enabled, Nexus respects the sampling decision from parent traces in distributed systems.
  1358. ```toml
  1359. [telemetry.tracing]
  1360. sampling = 0.15 # Default sampling rate for root traces
  1361. parent_based_sampler = true # Respect parent's sampling decision
  1362. ```
  1363. With `parent_based_sampler = true`:
  1364. - If a parent trace is sampled (W3C traceparent flag=01 or X-Ray Sampled=1), the child trace will be sampled
  1365. - If a parent trace is not sampled (flag=00 or Sampled=0), the child trace will not be sampled
  1366. - If no parent exists, the `sampling` rate is used
  1367. This ensures consistent sampling decisions across distributed traces, preventing incomplete trace data where some spans are sampled while others are not.
  1368. When a parent trace context is provided in request headers (via W3C Trace Context or AWS X-Ray), Nexus will continue that trace.
  1369. #### Performance Considerations
  1370. - **Zero Overhead When Disabled**: When no OTLP exporters are configured, no spans are created
  1371. - **Efficient Sampling**: Unsampled traces have minimal overhead
  1372. - **Batched Export**: Traces are batched before export to reduce network overhead
  1373. - **Configurable Limits**: Adjust collection limits to balance detail vs. resource usage
  1374. #### Integration with APM Tools
  1375. Nexus traces can be exported to any OpenTelemetry-compatible APM tool:
  1376. - **Jaeger**: Full trace visualization and analysis
  1377. - **Zipkin**: Distributed tracing system
  1378. - **AWS X-Ray**: Native AWS integration with X-Ray propagation
  1379. - **Datadog**: APM and distributed tracing
  1380. - **New Relic**: Application performance monitoring
  1381. - **Grafana Tempo**: Cloud-native tracing backend
  1382. Example Jaeger configuration:
  1383. ```toml
  1384. [telemetry.tracing.exporters.otlp]
  1385. enabled = true
  1386. endpoint = "http://jaeger:4317"
  1387. protocol = "grpc"
  1388. ```
  1389. ## Adding to AI Assistants
  1390. ### Cursor
  1391. Add to your Cursor settings:
  1392. 1. Open Cursor Settings (Cmd+, on macOS)
  1393. 2. Search for "Model Context Protocol"
  1394. 3. Enable MCP support
  1395. 4. Add to the MCP server configuration:
  1396. ```json
  1397. {
  1398. "nexus": {
  1399. "transport": {
  1400. "type": "http",
  1401. "url": "http://localhost:8000/mcp"
  1402. }
  1403. }
  1404. }
  1405. ```
  1406. Make sure Nexus is running on `localhost:8000` (or adjust the URL accordingly).
  1407. ### Claude Code
  1408. Add to your Claude Code configuration:
  1409. 1. Open Claude Code and run the command:
  1410. ```bash
  1411. claude mcp add --transport http nexus http://localhost:8000/mcp
  1412. ```
  1413. 2. Or add it to your project's `.mcp.json` file:
  1414. ```json
  1415. {
  1416. "mcpServers": {
  1417. "nexus": {
  1418. "type": "http",
  1419. "url": "http://localhost:8000/mcp"
  1420. }
  1421. }
  1422. }
  1423. ```
  1424. 3. Verify the connection:
  1425. ```bash
  1426. claude mcp list
  1427. ```
  1428. Make sure Nexus is running before starting Claude Code.
  1429. ### Codex CLI
  1430. Configure Codex CLI to talk to Nexus by editing `~/.codex/config.toml`:
  1431. > Note: Nexus serves its OpenAI-compatible endpoint at `/llm/openai/`; Codex expects the `/v1` suffix. Make sure the `base_url` ends with `/v1`.
  1432. ```toml
  1433. [model_providers.nexus]
  1434. name = "Nexus AI router"
  1435. base_url = "http://127.0.0.1:8000/llm/openai/v1"
  1436. wire_api = "chat"
  1437. query_params = {}
  1438. ```
  1439. - `base_url` must point to the Nexus OpenAI-compatible endpoint and include the `/v1` suffix (adjust host/port if Nexus runs elsewhere).
  1440. - `query_params` can stay empty, but the table must exist to satisfy Codex's schema.
  1441. Start Codex with a Nexus-managed model:
  1442. ```bash
  1443. codex -c model="anthropic/claude-3-5-haiku-latest" -c model_provider=nexus
  1444. ```
  1445. Swap the model name for any provider/model pair that you have configured in Nexus.
  1446. ## How It Works
  1447. ### MCP Tool Aggregation
  1448. Nexus provides two main tools to AI assistants:
  1449. 1. **`search`**: A context-aware tool search that uses fuzzy matching to find relevant tools across all connected MCP servers
  1450. 2. **`execute`**: Executes a specific tool with the provided parameters
  1451. When an AI assistant connects to Nexus, it can:
  1452. 1. Search for tools using natural language queries
  1453. 2. Discover tool names, descriptions, and required parameters
  1454. 3. Execute tools from any connected MCP server
  1455. All tools from downstream servers are namespaced with their server name (e.g., `github__search_code`, `filesystem__read_file`).
  1456. ### LLM Provider Routing
  1457. Nexus acts as a unified gateway for multiple LLM providers:
  1458. 1. **Model Discovery**: Lists all available models from configured providers with consistent naming
  1459. 2. **Request Routing**: Automatically routes requests to the correct provider based on model name
  1460. 3. **Format Translation**: Converts between OpenAI's API format and provider-specific formats
  1461. 4. **Response Normalization**: Returns consistent response format regardless of provider
  1462. Models are namespaced with their provider name (e.g., `openai/gpt-4`, `anthropic/claude-3-opus-20240229`).
  1463. ### STDIO Server Integration
  1464. STDIO servers are spawned as child processes and communicate via JSON-RPC over standard input/output:
  1465. 1. **Process Management**: Nexus automatically manages the lifecycle of STDIO server processes
  1466. 2. **Tool Discovery**: Tools from STDIO servers are discovered dynamically and indexed for search
  1467. 3. **Error Handling**: If a STDIO process crashes or outputs invalid JSON, appropriate errors are returned
  1468. 4. **Environment Isolation**: Each STDIO server runs in its own process with configurable environment
  1469. ## Example Usage
  1470. Once configured, AI assistants can interact with Nexus like this:
  1471. 1. **Search for tools**:
  1472. ```
  1473. User: "I need to search for code on GitHub"
  1474. Assistant: Let me search for GitHub-related tools...
  1475. [Calls search with keywords: ["github", "code", "search"]]
  1476. ```
  1477. 2. **Execute tools**:
  1478. ```
  1479. Assistant: I found the `github__search_code` tool. Let me search for your query...
  1480. [Calls execute with name: "github__search_code" and appropriate arguments]
  1481. ```
  1482. ## Common STDIO Server Examples
  1483. ### Python MCP Server
  1484. ```toml
  1485. [mcp.servers.python_tools]
  1486. cmd = ["python", "-m", "my_mcp_server"]
  1487. env = { PYTHONPATH = "/opt/mcp", PYTHONUNBUFFERED = "1" }
  1488. stderr = "inherit" # See Python output during development
  1489. ```
  1490. ### Node.js MCP Server
  1491. ```toml
  1492. [mcp.servers.node_tools]
  1493. cmd = ["node", "mcp-server.js"]
  1494. cwd = "/path/to/project"
  1495. env = { NODE_ENV = "production" }
  1496. ```
  1497. ### Using npx packages
  1498. ```toml
  1499. [mcp.servers.filesystem]
  1500. cmd = ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/home/user"]
  1501. ```
  1502. ## Troubleshooting STDIO Servers
  1503. ### Server doesn't start
  1504. - **Check executable path**: Ensure the command exists and is executable
  1505. - **View stderr output**: Set `stderr = "inherit"` temporarily to see error messages
  1506. - **Verify JSON-RPC output**: The server must output valid JSON-RPC on stdout
  1507. - **Check working directory**: Ensure `cwd` path exists if specified
  1508. ### Tools not appearing
  1509. - **Wait for initialization**: STDIO servers may take a moment to start
  1510. - **Use search**: STDIO tools only appear in search results, not in the base tool list
  1511. - **Check server logs**: Enable stderr logging to see if the server is responding to tool list requests
  1512. ## Security Considerations
  1513. - Always use environment variables for sensitive tokens
  1514. - Enable TLS verification for production deployments
  1515. - Use CORS configuration to restrict access
  1516. - Configure OAuth2 authentication for production deployments
  1517. - Ensure JWKs URLs use HTTPS in production
  1518. - Validate JWT token issuer and audience claims
  1519. - Keep your MCP servers and Nexus updated
  1520. - Be cautious when running STDIO servers with elevated privileges
  1521. - Validate and sanitize any user input passed to STDIO server commands
  1522. ### OAuth2 Security
  1523. When using OAuth2 authentication:
  1524. 1. **Use HTTPS**: Always use HTTPS for JWKs URLs and protected resources in production
  1525. 2. **Validate Claims**: Configure `expected_issuer` and `expected_audience` to validate JWT claims
  1526. 3. **Metadata Endpoint**: The `/.well-known/oauth-protected-resource` endpoint provides OAuth2 metadata and is publicly accessible
  1527. 4. **Health Checks**: The `/health` endpoint bypasses OAuth2 authentication for monitoring systems
  1528. ## License
  1529. Nexus is licensed under the Mozilla Public License 2.0 (MPL-2.0). See the LICENSE file for details.
  1530. ## Contributing
  1531. See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to contribute to Nexus.
  1532. ## Support
  1533. - Documentation: [https://nexusrouter.com/docs](https://nexusrouter.com/docs)
  1534. - Issues: [https://github.com/grafbase/nexus/issues](https://github.com/grafbase/nexus/issues)
  1535. - Discord: [Grafbase Discord](https://discord.gg/grafbase)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.