Skip to content
Back to search

UnrealGenAISupport

Unreal Engine plugin for LLM/GenAI models & MCP UE5 server. OpenAI GPT-5, Deepseek R1, Claude Opus/Sonnet, Gemini 3, Grok 4, Alibaba Qwen, Kimi, ElevenLabs TTS, Inworld, OpenRouter, Groq, GLM, Ollama, Local, Meshy, Tripo, Hunyuan3D, Rodin, fal, Dashscope, Seedream. NPC AI, agentic, chat, 3D gen, TTS, multimodal, image gen. UnrealMCP/UnrealClaude

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/prajwalshettydev--unrealgenaisupport/badge.svg)](/listings/prajwalshettydev--unrealgenaisupport)

HTML

<a href="/listings/prajwalshettydev--unrealgenaisupport"><img src="/api/listings/prajwalshettydev--unrealgenaisupport/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

For Claude Desktop App:

`claude_desktop_config.json` file in Claude Desktop App's installation directory. (might ask claude where its located for your platform!) The file will look something like this:

{
    "mcpServers": {
      "unreal-handshake": {
        "command": "python",
        "args": ["<your_project_directoy_path>/Plugins/GenerativeAISupport/Content/Python/mcp_server.py"],
        "env": {
          "UNREAL_HOST": "localhost",
          "UNREAL_PORT": "9877" 
        }
      }
    }
}

For Claude Code CLI:

`.mcp.json` file in your project root directory. The file will look something like this:

{
    "mcpServers": {
      "unreal-handshake": {
        "type": "stdio",
        "command": "python",
        "args": ["<your_project_directoy_path>/Plugins/GenerativeAISupport/Content/Python/mcp_server.py"],
        "env": {
          "UNREAL_HOST": "localhost",
          "UNREAL_PORT": "9877"
        }
      }
    }
}

For Cursor IDE:

`.cursor/mcp.json` file in your project directory. The file will look something like this:

{
    "mcpServers": {
      "unreal-handshake": {
        "command": "python",
        "args": ["<your_project_directoy_path>/Plugins/GenerativeAISupport/Content/Python/mcp_server.py"],
        "env": {
          "UNREAL_HOST": "localhost",
          "UNREAL_PORT": "9877"
        }
      }
    }
}

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 660 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ![Unreal Engine](https://img.shields.io/badge/Unreal%20Engine-5.4%20to%205.7+-313131?style=flat&logo=unrealengine&logoColor=white)
  2. ![Fab Plugins](https://img.shields.io/badge/Fab%20Plugins-UE%205.1%20to%205.7+-0E8A16?style=flat&logo=unrealengine&logoColor=white)
  3. ![C++](https://img.shields.io/badge/C%2B%2B-17-00599C?style=flat&logo=c%2B%2B&logoColor=white)
  4. ![Platform](https://img.shields.io/badge/Platform-Win64%20%7C%20Linux%20%7C%20Mac%20%7C%20Mobile%20%7C%20Consoles-0078D6?style=flat&logo=windows&logoColor=white)
  5. ![AI Models](https://img.shields.io/badge/AI%20Models-200%2B-FF6F00?style=flat&logo=openai&logoColor=white)
  6. ![MCP](https://img.shields.io/badge/MCP-Supported-8A2BE2?style=flat)
  7. [![License: MIT](https://img.shields.io/badge/License-MIT-007EC7?style=flat)](LICENSE)
  8. [![Discord](https://img.shields.io/badge/Discord-Join%20Community-7289DA?style=flat&logo=discord&logoColor=white)](https://discord.com/invite/KBWmkCKv5U)
  9. ## Usage Examples:
  10. #### MCP Example:
  11. Claude spawning scene objects and controlling their transformations and materials, generating blueprints, functions, variables, adding components, running python scripts etc.
  12. <img src="Docs/UnrealMcpDemo.gif" width="480"/>
  13. #### API Example:
  14. A project called become human, where NPCs are OpenAI agentic instances. Built using this plugin.
  15. ![Become Human](Docs/BHDemoGif.gif)
  16. # Unreal Engine Generative AI Support Plugin:
  17. Every month, hundreds of new AI models are released by various organizations, making it hard to keep up with the latest advancements.
  18. The "Unreal Engine Generative AI Support Plugin" allows you to focus on game development without worrying about the LLM/GenAI integration layer.
  19. ## Current Progress:
  20. **This plugin will continue to get updates with the latest features and models. Contributions are welcome.**
  21. **For production-ready alternatives with 200+ AI models, guaranteed stability, automated testing, and UE 5.1-5.7+ support, check out the pro plugins below.**
  22. **This free plugin covers many use cases (including the examples above) and you can use it for free, forever.**
  23. ### Pro Plugins on Fab.com
  24. <table>
  25. <tr>
  26. <td align="center" width="50%">
  27. <a href="https://muddyterrain.com/t/genai-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=genai-plugin">
  28. <img src="https://res.cloudinary.com/dqq9t4hyy/image/upload/q_60/v1772931701/MainBanners_4_d9nmqo.webp" width="100%" alt="Gen AI for Unreal - OpenAI GPT Anthropic Claude Gemini Grok ElevenLabs"/>
  29. </a>
  30. <p><b><a href="https://muddyterrain.com/t/genai-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=genai-plugin">Gen AI Pro</a></b><br/>
  31. <code>gpt-5.4</code> <code>claude-opus-4-6</code> <code>gemini-3.1-pro</code> <code>grok-4.1</code> <code>o3-pro</code> <code>eleven_v3</code><br/>
  32. Chat, Vision, Streaming, Realtime, Image Gen, TTS, Structured Output, Tool Use</p>
  33. </td>
  34. <td align="center" width="50%">
  35. <a href="https://muddyterrain.com/t/genai-china-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=genai-china-plugin">
  36. <img src="https://res.cloudinary.com/dqq9t4hyy/image/upload/q_60/v1773874915/MainBanners_6_tw1xai.webp" width="100%" alt="Gen AI Chinese Models - Alibaba Qwen DeepSeek Moonshot Kimi ByteDance ZhipuAI"/>
  37. </a>
  38. <p><b><a href="https://muddyterrain.com/t/genai-china-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=genai-china-plugin">Gen AI Pro China</a></b><br/>
  39. <code>qwen3.5-plus</code> <code>kimi-k2.5</code> <code>glm-5</code> <code>seed-2-0</code> <code>ernie-5.0</code> <code>seedream-4</code><br/>
  40. Worldwide support, Chat, Vision, Image Gen, TTS, Reasoning</p>
  41. </td>
  42. </tr>
  43. <tr>
  44. <td align="center" width="50%">
  45. <a href="https://muddyterrain.com/t/genai-model-generator-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=gen3d-plugin">
  46. <img src="https://res.cloudinary.com/dqq9t4hyy/image/upload/q_60/v1774387242/MainBanners_9_vdxmdm.webp" width="100%" alt="Gen AI Model Generator - Meshy Tripo Hunyuan3D Trellis Rodin 3D Generation"/>
  47. </a>
  48. <p><b><a href="https://muddyterrain.com/t/genai-model-generator-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=gen3d-plugin">GenAI Model Generator</a></b><br/>
  49. <code>meshy-6</code> <code>tripo-v2.5</code> <code>hunyuan3d-v3.1</code> <code>trellis-2</code> <code>rodin-gen-2</code><br/>
  50. Text-to-3D, Image-to-3D, Remesh, PBR Textures, Retexture, Auto-Rigging</p>
  51. </td>
  52. <td align="center" width="50%">
  53. <a href="https://muddyterrain.com/t/genai-llama-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=genai-llama-plugin">
  54. <img src="https://res.cloudinary.com/dqq9t4hyy/image/upload/q_60/v1774520082/MainBanners_10_gniskq.webp" width="100%" alt="Gen AI Llama - Ollama vLLM LM Studio GGUF Local LLMs Unreal Engine"/>
  55. </a>
  56. <p><b><a href="https://muddyterrain.com/t/genai-llama-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=genai-llama-plugin">Gen AI Llama</a></b><br/>
  57. <code>Ollama</code> <code>vLLM</code> <code>LM Studio</code> <code>GGUF</code> <code>Jan</code> <code>LocalAI</code> <code>GPT-oss</code> <code>LLaVA</code><br/>
  58. Local LLMs inside your Unreal Engine game - no API keys needed</p>
  59. </td>
  60. </tr>
  61. </table>
  62. ---
  63. ### Free Plugin (This Repo) - LLM/GenAI API Support:
  64. - **OpenAI**: Chat (`gpt-4.1`, `gpt-4.1-mini`, `o4-mini`, `o3`, `o3-pro`), Structured Outputs
  65. - **Anthropic Claude**: Chat (`claude-4-latest`, `claude-3-7-sonnet`, `claude-3-5-sonnet`, `claude-3-5-haiku`)
  66. - **XAI Grok**: Chat (`grok-3-latest`, `grok-3-mini-beta`)
  67. - **DeepSeek**: Chat (`deepseek-chat` V3.1), Reasoning (`deepseek-reasoning-r1`)
  68. - **Local AI**: [unreal-ollama (MIT)](https://github.com/MuddyTerrain/unreal-ollama) - `gpt-oss`, `qwen3-vl` and more
  69. <details>
  70. <summary><b>Full Model List (200+ models across all plugins)</b></summary>
  71. #### Gen AI Pro
  72. - **OpenAI**: `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.3-codex`, `gpt-5.2`, `gpt-5.1`, `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-4.1`, `o4-mini`, `o3`, `o3-pro` | Responses API, Vision, Realtime (`gpt-realtime`), Image Gen (`gpt-image-1.5`, `dall-e-3`), TTS (`gpt-4o-mini-tts`, `whisper-1`), Streaming, Function Calling, Multimodal
  73. - **Anthropic**: `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-4.5-opus`, `claude-4.5-sonnet`, `claude-4.5-haiku` | Extended Thinking, Vision, Tool Use
  74. - **Google Gemini**: `gemini-3.1-pro`, `gemini-3.1-flash-lite`, `gemini-2.5-pro`, `gemini-2.5-flash` | Imagen (`imagen-4.0-generate`, `imagen-4.0-ultra`), Realtime, TTS, Multimodal
  75. - **XAI Grok**: `grok-4.1`, `grok-4`, `grok-4-eu`, `grok-code-fast-1`, `grok-3` | Vision, Streaming, Reasoning
  76. - **ElevenLabs**: TTS (`eleven_v3`, `eleven_turbo_v2_5`), Transcription (`scribe_v2`), Sound Effects (`eleven_text_to_sound_v2`)
  77. - **Inworld AI**: TTS (`inworld-tts-1.5-max`, `inworld-tts-1.5-mini`)
  78. - **OpenAI Compatible Mode**: Alibaba Qwen, Mistral, Groq, OpenRouter, Meta Llama, GLM-4, Ollama
  79. #### Gen AI Pro China
  80. - **Alibaba Qwen**: `qwen3.5-plus`, `qwen3.5-flash`, `qwen3-max`, `qwen3-coder-plus` | Multimodal (`qwen-omni-turbo`, `qwen-vl-max`), Image Gen (`qwen-image`, `wan2.2-t2i-plus`), TTS (`qwen3-tts-flash`)
  81. - **Moonshot/Kimi**: `kimi-k2.5`, `kimi-k2-thinking`, `kimi-k2-thinking-turbo` | Multimodal (`kimi-latest`)
  82. - **ByteDance**: `seed-2-0-mini`, `seed-1-8`, `skylark-pro-250415` | Vision (`skylark-vision`), Image Gen (`seedream-4-0-250828`)
  83. - **ZhipuAI**: `glm-5`, `glm-4.7`, `glm-4.7-flash` | Multimodal (`glm-4.6v`)
  84. - **Baidu**: `ernie-5.0-8k`, `ernie-4.5-8k`, `ernie-x1-32k`
  85. #### GenAI Model Generator (3D)
  86. - **Meshy AI**: `meshy-6` - Text-to-3D, Image-to-3D, Retexture, Auto-Rigging
  87. - **Tripo AI**: `tripo-v2.5` - Text-to-3D, Image-to-3D
  88. - **Hunyuan3D** (Tencent): `hunyuan3d-v3.1-pro` Text-to-3D, `hunyuan3d-v2.1` Image-to-3D
  89. - **TripoSR**: Image-to-3D (fast, <1s)
  90. - **Rodin** (Hyper3D): `rodin-gen-2` - Text/Image-to-3D with PBR materials
  91. - **Trellis 2** (Microsoft): Image-to-3D with PBR materials
  92. - **Google Gemini**: `gemini-3.1-flash` - PBR texture generation
  93. </details>
  94. ### Additional Features:
  95. - Plugin Example Project [here](https://github.com/prajwalshettydev/unreal-llm-api-test-project)
  96. - Version Control: Git Submodule Support, Perforce (in progress)
  97. - Lightweight: No External Dependencies, Exclude MCP from build
  98. - Testing across platforms and engine versions (available in pro plugins)
  99. ### Unreal MCP (Model Control Protocol):
  100. > **Other MCP options:** Epic Games is working on an official Unreal MCP integration for UE 5.8+. There's also [UnrealClaude](https://github.com/Natfii/UnrealClaude) (MIT) by Natfii - a standalone Unreal MCP implementation worth checking out. This plugin's MCP support targets UE 5.4-5.7+ and works alongside Claude Desktop, Claude Code, and Cursor. **Note:** MCP in this free plugin is not being actively developed - the features listed below reflect the current state.
  101. <details>
  102. <summary><b>MCP Feature Status</b> (✅ = Done, 🛠️ = In Progress, 🚧 = Planned)</summary>
  103. - Clients Support ✅
  104. - Claude Desktop App Support ✅
  105. - Claude Code CLI Support ✅
  106. - Cursor IDE Support ✅
  107. - OpenAI Operator API Support 🚧
  108. - Blueprints Auto Generation 🛠️
  109. - Creating new blueprint of types ✅
  110. - Adding new functions, function/blueprint variables ✅
  111. - Adding nodes and connections 🛠️ (buggy, issues open)
  112. - Advanced Blueprints Generation 🛠️
  113. - Level/Scene Control for LLMs 🛠️
  114. - Spawning Objects and Shapes ✅
  115. - Moving, rotating and scaling objects ✅
  116. - Changing materials and color ✅
  117. - Advanced scene features 🛠️
  118. - Generative AI:
  119. - Prompt to 3D model fetch and spawn 🛠️
  120. - Control:
  121. - Ability to run Python scripts ✅
  122. - Ability to run Console Commands ✅
  123. - UI:
  124. - Widgets generation 🛠️
  125. - UI Blueprint generation 🛠️
  126. - Project Files:
  127. - Create/Edit project files/folders ️✅
  128. - Delete existing project files ❌
  129. - Others:
  130. - Project Cleanup 🛠️
  131. </details>
  132. ## Table of Contents
  133. - [Setting API Keys](#setting-api-keys)
  134. - [For Editor](#for-editor)
  135. - [For Packaged Builds](#for-packaged-builds)
  136. - [Setting up MCP](#setting-up-mcp)
  137. - [Adding the plugin to your project](#adding-the-plugin-to-your-project)
  138. - [With Git](#with-git)
  139. - [With Perforce](#with-perforce)
  140. - [With Fab (Unreal Marketplace)](#with-fab-unreal-marketplace)
  141. - [Fetching the Latest Plugin Changes](#fetching-the-latest-plugin-changes)
  142. - [With Git](#with-git-1)
  143. - [With Perforce](#with-perforce-1)
  144. - [Usage](#usage)
  145. - [OpenAI](#openai)
  146. - [1. Chat](#1-chat)
  147. - [2. Structured Outputs](#2-structured-outputs)
  148. - [DeepSeek API](#deepseek-api)
  149. - [1. Chat and Reasoning](#1-chat-and-reasoning)
  150. - [Anthropic API](#anthropic-api)
  151. - [1. Chat](#1-chat-1)
  152. - [XAI's Grok 3 API](#xais-grok-3-api)
  153. - [1. Chat](#1-chat-2)
  154. - [Model Control Protocol (MCP)](#model-control-protocol-mcp)
  155. - [Known Issues](#known-issues)
  156. - [Contribution Guidelines](#contribution-guidelines)
  157. - [Setting up for Development](#setting-up-for-development)
  158. - [Project Structure](#project-structure)
  159. - [References](#references)
  160. ## Setting API Keys:
  161. > [!NOTE]
  162. > There is no need to set the API key for testing the MCP features in Claude app. Anthropic key only needed for Claude API.
  163. ### For Editor:
  164. Set the environment variable `PS_<ORGNAME>` to your API key.
  165. #### For Windows:
  166. ```cmd
  167. setx PS_<ORGNAME> "your api key"
  168. ```
  169. #### For Linux/MacOS:
  170. 1. Run the following command in your terminal, replacing yourkey with your API key.
  171. ```bash
  172. echo "export PS_<ORGNAME>='yourkey'" >> ~/.zshrc
  173. ```
  174. 2. Update the shell with the new variable:
  175. ```bash
  176. source ~/.zshrc
  177. ```
  178. PS: Don't forget to restart the Editor and ALSO the connected IDE after setting the environment variable.
  179. Where `<ORGNAME>` can be:
  180. `PS_OPENAIAPIKEY`, `PS_DEEPSEEKAPIKEY`, `PS_ANTHROPICAPIKEY`, `PS_METAAPIKEY`, `PS_GOOGLEAPIKEY` etc.
  181. ### For Packaged Builds:
  182. Storing API keys in packaged builds is a security risk. This is what the OpenAI API documentation says about it:
  183. >"Exposing your OpenAI API key in client-side environments like browsers or mobile apps allows malicious users to take that key and make requests on your behalf – which may lead to unexpected charges or compromise of certain account data. Requests should always be routed through your own backend server where you can keep your API key secure."
  184. Read more about it [here](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).
  185. For test builds you can call the `GenSecureKey::SetGenAIApiKeyRuntime` either in c++ or blueprints function with your API key in the packaged build.
  186. ## Setting up MCP:
  187. > [!NOTE]
  188. > If your project only uses the LLM APIs and not the MCP, you can skip this section.
  189. > [!CAUTION]
  190. > Discalimer: If you are using the MCP feature of the plugin, it will directly let the Claude Desktop App control your Unreal Engine project.
  191. > Make sure you are aware of the security risks and only use it in a controlled environment.
  192. >
  193. > Please backup your project before using the MCP feature and use version control to track changes.
  194. ##### 1. Install any one of the below clients:
  195. * Claude Desktop App from [here](https://claude.anthropic.com/).
  196. * Claude Code CLI from [here](https://docs.anthropic.com/en/docs/claude-code).
  197. * Cursor IDE from [here](https://www.cursor.com/).
  198. ##### 2. Setup the mcp config json:
  199. ###### For Claude Desktop App:
  200. `claude_desktop_config.json` file in Claude Desktop App's installation directory. (might ask claude where its located for your platform!)
  201. The file will look something like this:
  202. ```json
  203. {
  204. "mcpServers": {
  205. "unreal-handshake": {
  206. "command": "python",
  207. "args": ["<your_project_directoy_path>/Plugins/GenerativeAISupport/Content/Python/mcp_server.py"],
  208. "env": {
  209. "UNREAL_HOST": "localhost",
  210. "UNREAL_PORT": "9877"
  211. }
  212. }
  213. }
  214. }
  215. ```
  216. ###### For Claude Code CLI:
  217. `.mcp.json` file in your project root directory. The file will look something like this:
  218. ```json
  219. {
  220. "mcpServers": {
  221. "unreal-handshake": {
  222. "type": "stdio",
  223. "command": "python",
  224. "args": ["<your_project_directoy_path>/Plugins/GenerativeAISupport/Content/Python/mcp_server.py"],
  225. "env": {
  226. "UNREAL_HOST": "localhost",
  227. "UNREAL_PORT": "9877"
  228. }
  229. }
  230. }
  231. }
  232. ```
  233. ###### For Cursor IDE:
  234. `.cursor/mcp.json` file in your project directory. The file will look something like this:
  235. ```json
  236. {
  237. "mcpServers": {
  238. "unreal-handshake": {
  239. "command": "python",
  240. "args": ["<your_project_directoy_path>/Plugins/GenerativeAISupport/Content/Python/mcp_server.py"],
  241. "env": {
  242. "UNREAL_HOST": "localhost",
  243. "UNREAL_PORT": "9877"
  244. }
  245. }
  246. }
  247. }
  248. ```
  249. ##### 3. Install FastMCP.
  250. ```bash
  251. pip install fastmcp
  252. ```
  253. ##### 4. Enable python plugin in Unreal Engine. (Edit -> Plugins -> Python Editor Script Plugin)
  254. ##### 5. [OPTIONAL] Enable AutoStart MCP server on editor open
  255. <img src="Docs/Settings.png" width="782"/>
  256. ## Adding the plugin to your project:
  257. ### With Git:
  258. 1. Add the Plugin Repository as a Submodule in your project's repository.
  259. ```cmd
  260. git submodule add https://github.com/prajwalshettydev/UnrealGenAISupport Plugins/GenerativeAISupport
  261. ```
  262. 2. Regenerate Project Files:
  263. Right-click your .uproject file and select Generate Visual Studio project files.
  264. 3. Enable the Plugin in Unreal Editor:
  265. Open your project in Unreal Editor. Go to Edit > Plugins. Search for the Plugin in the list and enable it.
  266. 4. For Unreal C++ Projects, include the Plugin's module in your project's Build.cs file:
  267. ```cpp
  268. PrivateDependencyModuleNames.AddRange(new string[] { "GenerativeAISupport" });
  269. ```
  270. ### With Perforce:
  271. Still in development..
  272. ### With Fab (Unreal Marketplace):
  273. This free plugin is available via Git (above). For the pro plugins, check [Fab.com](https://muddyterrain.com/t/genai-fab?utm_source=github.com&utm_medium=repo-free&utm_campaign=genai-plugin).
  274. ## Fetching the Latest Plugin Changes:
  275. ### With Git:
  276. you can pull the latest changes with:
  277. ```cmd
  278. cd Plugins/GenerativeAISupport
  279. git pull origin main
  280. ```
  281. Or update all submodules in the project:
  282. ```cmd
  283. git submodule update --recursive --remote
  284. ```
  285. ### With Perforce:
  286. Still in development..
  287. ## Usage:
  288. There is a example Unreal project that already implements the plugin. You can find it [here](https://github.com/prajwalshettydev/unreal-llm-api-test-project).
  289. ### OpenAI:
  290. Currently the plugin supports Chat and Structured Outputs from OpenAI API. Both for C++ and Blueprints.
  291. Tested models: `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`, `o4-mini`, `o3`, `o3-pro`, `o3-mini`.
  292. #### 1. Chat:
  293. ##### C++ Example:
  294. ```cpp
  295. void SomeDebugSubsystem::CallGPT(const FString& Prompt,
  296. const TFunction<void(const FString&, const FString&, bool)>& Callback)
  297. {
  298. FGenChatSettings ChatSettings;
  299. ChatSettings.Model = TEXT("gpt-4o-mini");
  300. ChatSettings.MaxTokens = 500;
  301. ChatSettings.Messages.Add(FGenChatMessage{ TEXT("system"), Prompt });
  302. FOnChatCompletionResponse OnComplete = FOnChatCompletionResponse::CreateLambda(
  303. [Callback](const FString& Response, const FString& ErrorMessage, bool bSuccess)
  304. {
  305. Callback(Response, ErrorMessage, bSuccess);
  306. });
  307. UGenOAIChat::SendChatRequest(ChatSettings, OnComplete);
  308. }
  309. ```
  310. ##### Blueprint Example:
  311. <img src="Docs/BpExampleOAIChat.png" width="782"/>
  312. #### 2. Structured Outputs:
  313. ##### C++ Example 1:
  314. Sending a custom schema json directly to function call
  315. ```cpp
  316. FString MySchemaJson = R"({
  317. "type": "object",
  318. "properties": {
  319. "count": {
  320. "type": "integer",
  321. "description": "The total number of users."
  322. },
  323. "users": {
  324. "type": "array",
  325. "items": {
  326. "type": "object",
  327. "properties": {
  328. "name": { "type": "string", "description": "The user's name." },
  329. "heading_to": { "type": "string", "description": "The user's destination." }
  330. },
  331. "required": ["name", "role", "age", "heading_to"]
  332. }
  333. }
  334. },
  335. "required": ["count", "users"]
  336. })";
  337. UGenAISchemaService::RequestStructuredOutput(
  338. TEXT("Generate a list of users and their details"),
  339. MySchemaJson,
  340. [](const FString& Response, const FString& Error, bool Success) {
  341. if (Success)
  342. {
  343. UE_LOG(LogTemp, Log, TEXT("Structured Output: %s"), *Response);
  344. }
  345. else
  346. {
  347. UE_LOG(LogTemp, Error, TEXT("Error: %s"), *Error);
  348. }
  349. }
  350. );
  351. ```
  352. ##### C++ Example 2:
  353. Sending a custom schema json from a file
  354. ```cpp
  355. #include "Misc/FileHelper.h"
  356. #include "Misc/Paths.h"
  357. FString SchemaFilePath = FPaths::Combine(
  358. FPaths::ProjectDir(),
  359. TEXT("Source/:ProjectName/Public/AIPrompts/SomeSchema.json")
  360. );
  361. FString MySchemaJson;
  362. if (FFileHelper::LoadFileToString(MySchemaJson, *SchemaFilePath))
  363. {
  364. UGenAISchemaService::RequestStructuredOutput(
  365. TEXT("Generate a list of users and their details"),
  366. MySchemaJson,
  367. [](const FString& Response, const FString& Error, bool Success) {
  368. if (Success)
  369. {
  370. UE_LOG(LogTemp, Log, TEXT("Structured Output: %s"), *Response);
  371. }
  372. else
  373. {
  374. UE_LOG(LogTemp, Error, TEXT("Error: %s"), *Error);
  375. }
  376. }
  377. );
  378. }
  379. ```
  380. ##### Blueprint Example:
  381. <img src="Docs/BpExampleOAIStructuredOp.png" width="782"/>
  382. ### DeepSeek API:
  383. Currently the plugin supports Chat and Reasoning from DeepSeek API. Both for C++ and Blueprints.
  384. Points to note:
  385. * System messages are currently mandatory for the reasoning model. API otherwise seems to return null
  386. * Also, from the documentation: "Please note that if the reasoning_content field is included in the sequence of input messages, the API will return a 400 error.
  387. Read more about it [here](https://api-docs.deepseek.com/guides/reasoning_model)"
  388. > [!WARNING]
  389. > While using the R1 reasoning model, make sure the Unreal's HTTP timeouts are not the default values at 30 seconds.
  390. > As these API calls can take longer than 30 seconds to respond. Simply setting the `HttpRequest->SetTimeout(<N Seconds>);` is not enough
  391. > So the following lines need to be added to your project's `DefaultEngine.ini` file:
  392. > ```ini
  393. > [HTTP]
  394. > HttpConnectionTimeout=180
  395. > HttpReceiveTimeout=180
  396. > ```
  397. #### 1. Chat and Reasoning:
  398. ##### C++ Example:
  399. ```cpp
  400. FGenDSeekChatSettings ReasoningSettings;
  401. ReasoningSettings.Model = EDeepSeekModels::Reasoner; // or EDeepSeekModels::Chat for Chat API
  402. ReasoningSettings.MaxTokens = 100;
  403. ReasoningSettings.Messages.Add(FGenChatMessage{TEXT("system"), TEXT("You are a helpful assistant.")});
  404. ReasoningSettings.Messages.Add(FGenChatMessage{TEXT("user"), TEXT("9.11 and 9.8, which is greater?")});
  405. ReasoningSettings.bStreamResponse = false;
  406. UGenDSeekChat::SendChatRequest(
  407. ReasoningSettings,
  408. FOnDSeekChatCompletionResponse::CreateLambda(
  409. [this](const FString& Response, const FString& ErrorMessage, bool bSuccess)
  410. {
  411. if (!UTHelper::IsContextStillValid(this))
  412. {
  413. return;
  414. }
  415. // Log response details regardless of success
  416. UE_LOG(LogTemp, Warning, TEXT("DeepSeek Reasoning Response Received - Success: %d"), bSuccess);
  417. UE_LOG(LogTemp, Warning, TEXT("Response: %s"), *Response);
  418. if (!ErrorMessage.IsEmpty())
  419. {
  420. UE_LOG(LogTemp, Error, TEXT("Error Message: %s"), *ErrorMessage);
  421. }
  422. })
  423. );
  424. ```
  425. ##### Blueprint Example:
  426. <img src="Docs/BpExampleDeepseekChat.png" width="782"/>
  427. ### Anthropic API:
  428. Currently the plugin supports Chat from Anthropic API. Both for C++ and Blueprints.
  429. Tested models: `claude-4-latest`, `claude-3-7-sonnet-latest`, `claude-3-5-sonnet`, `claude-3-5-haiku-latest`.
  430. #### 1. Chat:
  431. ##### C++ Example:
  432. ```cpp
  433. // ---- Claude Chat Test ----
  434. FGenClaudeChatSettings ChatSettings;
  435. ChatSettings.Model = EClaudeModels::Claude_3_7_Sonnet; // Use Claude 3.7 Sonnet model
  436. ChatSettings.MaxTokens = 4096;
  437. ChatSettings.Temperature = 0.7f;
  438. ChatSettings.Messages.Add(FGenChatMessage{TEXT("system"), TEXT("You are a helpful assistant.")});
  439. ChatSettings.Messages.Add(FGenChatMessage{TEXT("user"), TEXT("What is the capital of France?")});
  440. UGenClaudeChat::SendChatRequest(
  441. ChatSettings,
  442. FOnClaudeChatCompletionResponse::CreateLambda(
  443. [this](const FString& Response, const FString& ErrorMessage, bool bSuccess)
  444. {
  445. if (!UTHelper::IsContextStillValid(this))
  446. {
  447. return;
  448. }
  449. if (bSuccess)
  450. {
  451. UE_LOG(LogTemp, Warning, TEXT("Claude Chat Response: %s"), *Response);
  452. }
  453. else
  454. {
  455. UE_LOG(LogTemp, Error, TEXT("Claude Chat Error: %s"), *ErrorMessage);
  456. }
  457. })
  458. );
  459. ```
  460. ##### Blueprint Example:
  461. <img src="Docs/BpExampleClaudeChat.png" width="782"/>
  462. ### XAI's Grok 3 API:
  463. Currently the plugin supports Chat from XAI's Grok 3 API. Both for C++ and Blueprints.
  464. #### 1. Chat:
  465. ```cpp
  466. FGenXAIChatSettings ChatSettings;
  467. ChatSettings.Model = TEXT("grok-3-latest");
  468. ChatSettings.Messages.Add(FGenXAIMessage{
  469. TEXT("system"),
  470. TEXT("You are a helpful AI assistant for a game. Please provide concise responses.")
  471. });
  472. ChatSettings.Messages.Add(FGenXAIMessage{TEXT("user"), TEXT("Create a brief description for a forest level in a fantasy game")});
  473. ChatSettings.MaxTokens = 1000;
  474. UGenXAIChat::SendChatRequest(
  475. ChatSettings,
  476. FOnXAIChatCompletionResponse::CreateLambda(
  477. [this](const FString& Response, const FString& ErrorMessage, bool bSuccess)
  478. {
  479. if (!UTHelper::IsContextStillValid(this))
  480. {
  481. return;
  482. }
  483. UE_LOG(LogTemp, Warning, TEXT("XAI Chat response: %s"), *Response);
  484. if (!bSuccess)
  485. {
  486. UE_LOG(LogTemp, Error, TEXT("XAI Chat error: %s"), *ErrorMessage);
  487. }
  488. })
  489. );
  490. ```
  491. ## Model Control Protocol (MCP):
  492. This is currently work in progress. The plugin supports various clients like Claude Desktop App, Cursor etc.
  493. ### Usage:
  494. #### If Autostart MCP server is enabled: (In plugin's settings)
  495. ##### 1. Open the Unreal Engine Editor.
  496. ##### 2. Open the Claude Desktop App, Claude Code CLI, or Cursor IDE.
  497. That's it! You can now use the MCP features of the plugin.
  498. #### If Autostart MCP server is disabled:
  499. ##### 1. Run the MCP server from the plugin's python directory.
  500. ```bash
  501. python <your_project_directoy>/Plugins/GenerativeAISupport/Content/Python/mcp_server.py
  502. ```
  503. ##### 2. Run the MCP client by opening or restarting the Claude desktop app or Cursor IDE.
  504. ##### 3. Open a new Unreal Engine project and run the below python script from the plugin's python directory.
  505. > Tools -> Run Python Script -> Select the `Plugins/GenerativeAISupport/Content/Python/unreal_socket_server.py` file.
  506. #### 4. Now you should be able to prompt the Claude Desktop App to use Unreal Engine.
  507. ## Known Issues:
  508. - Nodes fail to connect properly with MCP
  509. - No undo redo support for MCP
  510. - No streaming support for Deepseek reasoning model
  511. - No complex material generation support for the create material tool
  512. - Issues with running some llm generated valid python scripts
  513. - When LLM compiles a blueprint no proper error handling in its response
  514. - Issues spawning certain nodes, especially with getters and setters
  515. - Doesn't open the right context window during scene and project files edit.
  516. - Doesn't dock the window properly in the editor for blueprints.
  517. ## Contribution Guidelines:
  518. ### Setting up for Development:
  519. 1. Install `unreal` python package and setup the IDE's python interpreter for proper intellisense.
  520. ```bash
  521. pip install unreal
  522. ```
  523. More details will be added soon.
  524. ### Project Structure:
  525. More details will be added soon.
  526. ## References:
  527. * Env Var set logic
  528. from: [OpenAI-Api-Unreal by KellanM](https://github.com/KellanM/OpenAI-Api-Unreal/blob/main/Source/OpenAIAPI/Private/OpenAIUtils.cpp)
  529. * MCP Server inspiration
  530. from: [Blender-MCP by ahujasid](https://github.com/ahujasid/blender-mcp)
  531. ## Quick Links:
  532. - [OpenAI API Documentation](https://platform.openai.com/docs/api-reference)
  533. - [Anthropic API Documentation](https://docs.anthropic.com/en/docs/about-claude/models)
  534. - [XAI API Documentation](https://docs.x.ai/api)
  535. - [Google Gemini API Documentation](https://ai.google.dev/gemini-api/docs/models/gemini)
  536. - [Meta AI API Documentation](https://docs.llama-api.com/quickstart#available-models)
  537. - [Deepseek API Documentation](https://api-docs.deepseek.com/)
  538. - [Model Control Protocol (MCP) Documentation](https://modelcontextprotocol.io/)
  539. - [TripoSt Documentation](https://huggingface.co/stabilityai/TripoSR)
  540. ## Support This Project
  541. <iframe src="https://github.com/sponsors/prajwalshettydev/button" title="Sponsor prajwalshettydev" height="32" width="114" style="border: 0; border-radius: 6px;"></iframe>
  542. If you find UnrealGenAISupport helpful, consider sponsoring me to keep the project going! Click the "Sponsor" button above to contribute.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.