Skip to content
Back to search

agent-plugins-for-aws

SkillGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/awslabs--agent-plugins/badge.svg)](/listings/awslabs--agent-plugins)

HTML

<a href="/listings/awslabs--agent-plugins"><img src="/api/listings/awslabs--agent-plugins/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Claude Code

Add the marketplace

/plugin marketplace add awslabs/agent-plugins

Install a plugin

/plugin install amazon-location-service@agent-plugins-for-aws

or

/plugin install aws-amplify@agent-plugins-for-aws

or

/plugin install aws-serverless@agent-plugins-for-aws

or

/plugin install codebase-documentor-for-aws@agent-plugins-for-aws

or

/plugin install databases-on-aws@agent-plugins-for-aws

or

/plugin install deploy-on-aws@agent-plugins-for-aws

or

/plugin install sagemaker-ai@agent-plugins-for-aws

Codex

Codex supports repo-local marketplaces and plugin manifests through `.agents/plugins/marketplace.json` and per-plugin manifests under `plugins/*/.codex-plugin/plugin.json`; this repo includes both, following the official Codex plugin packaging guidance in [Build plugins](https://developers.openai.com/codex/plugins/build).

To try these plugins in Codex from this repo:

1. Clone this repository locally. 2. Open the repo in Codex so Codex can discover `.agents/plugins/marketplace.json`. 3. Restart Codex. 4. Open the plugin directory, choose the `Agent Plugins for AWS` marketplace, and install the plugin you want to use.

Known limitation:

- Claude-specific automatic hooks are not yet wired into the Codex manifests. - The `databases-on-aws` prompt hook is included because it does not depend on Claude-specific environment variables.

Cursor

You can install the **deploy-on-aws** plugin from the [Cursor Marketplace](https://cursor.com/marketplace/aws). For additional information, please refer to the [Cursor plugin documentation](https://cursor.com/docs/plugins). You can also install within the Cursor application:

- Open Cursor Settings - Navigate to `Plugins` - Search for `AWS` - Select the plugin you want to install and click `Add to Cursor` - Select the scope for the installed plugin - The plugin should appear under `Plugins -> Installed`

Kiro (experimental)

You can convert Claude Code plugins into Kiro format using the third-party [`@every-env/compound-plugin`](https://www.npmjs.com/package/@every-env/compound-plugin) CLI. This converts skills (SKILL.md workflow instructions and reference documents) and MCP server configs into Kiro's `.kiro/` format.

Requires [Bun](https://bun.sh) (latest version recommended).

> [!NOTE] > Hook conversion is a work in progress — hooks are currently dropped entirely during conversion because the Claude Code and Kiro hook models are not 1:1. Skills and MCP servers are fully supported.

Project scope

Writes to `.kiro/` in the current working directory:

COMPOUND_PLUGIN_GITHUB_SOURCE=https://github.com/awslabs/agent-plugins \
  bunx @every-env/compound-plugin install deploy-on-aws --to kiro

Global scope

Writes to `~/.kiro/` so skills are available across all projects:

COMPOUND_PLUGIN_GITHUB_SOURCE=https://github.com/awslabs/agent-plugins \
  bunx @every-env/compound-plugin install deploy-on-aws --to kiro --output ~/.kiro

Replace `deploy-on-aws` with any plugin name from the table above (e.g., `aws-serverless`, `amazon-location-service`, `migration-to-aws`, `aws-amplify`).

> [!TIP] > If a skill isn't activated automatically, you can explicitly invoke it by saying "Use the deploy skill to ..." in your prompt. Kiro's intent matching may not always auto-trigger skills. > > If `bunx` fails with a "Cannot find package" error, clear the bun cache and retry: > > ```bash > rm -rf ~/.bun/install/cache/@every-env > ```

For issues with the conversion tool, please report them at [EveryInc/compound-engineering-plugin](https://github.com/EveryInc/compound-engineering-plugin/issues).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 248 lines
  1. {
  2. "metadata": {
  3. "description": "Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS.",
  4. "version": "1.1.0"
  5. },
  6. "name": "agent-plugins-for-aws",
  7. "owner": {
  8. "email": "aws-agent-plugins@amazon.com",
  9. "name": "Amazon Web Services"
  10. },
  11. "plugins": [
  12. {
  13. "category": "location",
  14. "description": "Guide developers through adding maps, places search, geocoding, routing, and other geospatial features with Amazon Location Service, including authentication setup, SDK integration, and best practices.",
  15. "keywords": [
  16. "location",
  17. "maps",
  18. "geocoding",
  19. "routing",
  20. "places",
  21. "geofencing",
  22. "tracking",
  23. "amazon-location",
  24. "maplibre",
  25. "address",
  26. "coordinates"
  27. ],
  28. "name": "amazon-location-service",
  29. "source": "./plugins/amazon-location-service",
  30. "tags": [
  31. "aws",
  32. "location",
  33. "maps",
  34. "geospatial"
  35. ],
  36. "version": "1.1.0"
  37. },
  38. {
  39. "category": "fullstack",
  40. "description": "Build full-stack apps with AWS Amplify Gen 2 using guided workflows for authentication, data models, storage, GraphQL APIs, and Lambda functions.",
  41. "keywords": [
  42. "aws",
  43. "amplify",
  44. "fullstack",
  45. "authentication",
  46. "data",
  47. "storage",
  48. "graphql",
  49. "functions"
  50. ],
  51. "name": "aws-amplify",
  52. "source": "./plugins/aws-amplify",
  53. "tags": [
  54. "aws",
  55. "amplify",
  56. "fullstack"
  57. ],
  58. "version": "1.0.0"
  59. },
  60. {
  61. "category": "development",
  62. "description": "Design, build, deploy, test, and debug serverless applications with AWS Serverless services.",
  63. "keywords": [
  64. "aws",
  65. "aws agent skills",
  66. "amazon",
  67. "serverless",
  68. "lambda",
  69. "durable functions",
  70. "managed-instances",
  71. "lmi",
  72. "capacity-provider",
  73. "development",
  74. "sam",
  75. "api gateway",
  76. "eventbridge",
  77. "step functions",
  78. "microvms",
  79. "firecracker"
  80. ],
  81. "name": "aws-serverless",
  82. "source": "./plugins/aws-serverless",
  83. "tags": [
  84. "aws",
  85. "serverless",
  86. "lambda",
  87. "durable functions",
  88. "managed-instances",
  89. "lmi",
  90. "development",
  91. "sam",
  92. "api gateway",
  93. "eventbridge",
  94. "step functions",
  95. "microvms",
  96. "firecracker"
  97. ],
  98. "version": "1.3.0"
  99. },
  100. {
  101. "category": "migration",
  102. "description": "Migrate, modernize, and upgrade codebases to AWS. Transforms .NET Framework to .NET 8/10, mainframe COBOL to Java, VMware VMs to EC2, SQL Server to Aurora, and upgrades Java/Python/Node.js versions and AWS SDKs. AWS Transform - continuous modernization analyzes codebases for tech debt, security issues, and upgrade opportunities, then remediates them.",
  103. "keywords": [
  104. "aws",
  105. "aws-transform",
  106. "migration",
  107. "modernization",
  108. "dotnet",
  109. "mainframe",
  110. "cobol",
  111. "vmware",
  112. "ec2",
  113. "sql-server",
  114. "aurora",
  115. "java",
  116. "python",
  117. "sdk",
  118. "continuous modernization",
  119. "AWS Transform - continuous modernization",
  120. "tech debt",
  121. "security vulnerabilities",
  122. "analyze repos",
  123. "security analysis",
  124. "modernization analysis",
  125. "agentic readiness"
  126. ],
  127. "name": "aws-transform",
  128. "source": "./plugins/aws-transform",
  129. "tags": [
  130. "aws",
  131. "migration",
  132. "modernization",
  133. "transform",
  134. "continuous modernization"
  135. ],
  136. "version": "1.9.0"
  137. },
  138. {
  139. "category": "development",
  140. "description": "Analyze codebases to generate structured technical documentation with source-of-truth citations. Optimized for AWS workloads using CDK, CloudFormation, and Terraform.",
  141. "keywords": [
  142. "aws",
  143. "aws agent skills",
  144. "documentation",
  145. "architecture",
  146. "diagram",
  147. "draw.io",
  148. "codebase-analysis",
  149. "cdk",
  150. "cloudformation",
  151. "terraform",
  152. "onboarding"
  153. ],
  154. "name": "codebase-documentor-for-aws",
  155. "source": "./plugins/codebase-documentor-for-aws",
  156. "tags": [
  157. "aws",
  158. "documentation",
  159. "architecture",
  160. "diagram"
  161. ],
  162. "version": "0.1.0"
  163. },
  164. {
  165. "category": "database",
  166. "description": "Expert database guidance for the AWS database portfolio. Design schemas, execute queries, handle migrations, and choose the right database for your workload.",
  167. "keywords": [
  168. "aws",
  169. "aws agent skills",
  170. "amazon",
  171. "aurora",
  172. "database",
  173. "databases",
  174. "distributed-sql",
  175. "dsql",
  176. "iam-auth",
  177. "migration",
  178. "postgresql",
  179. "postgres",
  180. "schema",
  181. "serverless",
  182. "sql"
  183. ],
  184. "name": "databases-on-aws",
  185. "source": "./plugins/databases-on-aws",
  186. "tags": [
  187. "aws",
  188. "database",
  189. "aurora",
  190. "dsql",
  191. "serverless",
  192. "postgresql"
  193. ],
  194. "version": "1.11.0"
  195. },
  196. {
  197. "category": "deployment",
  198. "description": "Deploy applications to AWS with architecture recommendations, cost estimates, and IaC deployment. Generate validated AWS architecture diagrams as draw.io XML.",
  199. "keywords": [
  200. "aws",
  201. "aws agent skills",
  202. "amazon",
  203. "deploy",
  204. "cdk",
  205. "cloudformation",
  206. "infrastructure",
  207. "pricing",
  208. "diagrams",
  209. "draw.io",
  210. "architecture"
  211. ],
  212. "name": "deploy-on-aws",
  213. "source": "./plugins/deploy-on-aws",
  214. "tags": [
  215. "aws",
  216. "deploy",
  217. "infrastructure",
  218. "cdk",
  219. "diagrams"
  220. ],
  221. "version": "1.4.0"
  222. },
  223. {
  224. "category": "ai",
  225. "description": "Build, train, and deploy AI models with deep AWS AI/ML expertise brought directly into your coding assistants, covering the surface area of Amazon SageMaker AI.",
  226. "keywords": [
  227. "aws",
  228. "sagemaker",
  229. "machine-learning",
  230. "generative-ai",
  231. "fine-tuning",
  232. "training",
  233. "deployment",
  234. "inference",
  235. "mlops"
  236. ],
  237. "name": "sagemaker-ai",
  238. "source": "./plugins/sagemaker-ai",
  239. "tags": [
  240. "aws",
  241. "sagemaker",
  242. "machine-learning",
  243. "generative-ai"
  244. ],
  245. "version": "1.3.0"
  246. }
  247. ]
  248. }
README.md · README · 349 lines
  1. # Agent Plugins for AWS
  2. _Read this in other languages: [日本語](README.jp.md)_
  3. > [!IMPORTANT]
  4. > Generative AI can make mistakes. You should consider reviewing all output and costs generated by your chosen AI model and agentic coding assistant. See [AWS Responsible AI Policy](https://aws.amazon.com/ai/responsible-ai/policy/).
  5. ## What's new
  6. > [!TIP]
  7. > The [Agent Toolkit for AWS](https://aws.amazon.com/about-aws/whats-new/2026/05/agent-toolkit/) is now live! The Agent Toolkit for AWS is the successor to the MCP servers, plugins, and skills available on AWS Labs, and was informed by feedback from customers like you. If you're building production software using coding agents or building agents for your own customers, we recommend Agent Toolkit for AWS. It includes IAM condition keys to distinguish agent actions from human ones, CloudWatch and CloudTrail visibility, and skills that have been evaluated for accuracy and effectiveness. This repo continues to work and accept contributions. Over time, the most useful projects here will move into Agent Toolkit for AWS.
  8. ---
  9. ## Agent Plugins for AWS
  10. Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS. Agent plugins are currently supported by Claude Code, Codex, and Cursor.
  11. AI coding agents are increasingly used in software development, helping developers write, review, and deploy code more efficiently. Agent skills and the broader agent plugin packaging model are emerging as best practices for steering coding agents toward reliable outcomes without bloating model context. Instead of repeatedly pasting long AWS guidance into prompts, developers can now encode that guidance as reusable, versioned capabilities that agents invoke when relevant. This improves determinism, reduces context overhead, and makes agent behavior easier to standardize across teams. Agent plugins act as containers that package different types of expertise artifacts together. A single agent plugin can include:
  12. - Agent skills – Structured workflows and best-practice playbooks that guide AI through complex tasks like deployment, code review, or architecture planning. Agent skills encode domain expertise as step-by-step processes.
  13. - MCP servers – Connections to external services, data sources, and APIs. MCP servers give your assistant access to live documentation, pricing data, and other resources at runtime. Learn more about [MCP servers for AWS](https://github.com/awslabs/mcp).
  14. - Hooks – Automation and guardrails that run on developer actions. Hooks can validate changes, enforce standards, or trigger workflows automatically.
  15. - References – Documentation, configuration defaults, and knowledge that the agent skill can consult. References make agent skills smarter without bloating the prompt.
  16. As new types of expertise artifacts emerge in this space, they can be packaged into agent plugins, making the evolution transparent to developers.
  17. ## Best practices
  18. To maximize the benefits of plugin-assisted development while maintaining security and code quality, follow these essential guidelines:
  19. - Always review generated code before deployment (for example, against your constraints for security, cost, resilience)
  20. - Use plugins as accelerators, not replacements for developer judgment and expertise.
  21. - Keep plugins updated to benefit from the latest AWS best practices.
  22. - Follow the principle of least privilege when configuring AWS credentials.
  23. - Run security scanning tools on generated infrastructure code.
  24. ## Plugins
  25. | Plugin | Description | Status |
  26. | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
  27. | **amazon-location-service** | Add maps, geocoding, routing, places search, and geospatial features to applications with Amazon Location Service | Available |
  28. | **aws-amplify** | Build full-stack apps with AWS Amplify Gen 2 using guided workflows for auth, data, storage, and functions | Available |
  29. | **aws-serverless** | Build serverless applications with Lambda, API Gateway, EventBridge, Step Functions, durable functions, and Lambda Managed Instances | Available |
  30. | **aws-transform** | Migrate, modernize, and upgrade codebases to AWS — .NET to .NET 8/10, mainframe COBOL to Java, VMware to EC2, SQL Server to Aurora, and language/SDK upgrades | Available |
  31. | **codebase-documentor-for-aws** | Analyze AWS-deployed services and codebases to generate structured technical documentation with source-of-truth citations | Available |
  32. | **databases-on-aws** | Database guidance for the AWS database portfolio — schema design, queries, migrations, and multi-tenant patterns | Some Services Available (Aurora DSQL) |
  33. | **deploy-on-aws** | Deploy applications to AWS with architecture recommendations, cost estimates, and IaC deployment | Available |
  34. | **migration-to-aws** | Migrate GCP infrastructure to AWS with resource discovery, architecture mapping, cost analysis, and execution planning | Available |
  35. | **sagemaker-ai** | Build, train, and deploy AI models with deep AWS AI/ML expertise brought directly into your coding assistants, covering the surface area of [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/ai/) | Available |
  36. ## Installation
  37. ### Claude Code
  38. #### Add the marketplace
  39. ```bash
  40. /plugin marketplace add awslabs/agent-plugins
  41. ```
  42. #### Install a plugin
  43. ```bash
  44. /plugin install amazon-location-service@agent-plugins-for-aws
  45. ```
  46. or
  47. ```bash
  48. /plugin install aws-amplify@agent-plugins-for-aws
  49. ```
  50. or
  51. ```bash
  52. /plugin install aws-serverless@agent-plugins-for-aws
  53. ```
  54. or
  55. ```bash
  56. /plugin install codebase-documentor-for-aws@agent-plugins-for-aws
  57. ```
  58. or
  59. ```bash
  60. /plugin install databases-on-aws@agent-plugins-for-aws
  61. ```
  62. or
  63. ```bash
  64. /plugin install deploy-on-aws@agent-plugins-for-aws
  65. ```
  66. or
  67. ```bash
  68. /plugin install sagemaker-ai@agent-plugins-for-aws
  69. ```
  70. ### Codex
  71. Codex supports repo-local marketplaces and plugin manifests through
  72. `.agents/plugins/marketplace.json` and per-plugin manifests under
  73. `plugins/*/.codex-plugin/plugin.json`; this repo includes both, following the
  74. official Codex plugin packaging guidance in
  75. [Build plugins](https://developers.openai.com/codex/plugins/build).
  76. To try these plugins in Codex from this repo:
  77. 1. Clone this repository locally.
  78. 2. Open the repo in Codex so Codex can discover
  79. `.agents/plugins/marketplace.json`.
  80. 3. Restart Codex.
  81. 4. Open the plugin directory, choose the `Agent Plugins for AWS` marketplace,
  82. and install the plugin you want to use.
  83. Known limitation:
  84. - Claude-specific automatic hooks are not yet wired into the Codex manifests.
  85. - The `databases-on-aws` prompt hook is included because it does not depend on
  86. Claude-specific environment variables.
  87. ### Cursor
  88. You can install the **deploy-on-aws** plugin from the [Cursor Marketplace](https://cursor.com/marketplace/aws). For additional information, please refer to the [Cursor plugin documentation](https://cursor.com/docs/plugins). You can also install within the Cursor application:
  89. - Open Cursor Settings
  90. - Navigate to `Plugins`
  91. - Search for `AWS`
  92. - Select the plugin you want to install and click `Add to Cursor`
  93. - Select the scope for the installed plugin
  94. - The plugin should appear under `Plugins -> Installed`
  95. ### Kiro (experimental)
  96. You can convert Claude Code plugins into Kiro format using the third-party [`@every-env/compound-plugin`](https://www.npmjs.com/package/@every-env/compound-plugin) CLI. This converts skills (SKILL.md workflow instructions and reference documents) and MCP server configs into Kiro's `.kiro/` format.
  97. Requires [Bun](https://bun.sh) (latest version recommended).
  98. > [!NOTE]
  99. > Hook conversion is a work in progress — hooks are currently dropped entirely during conversion because the Claude Code and Kiro hook models are not 1:1. Skills and MCP servers are fully supported.
  100. #### Project scope
  101. Writes to `.kiro/` in the current working directory:
  102. ```bash
  103. COMPOUND_PLUGIN_GITHUB_SOURCE=https://github.com/awslabs/agent-plugins \
  104. bunx @every-env/compound-plugin install deploy-on-aws --to kiro
  105. ```
  106. #### Global scope
  107. Writes to `~/.kiro/` so skills are available across all projects:
  108. ```bash
  109. COMPOUND_PLUGIN_GITHUB_SOURCE=https://github.com/awslabs/agent-plugins \
  110. bunx @every-env/compound-plugin install deploy-on-aws --to kiro --output ~/.kiro
  111. ```
  112. Replace `deploy-on-aws` with any plugin name from the table above (e.g., `aws-serverless`, `amazon-location-service`, `migration-to-aws`, `aws-amplify`).
  113. > [!TIP]
  114. > If a skill isn't activated automatically, you can explicitly invoke it by saying "Use the deploy skill to ..." in your prompt. Kiro's intent matching may not always auto-trigger skills.
  115. >
  116. > If `bunx` fails with a "Cannot find package" error, clear the bun cache and retry:
  117. >
  118. > ```bash
  119. > rm -rf ~/.bun/install/cache/@every-env
  120. > ```
  121. For issues with the conversion tool, please report them at [EveryInc/compound-engineering-plugin](https://github.com/EveryInc/compound-engineering-plugin/issues).
  122. ## amazon-location-service
  123. Guides developers through adding maps, places search, geocoding, routing, and other geospatial features with Amazon Location Service, including authentication setup, SDK integration, and best practices.
  124. ### Agent Skill Triggers
  125. | Agent Skill | Triggers |
  126. | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
  127. | **amazon-location-service** | "add a map", "geocode an address", "calculate a route", "location-aware app", "Amazon Location Service", "geospatial", "places search" |
  128. ### MCP Servers
  129. | Server | Purpose |
  130. | ----------- | -------------------------------------- |
  131. | **aws-mcp** | AWS documentation and service guidance |
  132. ## aws-amplify
  133. Build full-stack apps with AWS Amplify Gen 2 using TypeScript code-first development, guided by official AWS Agent Standard Operating Procedures (SOPs).
  134. ### Workflow
  135. 1. **Backend** - Create Amplify Gen 2 resources (auth, data, storage, functions)
  136. 2. **Sandbox** - Deploy to sandbox for testing
  137. 3. **Frontend & Test** - Connect frontend to backend, verify locally
  138. 4. **Production** - Deploy to production
  139. ### Agent Skill Triggers
  140. | Agent Skill | Triggers |
  141. | -------------------- | ------------------------------------------------------------------------------------------------------------ |
  142. | **amplify-workflow** | "build Amplify app", "create Amplify project", "add auth to Amplify", "deploy Amplify", "full-stack Amplify" |
  143. ### MCP Servers
  144. | Server | Purpose |
  145. | ----------- | -------------------------------------- |
  146. | **aws-mcp** | AWS documentation and service guidance |
  147. ## aws-serverless
  148. Design, build, deploy, test, and debug serverless applications with AWS Lambda, API Gateway, EventBridge, Step Functions, and durable functions. Includes SAM and CDK deployment workflows, a SAM template validation hook, the AWS Lambda durable functions skill for building resilient, long-running, multi-step applications, and the Lambda Managed Instances skill for evaluating, configuring, and migrating workloads to EC2-backed Lambda.
  149. ### Agent Skill Triggers
  150. | Agent Skill | Triggers |
  151. | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  152. | **aws-lambda** | "Lambda function", "event source", "serverless application", "API Gateway", "EventBridge", "Step Functions", "serverless API", "event-driven architecture", "Lambda trigger" |
  153. | **aws-serverless-deployment** | "use SAM", "SAM template", "SAM init", "SAM deploy", "CDK serverless", "CDK Lambda construct", "NodejsFunction", "PythonFunction", "serverless CI/CD pipeline" |
  154. | **aws-lambda-durable-functions** | "lambda durable functions", "workflow orchestration", "state machines", "retry/checkpoint patterns", "long-running stateful Lambda", "saga pattern", "human-in-the-loop" |
  155. | **aws-lambda-managed-instances** | "Lambda Managed Instances", "LMI", "capacity provider", "multi-concurrency Lambda", "EC2-backed Lambda", "cold start elimination", "Graviton Lambda", "Lambda cost optimization with Reserved Instances" |
  156. ### MCP Servers
  157. | Server | Purpose |
  158. | ---------------------- | ------------------------------------------------------------------------------------ |
  159. | **aws-serverless-mcp** | Serverless development guidance, project scaffolding, IaC generation, and deployment |
  160. ### Hooks
  161. | Hook | Trigger | Action |
  162. | --------------------------- | --------------------------------------------- | --------------------------------------------- |
  163. | **SAM template validation** | After edits to `template.yaml`/`template.yml` | Runs `sam validate` and reports errors inline |
  164. ## codebase-documentor-for-aws
  165. Analyzes codebases to generate structured technical documentation with source-of-truth citations linking every finding back to the exact code that produced it. Uses an outline-driven pipeline to systematically analyze codebases of any size with a persistent task board for resumability.
  166. ### Agent Skill Triggers
  167. | Agent Skill | Triggers |
  168. | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  169. | **document-service** | "analyze this codebase", "generate documentation", "document this service", "I inherited this code", "help me understand this system", "draw the architecture", "what does this system look like" |
  170. ### MCP Servers
  171. | Server | Purpose |
  172. | ---------------- | -------------------------------------------------------------------- |
  173. | **awsknowledge** | AWS service descriptions, architecture guidance, documentation links |
  174. | **awsiac** | CDK/CloudFormation resource schema validation and IaC best practices |
  175. ## databases-on-aws
  176. Database guidance for the AWS database portfolio. Design schemas, execute queries, handle migrations, build applications, and choose the right database for your workload. Currently includes Aurora DSQL — a serverless, PostgreSQL-compatible distributed SQL database.
  177. ### Agent Skill Triggers
  178. | Agent Skill | Triggers |
  179. | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
  180. | **dsql** | "Aurora DSQL", "DSQL schema", "distributed SQL database", "serverless PostgreSQL-compatible database", "create DSQL table", "migrate to DSQL" |
  181. ### MCP Servers
  182. | Server | Purpose |
  183. | ---------------- | -------------------------------------------------------------------------------- |
  184. | **awsknowledge** | AWS documentation, architecture guidance, and best practices |
  185. | **aurora-dsql** | Direct database operations — queries, schema, transactions (disabled by default) |
  186. ### Hooks
  187. | Hook | Trigger | Action |
  188. | ----------------------- | --------------------------- | -------------------------------------------------------- |
  189. | **Schema verification** | After `transact` operations | Prompts verification of schema changes and affected rows |
  190. ## deploy-on-aws
  191. Equips agents with the skills to accelerate AWS deployment - recommending AWS architectures and services, estimating costs, generating Infrastructure as Code (CDK or CloudFormation), and guiding you through deployment.
  192. ### Workflow
  193. 1. **Analyze** - Scan codebase for framework, database, dependencies
  194. 2. **Recommend** - Select AWS services with concise rationale
  195. 3. **Estimate** - Show cost estimates before proceeding
  196. 4. **Generate** - Write IaC code (CDK/CloudFormation)
  197. 5. **Deploy** - Execute with user confirmation
  198. ### Agent Skill Triggers
  199. | Agent Skill | Triggers |
  200. | ----------- | --------------------------------------------------------------------------------------------------------------------- |
  201. | **deploy** | "deploy to AWS", "host on AWS", "run this on AWS", "AWS architecture", "estimate AWS cost", "generate infrastructure" |
  202. ### MCP Servers
  203. | Server | Purpose |
  204. | ---------------- | ------------------------------------------------------------ |
  205. | **awsknowledge** | AWS documentation, architecture guidance, and best practices |
  206. | **awspricing** | Real-time AWS service pricing for cost estimates |
  207. | **aws-iac-mcp** | IaC best practices for CDK/CloudFormation |
  208. ## sagemaker-ai
  209. Build, train, and deploy AI models with deep AWS AI/ML expertise brought directly into your coding assistants, covering the surface area of [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/ai/). [Learn more](./plugins/sagemaker-ai/README.md).
  210. ## Requirements
  211. - Claude Code >=2.1.29 or [Cursor >= 2.5](https://cursor.com/changelog/2-5)
  212. - AWS CLI configured with appropriate credentials
  213. ## Troubleshooting
  214. Having issues with plugin installation or usage? See our [troubleshooting guide](./docs/TROUBLESHOOTING.md) for common solutions.
  215. ## Contributing
  216. Big shout out to our awesome contributors! Thank you for making this project better!
  217. Contributions of all kinds are welcome! Check out our [contributor guide](./CONTRIBUTING.md) for more information.
  218. ## Developer guide
  219. If you want to add a new plugin to the library, check out our [design guidelines](./docs/DESIGN_GUIDELINES.md) and [development guide](./docs/DEVELOPMENT_GUIDE.md).
  220. ## Maintainers
  221. Repository reviewers, maintainers, and admins can find PR review workflows, merge rules, and CI/CD documentation in the [maintainers guide](./docs/MAINTAINERS_GUIDE.md).
  222. ## Administrators
  223. GitHub environment and AWS account configuration are documented in the [administrators guide](./docs/ADMINISTRATORS_GUIDE.md).
  224. ## Additional Resources
  225. - [Introducing Agent Plugins for AWS Blog](https://aws.amazon.com/blogs/developer/introducing-agent-plugins-for-aws/)
  226. - [Extend Cursor with plugins ft. AWS](https://cursor.com/blog/marketplace)
  227. - [MCP Servers for AWS](https://github.com/awslabs/mcp)
  228. - [Claude Code plugins documentation](https://code.claude.com/docs/en/plugins)
  229. - [Cursor plugins documentation](https://cursor.com/docs/plugins)
  230. ## License
  231. This project is licensed under the Apache-2.0 License.
plugins/sagemaker-ai/skills/hyperpod-cluster-debugger/scripts/diagnose-cluster.sh · Bundled script · 1,621 lines · 1 finding marked
  1. #!/usr/bin/env bash
  2. # diagnose-cluster.sh — read-only HyperPod cluster-level diagnostic.
  3. # See SKILL.md and references/cluster-diagnostics-detail.md for remediation.
  4. #
  5. # Exit codes:
  6. # 0 No critical (P0/P1) failures; P2 warnings are informational-only.
  7. # 1 One or more critical failures, or a fatal prerequisite.
  8. # 2 Invalid argument.
  9. set -euo pipefail
  10. for cmd in aws jq python3; do
  11. command -v "$cmd" &>/dev/null || {
  12. echo "ERROR: '$cmd' is required but not found. Install it and retry."
  13. exit 1
  14. }
  15. done
  16. # unbuffer is only needed if the Slurm-controller SSM probe runs
  17. # (activated when the cluster's orchestrator is Slurm). Warn at startup
  18. # but don't exit — EKS-only users shouldn't be blocked.
  19. if ! command -v unbuffer &>/dev/null; then
  20. echo "WARN: 'unbuffer' not found. Required for the Slurm-controller SSM probe." >&2
  21. echo " Install via 'yum install expect' / 'apt install expect' / 'brew install expect'." >&2
  22. echo " EKS diagnostics will continue; Slurm-controller-only checks will be skipped." >&2
  23. fi
  24. CLUSTER=""
  25. REGION="${AWS_DEFAULT_REGION:-}"
  26. USE_COLOR=true
  27. VALIDATE_MODE=false
  28. VALIDATE_SG_IDS=""
  29. VALIDATE_SUBNET_IDS=""
  30. VALIDATE_IAM_ROLE=""
  31. VALIDATE_S3_URI=""
  32. VALIDATE_INSTANCE_TYPE=""
  33. while [[ $# -gt 0 ]]; do
  34. case "$1" in
  35. --cluster) [[ $# -lt 2 ]] && { echo "ERROR: --cluster needs a value"; exit 2; }
  36. [[ ! "$2" =~ ^(arn:aws[a-z-]*:sagemaker:[a-z0-9-]+:[0-9]{12}:cluster/[a-z0-9]{12}|[a-zA-Z0-9]([-a-zA-Z0-9]{0,62}))$ ]] && { echo "ERROR: --cluster must be a valid HyperPod cluster name or ARN (got '$2')"; exit 2; }
  37. CLUSTER="$2"; shift 2 ;;
  38. --region) [[ $# -lt 2 ]] && { echo "ERROR: --region needs a value"; exit 2; }
  39. [[ ! "$2" =~ ^[a-z]{2}-[a-z]+-[0-9]+$ ]] && { echo "ERROR: --region must be a valid AWS region (got '$2')"; exit 2; }
  40. REGION="$2"; shift 2 ;;
  41. --sg-ids) [[ $# -lt 2 ]] && { echo "ERROR: --sg-ids needs a value"; exit 2; }; VALIDATE_SG_IDS="$2"; shift 2 ;;
  42. --subnet-ids) [[ $# -lt 2 ]] && { echo "ERROR: --subnet-ids needs a value"; exit 2; }; VALIDATE_SUBNET_IDS="$2"; shift 2 ;;
  43. --iam-role) [[ $# -lt 2 ]] && { echo "ERROR: --iam-role needs a value"; exit 2; }; VALIDATE_IAM_ROLE="$2"; shift 2 ;;
  44. --s3-uri) [[ $# -lt 2 ]] && { echo "ERROR: --s3-uri needs a value"; exit 2; }; VALIDATE_S3_URI="$2"; shift 2 ;;
  45. --instance-type) [[ $# -lt 2 ]] && { echo "ERROR: --instance-type needs a value"; exit 2; }; VALIDATE_INSTANCE_TYPE="$2"; shift 2 ;;
  46. --no-color) USE_COLOR=false; shift ;;
  47. --validate) VALIDATE_MODE=true; shift ;;
  48. -h|--help)
  49. cat <<'EOF'
  50. Usage: diagnose-cluster.sh --cluster <name-or-arn> --region <region> [--no-color]
  51. diagnose-cluster.sh --validate --region <region> \
  52. --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
  53. [--s3-uri s3://bucket/path/] [--instance-type ml.p5.48xlarge]
  54. Read-only diagnostic for HyperPod cluster-level issues: provisioning, access,
  55. node replacement, VPC/SG, EKS config + add-ons, SSM, CloudWatch logs. Each
  56. [FAIL] line in the summary includes a pointer of the form
  57. "→ references/cluster-diagnostics-detail.md § <section>"
  58. so the hyperpod-cluster-debugger skill can look up the remediation runbook.
  59. The script never modifies cluster state and never prints remediation commands.
  60. Modes:
  61. (default) Diagnose an existing cluster.
  62. --validate Pre-flight config validation (validates SGs / subnets / IAM /
  63. VPC endpoints / optional S3 lifecycle scripts / optional per-AZ
  64. instance-type capacity before creating a cluster; no cluster
  65. needed).
  66. See references/cluster-diagnostics-detail.md for full remediation runbooks.
  67. See references/capacity-planning.md, lifecycle-scripts.md, cloudformation-errors.md
  68. for deep-dive companions to sections B / C / H.
  69. EOF
  70. exit 0
  71. ;;
  72. *) echo "Unknown argument: $1"; exit 2 ;;
  73. esac
  74. done
  75. if [[ -z "$REGION" ]]; then
  76. echo "ERROR: --region is required (or set AWS_DEFAULT_REGION before running)." >&2
  77. exit 2
  78. fi
  79. if ! "$VALIDATE_MODE"; then
  80. [[ -z "$CLUSTER" ]] && echo "Usage: $0 --cluster <name-or-arn> --region <region>" && exit 1
  81. fi
  82. if ! [ -t 1 ] || [ "${TERM:-}" = "dumb" ]; then
  83. USE_COLOR=false
  84. fi
  85. if "$USE_COLOR"; then
  86. RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
  87. CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
  88. else
  89. RED=''; GREEN=''; YELLOW=''; CYAN=''; BOLD=''; NC=''
  90. fi
  91. CALLER_IDENTITY=$(aws sts get-caller-identity --output json 2>&1) || {
  92. echo -e "${RED}ERROR: AWS credentials not configured or expired.${NC}"
  93. echo "$CALLER_IDENTITY"
  94. echo ""
  95. echo "→ references/cluster-diagnostics-detail.md § D (EKS Access / kubectl) for credential setup"
  96. exit 1
  97. }
  98. CALLER_ARN=$(echo "$CALLER_IDENTITY" | python3 -c "import sys,json; print(json.load(sys.stdin).get('Arn','unknown'))" 2>/dev/null || echo "unknown")
  99. CRITICAL_FAILURES=0
  100. WARNINGS=0
  101. ISSUES_FOUND=()
  102. pass() { echo -e " ${GREEN}[PASS]${NC} $1${2:+ — $2}"; }
  103. fail() { CRITICAL_FAILURES=$((CRITICAL_FAILURES+1)); echo -e " ${RED}[FAIL]${NC} $1${2:+ — $2}"; }
  104. warn() { WARNINGS=$((WARNINGS+1)); echo -e " ${YELLOW}[WARN]${NC} $1${2:+ — $2}"; }
  105. info() { echo -e " ${CYAN}[INFO]${NC} $1${2:+ — $2}"; }
  106. header() { echo ""; echo -e "${BOLD}--- $1 ---${NC}"; }
  107. section() { echo ""; echo -e "${BOLD}=== $1 ===${NC}"; }
  108. add_issue() {
  109. local priority="${2:-P1}"
  110. ISSUES_FOUND+=("${priority}|$1")
  111. }
  112. _CD_TEMP_FILES=()
  113. trap '[[ ${#_CD_TEMP_FILES[@]} -gt 0 ]] && rm -f "${_CD_TEMP_FILES[@]}" 2>/dev/null || true' EXIT
  114. # Run a shell command on a HyperPod instance via SSM. Payload is base64-encoded
  115. # so shell metacharacters in the command are safely passed through argv.
  116. ssm_run_on_node() {
  117. local iid="$1" grp="$2" cmd="$3"
  118. [[ -z "$iid" || -z "$grp" || -z "$cmd" ]] && return 1
  119. [[ ! "$iid" =~ ^i-[0-9a-f]{8,17}$ ]] && return 1
  120. [[ -z "${CLUSTER_ID:-}" ]] && return 1
  121. [[ ! "$grp" =~ ^[A-Za-z0-9._-]+$ ]] && return 1
  122. local target="sagemaker-cluster:${CLUSTER_ID}_${grp}-${iid}"
  123. local tmp; tmp=$(mktemp 2>/dev/null) || return 1
  124. chmod 600 "$tmp" 2>/dev/null || true
  125. _CD_TEMP_FILES+=("$tmp")
  126. local cmd_b64
  127. cmd_b64=$(printf '%s' "$cmd" | base64 | tr -d '\n') || return 1
  128. local remote="bash -c \"echo $cmd_b64 | base64 -d | bash\""
    • Note: Quoted as data in code: Decodes base64 and runs the result in a shell
    • Line 146 on GitHub
  129. python3 -c "import json,sys; print(json.dumps({'command':[sys.argv[1]]}))" "$remote" > "$tmp" || return 1
  130. # unbuffer avoids the session-manager-plugin "Cannot perform start session:
  131. # EOF" race. Only required on Slurm clusters (controller probe); guard at
  132. # call site so EKS-only users aren't blocked if unbuffer is absent.
  133. local _ssm_wrap=""
  134. command -v unbuffer >/dev/null 2>&1 && _ssm_wrap="unbuffer"
  135. local attempt=0 out rc
  136. while (( attempt < 5 )); do
  137. out=$($_ssm_wrap timeout 180 aws ssm start-session \
  138. --target "$target" \
  139. --document-name AWS-StartNonInteractiveCommand \
  140. --parameters "file://$tmp" \
  141. --region "$REGION" 2>&1)
  142. rc=$?
  143. # Retry transient SSM transport errors (rc=0 with EOF/plugin/timeout in stdout).
  144. if (( rc == 0 )) && ! echo "$out" | grep -qiE "Cannot perform start session|EOF$|SessionManagerPlugin is not found|ERROR: Unable to|i/o timeout"; then
  145. # Strip SSM session banners and the echoed base64 command line.
  146. echo "$out" | grep -vE '^(Starting session with SessionId:|Exiting session with sessionId:|\s*$)' \
  147. | grep -vE "^(bash -c \"echo [A-Za-z0-9+/=]+ \| base64 -d \| bash\"|echo '[A-Za-z0-9+/=]+'|[A-Za-z0-9+/=]{40,}={0,2})[[:space:]]*\|?[[:space:]]*base64?[[:space:]]*-?d?[[:space:]]*\|?[[:space:]]*bash\"?\$" || true
  148. return 0
  149. fi
  150. if echo "$out" | grep -qiE "ThrottlingException|RequestLimitExceeded|InternalFailure|InternalError|ServiceUnavailable|TooManyUpdates|Cannot perform start session|EOF$|SessionManagerPlugin is not found|i/o timeout"; then
  151. attempt=$((attempt + 1))
  152. sleep $((attempt * 3))
  153. continue
  154. fi
  155. echo "$out" >&2
  156. return 1
  157. done
  158. return 1
  159. }
  160. # Check SG self-referencing rules. Reads SG JSON from stdin, outputs PASS/FAIL/WARN lines.
  161. check_sg_self_ref() {
  162. local sg_id="$1"
  163. SG_CHECK_ID="$sg_id" python3 -c "
  164. import sys, json, os
  165. sg_id = os.environ['SG_CHECK_ID']
  166. sgs = json.load(sys.stdin).get('SecurityGroups', [])
  167. if not sgs:
  168. print(f'SKIP:Could not describe {sg_id}')
  169. sys.exit(0)
  170. sg = sgs[0]
  171. inbound_self = any(
  172. any(p.get('GroupId') == sg_id for p in r.get('UserIdGroupPairs', []))
  173. for r in sg.get('IpPermissions', [])
  174. )
  175. outbound_self = any(
  176. any(p.get('GroupId') == sg_id for p in r.get('UserIdGroupPairs', []))
  177. for r in sg.get('IpPermissionsEgress', [])
  178. )
  179. outbound_all = any(
  180. any(r2.get('CidrIp') == '0.0.0.0/0' for r2 in r.get('IpRanges', []))
  181. for r in sg.get('IpPermissionsEgress', [])
  182. )
  183. if inbound_self: print(f'PASS:inbound:SG {sg_id}: Inbound self-ref present')
  184. else: print(f'FAIL:inbound:SG {sg_id}: Inbound self-ref MISSING — required for inter-node communication')
  185. if outbound_self: print(f'PASS:outbound:SG {sg_id}: Outbound self-ref present')
  186. else: print(f'FAIL:outbound:SG {sg_id}: Outbound self-ref MISSING — required for EFA RDMA traffic')
  187. if outbound_all: print(f'PASS:internet:SG {sg_id}: Outbound 0.0.0.0/0 present')
  188. else: print(f'WARN:internet:SG {sg_id}: Outbound 0.0.0.0/0 missing — may be needed for AWS API calls')
  189. " 2>/dev/null || echo ""
  190. }
  191. # AWS API wrapper that detects permission failures
  192. aws_check() {
  193. local api_label="$1"; shift
  194. local result
  195. result=$("$@" 2>&1)
  196. local rc=$?
  197. if [[ $rc -ne 0 ]]; then
  198. if echo "$result" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized|AuthorizationError"; then
  199. warn "$api_label" "IAM permission denied — results may be incomplete"
  200. add_issue "Missing IAM permission for $api_label → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
  201. echo ""
  202. return 1
  203. fi
  204. echo "$result"
  205. return "$rc"
  206. fi
  207. echo "$result"
  208. }
  209. if "$VALIDATE_MODE"; then
  210. section "HyperPod Pre-Creation Validation"
  211. echo -e "Region: ${BOLD}${REGION}${NC}"
  212. echo -e "Caller: ${BOLD}${CALLER_ARN}${NC}"
  213. echo -e "Time: $(date -u +"%Y-%m-%dT%H:%M:%SZ")"
  214. if [[ -n "$VALIDATE_SG_IDS" ]]; then
  215. header "V1. Security Group Rules"
  216. for SG in $(echo "$VALIDATE_SG_IDS" | tr ',' ' '); do
  217. SG_JSON=$(aws_check "describe-sg-$SG" aws ec2 describe-security-groups \
  218. --group-ids "$SG" --region "$REGION" --output json) || continue
  219. _SG_CHECK_OUT=$(echo "$SG_JSON" | check_sg_self_ref "$SG")
  220. while IFS=: read -r level check msg; do
  221. [[ -z "$level" ]] && continue
  222. case "$level" in
  223. PASS) pass "$msg" ;;
  224. FAIL)
  225. fail "$msg"
  226. add_issue "SG $SG missing $check self-ref → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
  227. ;;
  228. WARN) warn "$msg" ;;
  229. esac
  230. done <<< "$_SG_CHECK_OUT"
  231. done
  232. fi
  233. if [[ -n "$VALIDATE_SUBNET_IDS" ]]; then
  234. header "V2. Subnet Configuration"
  235. IFS=',' read -ra _subnet_ids <<< "$VALIDATE_SUBNET_IDS"
  236. SUB_JSON=$(aws_check "describe-subnets" aws ec2 describe-subnets \
  237. --subnet-ids "${_subnet_ids[@]}" \
  238. --region "$REGION" --output json) || SUB_JSON='{"Subnets":[]}'
  239. echo "$SUB_JSON" | python3 -c "
  240. import sys, json
  241. subnets = json.load(sys.stdin).get('Subnets', [])
  242. vpcs = set()
  243. azs = set()
  244. for s in subnets:
  245. sid = s.get('SubnetId', '?')
  246. vpc = s.get('VpcId', '?')
  247. az = s.get('AvailabilityZone', '?')
  248. free = s.get('AvailableIpAddressCount', 0)
  249. vpcs.add(vpc)
  250. azs.add(az)
  251. status = 'LOW' if free < 10 else 'OK'
  252. print(f'SUBNET:{sid}:{vpc}:{az}:{free}:{status}')
  253. print(f'VPC_COUNT:{len(vpcs)}')
  254. print(f'AZ_COUNT:{len(azs)}')
  255. " 2>/dev/null | while IFS=: read -r tag rest; do
  256. case "$tag" in
  257. SUBNET)
  258. IFS=: read -r sid _vpc az free status <<< "$rest"
  259. if [[ "$status" == "LOW" ]]; then
  260. warn "Subnet $sid (AZ=$az) — only $free IPs available"
  261. else
  262. pass "Subnet $sid" "AZ=$az FreeIPs=$free"
  263. fi
  264. ;;
  265. VPC_COUNT)
  266. if [[ "$rest" -gt 1 ]]; then
  267. fail "Subnets are in DIFFERENT VPCs — all must be in same VPC"
  268. add_issue "Subnets in different VPCs → references/cluster-diagnostics-detail.md § B (Capacity & AZ)" "P0"
  269. else
  270. pass "All subnets in same VPC"
  271. fi
  272. ;;
  273. AZ_COUNT)
  274. info "Subnets span $rest availability zone(s)"
  275. ;;
  276. esac
  277. done
  278. fi
  279. if [[ -n "$VALIDATE_IAM_ROLE" ]]; then
  280. header "V3. IAM Execution Role"
  281. ROLE_NAME=$(echo "$VALIDATE_IAM_ROLE" | awk -F/ '{print $NF}')
  282. ROLE_INFO=$(aws_check "get-role" aws iam get-role --role-name "$ROLE_NAME" --output json) || ROLE_INFO=""
  283. if [[ -n "$ROLE_INFO" ]]; then
  284. pass "IAM role exists" "$ROLE_NAME"
  285. TRUST_SM=$(echo "$ROLE_INFO" | python3 -c "
  286. import sys,json
  287. doc=json.load(sys.stdin).get('Role',{}).get('AssumeRolePolicyDocument',{})
  288. stmts=doc.get('Statement',[])
  289. for s in stmts:
  290. p=s.get('Principal',{})
  291. svc=p.get('Service',[]) if isinstance(p.get('Service'), list) else [p.get('Service','')]
  292. if 'sagemaker.amazonaws.com' in svc:
  293. print('true')
  294. break
  295. else:
  296. print('false')
  297. " 2>/dev/null)
  298. if [[ "$TRUST_SM" == "true" ]]; then
  299. pass "Trust policy" "allows sagemaker.amazonaws.com"
  300. else
  301. fail "Trust policy" "missing sagemaker.amazonaws.com — cluster creation will fail"
  302. add_issue "IAM execution role trust policy missing sagemaker.amazonaws.com → references/cluster-diagnostics-detail.md § H (CloudFormation Errors / SLR)" "P0"
  303. fi
  304. POLICIES=$(aws_check "list-attached-role-policies-$ROLE_NAME" \
  305. aws iam list-attached-role-policies --role-name "$ROLE_NAME" \
  306. --query 'AttachedPolicies[*].PolicyArn' --output text) || POLICIES=""
  307. if [[ -n "$POLICIES" ]]; then
  308. if echo "$POLICIES" | grep -q "AmazonSageMakerClusterInstanceRolePolicy"; then
  309. pass "Managed policy" "AmazonSageMakerClusterInstanceRolePolicy attached"
  310. else
  311. warn "Managed policy" "AmazonSageMakerClusterInstanceRolePolicy not attached — cluster bootstrap will fail"
  312. add_issue "IAM execution role missing AmazonSageMakerClusterInstanceRolePolicy → references/cluster-diagnostics-detail.md § H (CloudFormation Errors / SLR)" "P0"
  313. fi
  314. if echo "$POLICIES" | grep -q "AmazonSSMManagedInstanceCore"; then
  315. pass "Managed policy" "AmazonSSMManagedInstanceCore attached (SSM access)"
  316. else
  317. warn "Managed policy" "AmazonSSMManagedInstanceCore not attached — SSM node access will not work"
  318. add_issue "IAM execution role missing AmazonSSMManagedInstanceCore → references/cluster-diagnostics-detail.md § F (SSM Connectivity)" "P1"
  319. fi
  320. fi
  321. else
  322. fail "IAM role" "cannot find role '$ROLE_NAME'"
  323. add_issue "IAM execution role not found → references/cluster-diagnostics-detail.md § H (CloudFormation Errors / SLR)" "P0"
  324. fi
  325. fi
  326. if [[ -n "$VALIDATE_SUBNET_IDS" ]]; then
  327. header "V4. VPC Endpoints"
  328. FIRST_SUBNET=$(echo "$VALIDATE_SUBNET_IDS" | cut -d, -f1)
  329. VPC_FOR_EP=$(aws ec2 describe-subnets --subnet-ids "$FIRST_SUBNET" \
  330. --region "$REGION" --query 'Subnets[0].VpcId' --output text 2>/dev/null || echo "")
  331. if [[ -n "$VPC_FOR_EP" && "$VPC_FOR_EP" != "None" ]]; then
  332. ENDPOINTS=$(aws ec2 describe-vpc-endpoints \
  333. --filters "Name=vpc-id,Values=$VPC_FOR_EP" \
  334. --region "$REGION" \
  335. --query "VpcEndpoints[?State==\`available\`].ServiceName" \
  336. --output text 2>/dev/null || echo "")
  337. for SVC in s3 ssm ssmmessages ec2messages; do
  338. if echo "$ENDPOINTS" | grep -qE "(^|[.])${SVC}($|[[:space:]])"; then
  339. pass "VPC endpoint: $SVC"
  340. else
  341. warn "VPC endpoint: $SVC" "not found — needed for private VPC clusters"
  342. add_issue "Missing VPC endpoint for $SVC → references/cluster-diagnostics-detail.md § C (Lifecycle Scripts)" "P2"
  343. fi
  344. done
  345. fi
  346. fi
  347. if [[ -n "$VALIDATE_INSTANCE_TYPE" && -n "$VALIDATE_SUBNET_IDS" ]]; then
  348. header "V5. Instance-Type Capacity per AZ"
  349. # EC2 API takes the bare type, not the ml. prefix.
  350. EC2_TYPE="${VALIDATE_INSTANCE_TYPE#ml.}"
  351. AZ_OFFERINGS=$(aws_check "describe-instance-type-offerings-$EC2_TYPE" \
  352. aws ec2 describe-instance-type-offerings \
  353. --location-type availability-zone \
  354. --filters "Name=instance-type,Values=${EC2_TYPE}" \
  355. --region "$REGION" \
  356. --query 'InstanceTypeOfferings[*].Location' --output text) || AZ_OFFERINGS=""
  357. if [[ -z "$AZ_OFFERINGS" ]]; then
  358. fail "Instance type $VALIDATE_INSTANCE_TYPE" "not offered in region $REGION"
  359. add_issue "$VALIDATE_INSTANCE_TYPE is not offered in any AZ in $REGION → references/capacity-planning.md" "P0"
  360. else
  361. info "$VALIDATE_INSTANCE_TYPE available in AZ(s): $AZ_OFFERINGS"
  362. IFS=',' read -ra _subnet_ids <<< "$VALIDATE_SUBNET_IDS"
  363. SUB_AZ_JSON=$(aws_check "describe-subnets-validate" aws ec2 describe-subnets \
  364. --subnet-ids "${_subnet_ids[@]}" \
  365. --region "$REGION" \
  366. --query 'Subnets[*].{SubnetId:SubnetId,AZ:AvailabilityZone}' --output json) || SUB_AZ_JSON="[]"
  367. MATCHED=0
  368. while IFS=$'\t' read -r sid az; do
  369. [[ -z "$sid" ]] && continue
  370. if echo "$AZ_OFFERINGS" | tr '\t' '\n' | grep -qx "$az"; then
  371. pass "Subnet $sid (AZ=$az)" "$VALIDATE_INSTANCE_TYPE is available"
  372. MATCHED=$((MATCHED+1))
  373. else
  374. fail "Subnet $sid (AZ=$az)" "$VALIDATE_INSTANCE_TYPE NOT offered here"
  375. add_issue "Subnet $sid AZ=$az does not offer $VALIDATE_INSTANCE_TYPE → references/capacity-planning.md" "P0"
  376. fi
  377. done < <(echo "$SUB_AZ_JSON" | python3 -c "
  378. import sys, json
  379. for s in json.load(sys.stdin):
  380. print(f\"{s.get('SubnetId','')}\t{s.get('AZ','')}\")
  381. " 2>/dev/null)
  382. if [[ $MATCHED -eq 0 ]]; then
  383. warn "No provided subnet is in an AZ that offers $VALIDATE_INSTANCE_TYPE — cluster creation will fail with Insufficient capacity / No subnets in the capacity AZ"
  384. fi
  385. fi
  386. fi
  387. if [[ -n "$VALIDATE_S3_URI" ]]; then
  388. header "V6. S3 Lifecycle Scripts"
  389. if [[ ! "$VALIDATE_S3_URI" =~ ^s3:// ]]; then
  390. fail "S3 URI" "must start with s3:// (got '$VALIDATE_S3_URI')"
  391. add_issue "S3 URI is not a valid s3:// URI → references/lifecycle-scripts.md" "P0"
  392. else
  393. S3_URI_NORM="${VALIDATE_S3_URI%/}/"
  394. info "S3 URI: $S3_URI_NORM"
  395. S3_LIST=$(aws_check "s3-ls-$S3_URI_NORM" \
  396. aws s3 ls "$S3_URI_NORM" --region "$REGION") || S3_LIST=""
  397. if [[ -z "$S3_LIST" ]]; then
  398. fail "S3 access" "cannot list $S3_URI_NORM — bucket missing, permissions denied, or empty prefix"
  399. add_issue "S3 URI not accessible or empty: $S3_URI_NORM → references/lifecycle-scripts.md" "P0"
  400. else
  401. pass "S3 access" "prefix is listable"
  402. if echo "$S3_LIST" | grep -q "on_create.sh"; then
  403. pass "on_create.sh" "entry script present"
  404. TMPFILE=$(mktemp)
  405. if aws s3 cp "${S3_URI_NORM}on_create.sh" "$TMPFILE" \
  406. --region "$REGION" --only-show-errors 2>/dev/null; then
  407. if file "$TMPFILE" | grep -q "CRLF"; then
  408. fail "on_create.sh" "has Windows CRLF line endings — will fail on Linux"
  409. add_issue "on_create.sh has CRLF line endings → references/lifecycle-scripts.md" "P0"
  410. else
  411. pass "on_create.sh" "Unix line endings"
  412. fi
  413. if head -1 "$TMPFILE" | grep -q "^#!"; then
  414. pass "on_create.sh" "shebang present"
  415. else
  416. warn "on_create.sh" "missing shebang (#!/bin/bash)"
  417. add_issue "on_create.sh missing shebang → references/lifecycle-scripts.md" "P1"
  418. fi
  419. else
  420. warn "on_create.sh" "could not download for inspection"
  421. fi
  422. rm -f "$TMPFILE"
  423. else
  424. fail "on_create.sh" "entry script NOT FOUND at $S3_URI_NORM — cluster creation will fail"
  425. add_issue "Missing on_create.sh at $S3_URI_NORM → references/lifecycle-scripts.md" "P0"
  426. fi
  427. if echo "$S3_LIST" | grep -q "lifecycle_script.py"; then
  428. pass "Orchestrator script" "lifecycle_script.py present (Slurm)"
  429. elif echo "$S3_LIST" | grep -q "on_create_main.sh"; then
  430. pass "Orchestrator script" "on_create_main.sh present (EKS)"
  431. else
  432. warn "Orchestrator script" "neither lifecycle_script.py (Slurm) nor on_create_main.sh (EKS) found at $S3_URI_NORM"
  433. add_issue "Missing orchestrator-specific lifecycle script at $S3_URI_NORM → references/lifecycle-scripts.md" "P1"
  434. fi
  435. fi
  436. fi
  437. fi
  438. echo ""
  439. echo -e "${BOLD}========================================${NC}"
  440. echo -e "${BOLD} VALIDATION SUMMARY ${NC}"
  441. echo -e "${BOLD}========================================${NC}"
  442. echo ""
  443. echo -e " Results: ${RED}${CRITICAL_FAILURES} critical${NC} | ${YELLOW}${WARNINGS} warnings${NC}"
  444. echo -e " Mode: READ-ONLY (no changes made; each [FAIL] points to a references section)"
  445. echo ""
  446. if [[ ${#ISSUES_FOUND[@]} -gt 0 ]]; then
  447. echo -e "${BOLD} Issues:${NC}"
  448. for priority in P0 P1 P2; do
  449. for issue in "${ISSUES_FOUND[@]}"; do
  450. if [[ "$issue" == "${priority}|"* ]]; then
  451. desc="${issue#*|}"
  452. case "$priority" in
  453. P0) echo -e " ${RED}[${priority}]${NC} $desc" ;;
  454. P1) echo -e " ${YELLOW}[${priority}]${NC} $desc" ;;
  455. P2) echo -e " [${priority}] $desc" ;;
  456. esac
  457. fi
  458. done
  459. done
  460. echo ""
  461. fi
  462. if [[ $CRITICAL_FAILURES -eq 0 ]]; then
  463. echo -e " ${GREEN}${BOLD}Pre-flight validation passed. Safe to create cluster.${NC}"
  464. else
  465. echo -e " ${RED}${BOLD}Fix P0 issues above before creating the cluster.${NC}"
  466. fi
  467. echo ""
  468. exit "$([[ $CRITICAL_FAILURES -eq 0 ]] && echo 0 || echo 1)"
  469. fi
  470. section "HyperPod Cluster Diagnostics (read-only)"
  471. echo -e "Cluster: ${BOLD}${CLUSTER}${NC}"
  472. echo -e "Region: ${BOLD}${REGION}${NC}"
  473. echo -e "Time: $(date -u +"%Y-%m-%dT%H:%M:%SZ")"
  474. echo -e "${CYAN} No cluster state will be modified. Each issue line below includes a${NC}"
  475. echo -e "${CYAN} pointer to references/cluster-diagnostics-detail.md for remediation.${NC}"
  476. header "1. Cluster Identity & Status"
  477. CLUSTER_JSON=$(aws sagemaker describe-cluster \
  478. --cluster-name "$CLUSTER" \
  479. --region "$REGION" \
  480. --cli-read-timeout 30 \
  481. --output json 2>&1) || {
  482. echo -e "${RED}ERROR: Could not describe cluster '$CLUSTER' in region '$REGION'${NC}"
  483. echo "$CLUSTER_JSON" | head -3
  484. echo ""
  485. if echo "$CLUSTER_JSON" | grep -qiE "ResourceNotFound|Cluster with name .* not found"; then
  486. echo "Available clusters in $REGION:"
  487. aws sagemaker list-clusters --region "$REGION" \
  488. --query 'ClusterSummaries[*].{Name:ClusterName,Status:ClusterStatus}' \
  489. --output table 2>/dev/null || echo " (unable to list clusters — check IAM)"
  490. else
  491. echo "Verify:"
  492. echo " 1. Cluster name is correct (use: aws sagemaker list-clusters --region $REGION)"
  493. echo " 2. Region is correct"
  494. echo " 3. IAM permissions include sagemaker:DescribeCluster"
  495. fi
  496. exit 1
  497. }
  498. CLUSTER_ARN=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ClusterArn',''))" 2>/dev/null)
  499. CLUSTER_ID=$(echo "$CLUSTER_ARN" | awk -F'/' '{print $NF}')
  500. if [[ -z "$CLUSTER_ID" ]]; then
  501. echo "ERROR: Could not extract cluster ID from ARN '$CLUSTER_ARN'. Verify the cluster name/ARN."
  502. exit 1
  503. fi
  504. CLUSTER_STATUS=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ClusterStatus','unknown'))" 2>/dev/null)
  505. ORCHESTRATOR=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); o=d.get('Orchestrator',{}); print('EKS' if 'Eks' in o else 'Slurm')" 2>/dev/null)
  506. NODE_RECOVERY=$(echo "$CLUSTER_JSON" | python3 -c "
  507. import sys,json
  508. d=json.load(sys.stdin)
  509. # Prefer cluster-level NodeRecovery (the API's canonical location); fall back to
  510. # per-InstanceGroup only when top-level is absent. Reading only per-group yields
  511. # 'Unknown' on every cluster because the field is null at group level when set
  512. # cluster-wide.
  513. top=d.get('NodeRecovery')
  514. if top:
  515. print(top)
  516. else:
  517. groups=d.get('InstanceGroups',[])
  518. recoveries={g.get('NodeRecovery') for g in groups if g.get('NodeRecovery')}
  519. print(','.join(sorted(recoveries)) if recoveries else 'Unknown')
  520. " 2>/dev/null || echo "Unknown")
  521. info "ARN: $CLUSTER_ARN"
  522. info "Cluster ID: $CLUSTER_ID"
  523. info "Status: $CLUSTER_STATUS"
  524. info "Orchestrator: $ORCHESTRATOR"
  525. info "NodeRecovery: $NODE_RECOVERY"
  526. # Flag auto-recovery disabled regardless of orchestrator.
  527. if [[ "$NODE_RECOVERY" == *"None"* && "$NODE_RECOVERY" == *"Automatic"* ]]; then
  528. warn "NodeRecovery" "mixed settings — some instance groups have recovery disabled"
  529. add_issue "NodeRecovery disabled on some instance groups → references/cluster-diagnostics-detail.md § G (Node Replacement)" "P2"
  530. elif [[ "$NODE_RECOVERY" == *"None"* ]]; then
  531. warn "NodeRecovery" "disabled on all instance groups — auto-replacement won't trigger"
  532. add_issue "NodeRecovery disabled → references/cluster-diagnostics-detail.md § G (Node Replacement)" "P2"
  533. fi
  534. CREATION_TIME=$(echo "$CLUSTER_JSON" | python3 -c "
  535. import sys,json
  536. d=json.load(sys.stdin)
  537. ct=d.get('CreationTime','')
  538. print(ct if ct else '')
  539. " 2>/dev/null || echo "")
  540. LAST_MODIFIED_TIME=$(echo "$CLUSTER_JSON" | python3 -c "
  541. import sys,json
  542. d=json.load(sys.stdin)
  543. lm=d.get('LastModifiedTime','')
  544. print(lm if lm else '')
  545. " 2>/dev/null || echo "")
  546. STUCK_THRESHOLD_SECONDS=3600
  547. is_stuck() {
  548. local creation_time="$1"
  549. if [[ -z "$creation_time" ]]; then echo "false"; return; fi
  550. CREATION_TS="$creation_time" THRESHOLD="$STUCK_THRESHOLD_SECONDS" python3 -c "
  551. import os
  552. from datetime import datetime, timezone
  553. ct = os.environ['CREATION_TS']
  554. threshold = int(os.environ['THRESHOLD'])
  555. try:
  556. ct=ct.replace('+00:00','Z').rstrip('Z')
  557. if '.' in ct: ct=ct[:ct.index('.')+7]
  558. created=datetime.fromisoformat(ct).replace(tzinfo=timezone.utc)
  559. elapsed=(datetime.now(timezone.utc)-created).total_seconds()
  560. print('true' if elapsed > threshold else 'false')
  561. except (ValueError, TypeError):
  562. # Unparseable timestamp — assume not stuck rather than abort the whole run.
  563. print('false')
  564. " 2>/dev/null || echo "false"
  565. }
  566. case "$CLUSTER_STATUS" in
  567. InService) pass "Cluster status" "InService" ;;
  568. Creating)
  569. STUCK=$(is_stuck "$CREATION_TIME")
  570. if [[ "$STUCK" == "true" ]]; then
  571. fail "Cluster status" "Creating for over 1 hour — likely stuck"
  572. add_issue "Cluster stuck in Creating > 1hr → references/cluster-diagnostics-detail.md § E (Cluster Provisioning), § H (CloudFormation)" "P0"
  573. else
  574. warn "Cluster status" "Creating — cluster is still being provisioned"
  575. add_issue "Cluster still creating → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P1"
  576. fi ;;
  577. Updating)
  578. STUCK=$(is_stuck "${LAST_MODIFIED_TIME:-$CREATION_TIME}")
  579. if [[ "$STUCK" == "true" ]]; then
  580. fail "Cluster status" "Updating — check if operation is stuck"
  581. add_issue "Cluster may be stuck Updating → references/cluster-diagnostics-detail.md § E (Cluster Provisioning), § H (CloudFormation)" "P1"
  582. else
  583. warn "Cluster status" "Updating — cluster operation in progress"
  584. fi ;;
  585. Failed) fail "Cluster status" "Failed — check events and CloudFormation"; add_issue "Cluster FAILED → references/cluster-diagnostics-detail.md § E (Cluster Provisioning), § H (CloudFormation)" "P0" ;;
  586. Deleting)
  587. STUCK=$(is_stuck "${LAST_MODIFIED_TIME:-$CREATION_TIME}")
  588. if [[ "$STUCK" == "true" ]]; then
  589. warn "Cluster status" "Deleting for extended time — may be blocked by VPC ENI dependencies"
  590. add_issue "Cluster stuck Deleting → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P1"
  591. else
  592. warn "Cluster status" "Deleting"
  593. fi ;;
  594. RollingBack) warn "Cluster status" "RollingBack — update is being rolled back"; add_issue "Cluster RollingBack → references/cluster-diagnostics-detail.md § J (AMI & Cluster Updates)" "P1" ;;
  595. *RollbackFailed*|*MaintenanceFailed*)
  596. fail "Cluster status" "$CLUSTER_STATUS — cluster is stuck in a non-recoverable state"
  597. add_issue "Cluster stuck in $CLUSTER_STATUS → references/cluster-diagnostics-detail.md § J (AMI & Cluster Updates)" "P0" ;;
  598. *) warn "Cluster status" "$CLUSTER_STATUS" ;;
  599. esac
  600. EKS_NAME=""
  601. if [[ "$ORCHESTRATOR" == "EKS" ]]; then
  602. EKS_NAME=$(echo "$CLUSTER_JSON" | python3 -c "
  603. import sys,json
  604. d=json.load(sys.stdin)
  605. arn=d.get('Orchestrator',{}).get('Eks',{}).get('ClusterArn','')
  606. print(arn.split('/')[-1] if arn else '')
  607. " 2>/dev/null || echo "")
  608. if [[ -n "$EKS_NAME" ]]; then
  609. info "EKS Cluster: $EKS_NAME"
  610. fi
  611. fi
  612. header "2. Instance Groups & Node Health"
  613. echo "$CLUSTER_JSON" | python3 -c "
  614. import sys, json
  615. d = json.load(sys.stdin)
  616. groups = d.get('InstanceGroups', [])
  617. if not groups:
  618. print(' No instance groups found')
  619. else:
  620. for g in groups:
  621. name = g.get('InstanceGroupName', '?')
  622. itype = g.get('InstanceType', '?')
  623. target = g.get('TargetCount', 0)
  624. current = g.get('CurrentCount', 0)
  625. status = g.get('Status', g.get('InstanceGroupStatus', '?'))
  626. threads = g.get('ThreadsPerCore', '?')
  627. # TargetStateCount is the count the service is working toward when a
  628. # resize is in flight; print when it differs from TargetCount.
  629. tstate = g.get('TargetStateCount', None)
  630. # Note: NodeRecovery is a cluster-level field in the DescribeCluster
  631. # response, not per-group; shown on the cluster header line above.
  632. print(f' {name}: type={itype} target={target} current={current} status={status} threads/core={threads}')
  633. if tstate is not None and tstate != target:
  634. print(f' TargetStateCount={tstate} (resize in progress)')
  635. if current < target:
  636. print(f' Current count ({current}) < target ({target}) — instances may still be provisioning or failed')
  637. " 2>/dev/null
  638. # Check node-level details. Paginate — default page is small and large clusters
  639. # silently truncate, which would break dangling-node reconciliation below.
  640. fetch_all_cluster_nodes_cd() {
  641. local merged='[]' token='' page_json combined i=0
  642. local max_pages=200 # 200 × 100 = 20 000 nodes, supports 7k+ clusters
  643. while (( i < max_pages )); do
  644. if [[ -n "$token" ]]; then
  645. page_json=$(aws sagemaker list-cluster-nodes \
  646. --cluster-name "$CLUSTER" --region "$REGION" \
  647. --max-results 100 --next-token "$token" \
  648. --cli-read-timeout 30 --output json 2>&1) || break
  649. else
  650. page_json=$(aws sagemaker list-cluster-nodes \
  651. --cluster-name "$CLUSTER" --region "$REGION" \
  652. --max-results 100 \
  653. --cli-read-timeout 30 --output json 2>&1) || break
  654. fi
  655. if echo "$page_json" | grep -qiE "AccessDenied|not authorized|UnauthorizedAccess"; then
  656. echo "__AUTH_DENIED__"
  657. return 1
  658. fi
  659. # Merge via stdin (NUL-delimited blobs) instead of argv — argv is capped at
  660. # ARG_MAX (~128KB on Linux), which fails at ~500 nodes of accumulated JSON.
  661. # Large clusters (7k+) need this path to avoid silent truncation.
  662. combined=$(printf '%s\0%s' "$merged" "$page_json" | python3 -c "
  663. import sys, json
  664. blob = sys.stdin.buffer.read()
  665. try:
  666. a, b = blob.split(b'\0', 1)
  667. merged = json.loads(a)
  668. page = json.loads(b)
  669. except (json.JSONDecodeError, ValueError):
  670. sys.exit(2)
  671. merged.extend(page.get('ClusterNodeSummaries', []))
  672. print(json.dumps(merged))
  673. print(page.get('NextToken', ''))
  674. " 2>/dev/null) || break
  675. merged=$(printf '%s\n' "$combined" | sed -n '1p')
  676. token=$(printf '%s\n' "$combined" | sed -n '2p')
  677. i=$((i+1))
  678. [[ -z "$token" ]] && break
  679. done
  680. if (( i == max_pages )) && [[ -n "$token" ]]; then
  681. # Surface truncation via a marker file — this function runs inside $(...)
  682. # (command substitution subshell), so add_issue would be lost. The parent
  683. # shell checks for the marker after the call returns.
  684. echo "WARN: list-cluster-nodes truncated at ${max_pages} pages (~$((max_pages*100)) nodes). Diagnostic sample is incomplete for very large clusters." >&2
  685. : > "${_NODE_TRUNC_MARKER:-/dev/null}" 2>/dev/null || true
  686. fi
  687. printf '%s' "$merged" | python3 -c "
  688. import sys, json
  689. try:
  690. print(json.dumps({'ClusterNodeSummaries': json.loads(sys.stdin.read())}))
  691. except json.JSONDecodeError:
  692. print('{\"ClusterNodeSummaries\":[]}')
  693. " 2>/dev/null || echo '{"ClusterNodeSummaries":[]}'
  694. }
  695. _NODE_TRUNC_MARKER=$(mktemp 2>/dev/null) && _CD_TEMP_FILES+=("$_NODE_TRUNC_MARKER") || _NODE_TRUNC_MARKER=""
  696. export _NODE_TRUNC_MARKER
  697. rm -f "$_NODE_TRUNC_MARKER" 2>/dev/null || true
  698. NODE_LIST=$(fetch_all_cluster_nodes_cd)
  699. if [[ "$NODE_LIST" == "__AUTH_DENIED__" ]]; then
  700. warn "list-cluster-nodes" "IAM permission denied — add sagemaker:ListClusterNodes to your role"
  701. add_issue "Missing IAM permission for sagemaker:ListClusterNodes → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
  702. NODE_LIST='{"ClusterNodeSummaries":[]}'
  703. fi
  704. # Parent-shell follow-up for the truncation marker set inside the subshell.
  705. if [[ -n "$_NODE_TRUNC_MARKER" && -e "$_NODE_TRUNC_MARKER" ]]; then
  706. add_issue "Node list truncated at 200 pages (~20000 nodes); diagnostic sample incomplete → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P2"
  707. fi
  708. TOTAL_NODES=$(echo "$NODE_LIST" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('ClusterNodeSummaries',[])))" 2>/dev/null || echo 0)
  709. info "Total nodes reported: $TOTAL_NODES"
  710. UNHEALTHY_NODES=$(echo "$NODE_LIST" | python3 -c "
  711. import sys, json
  712. nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
  713. unhealthy = [n for n in nodes if n.get('InstanceStatus', {}).get('Status', '') not in ('Running', 'Pending')]
  714. if unhealthy:
  715. for n in unhealthy:
  716. nid = n.get('InstanceId', '?')
  717. group = n.get('InstanceGroupName', '?')
  718. status = n.get('InstanceStatus', {}).get('Status', '?')
  719. msg = n.get('InstanceStatus', {}).get('Message', '')
  720. print(f' {nid} ({group}): {status} {msg}')
  721. print(f'UNHEALTHY_COUNT={len(unhealthy)}')
  722. else:
  723. print('UNHEALTHY_COUNT=0')
  724. " 2>/dev/null || echo "UNHEALTHY_COUNT=0")
  725. UNHEALTHY_COUNT=$(echo "$UNHEALTHY_NODES" | grep "^UNHEALTHY_COUNT=" | cut -d= -f2)
  726. [[ -z "$UNHEALTHY_COUNT" ]] && UNHEALTHY_COUNT=0
  727. echo "$UNHEALTHY_NODES" | grep -v "^UNHEALTHY_COUNT=" || true
  728. if [[ "$UNHEALTHY_COUNT" -gt 0 ]]; then
  729. warn "Node health" "$UNHEALTHY_COUNT unhealthy node(s)"
  730. add_issue "$UNHEALTHY_COUNT unhealthy node(s) → references/cluster-diagnostics-detail.md § G (Node Replacement); delegate to hyperpod-node-debugger" "P1"
  731. echo "$NODE_LIST" | python3 -c "
  732. import sys, json
  733. from collections import defaultdict
  734. nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
  735. groups = defaultdict(lambda: {'total': 0, 'unhealthy': 0})
  736. for n in nodes:
  737. g = n.get('InstanceGroupName', 'unknown')
  738. groups[g]['total'] += 1
  739. st = n.get('InstanceStatus', {}).get('Status', '')
  740. if st not in ('Running', 'Pending', ''):
  741. groups[g]['unhealthy'] += 1
  742. for g, c in groups.items():
  743. if c['unhealthy'] > 0:
  744. pct = int(c['unhealthy'] / c['total'] * 100) if c['total'] > 0 else 0
  745. print(f' [WARN] Group {g}: {c[\"unhealthy\"]}/{c[\"total\"]} unhealthy ({pct}%)')
  746. " 2>/dev/null
  747. elif [[ "$TOTAL_NODES" -eq 0 && "$CLUSTER_STATUS" == "InService" ]]; then
  748. warn "Node health" "Cluster InService but 0 nodes reported"
  749. add_issue "Cluster InService but no nodes → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P1"
  750. else
  751. pass "Node health" "$TOTAL_NODES node(s), $UNHEALTHY_COUNT unhealthy"
  752. fi
  753. header "3. Cluster Events (Recent)"
  754. # Paginate up to 5 pages (500 events) so the event scan covers incident windows
  755. # longer than the default page. Long-lived clusters with rolling replacements
  756. # regularly generate >100 events.
  757. fetch_cluster_events_cd() {
  758. local merged='[]' token='' page_json combined i=0 denied=0
  759. while (( i < 5 )); do
  760. if [[ -n "$token" ]]; then
  761. page_json=$(aws sagemaker list-cluster-events \
  762. --cluster-name "$CLUSTER" --region "$REGION" \
  763. --max-results 100 --next-token "$token" \
  764. --cli-read-timeout 30 --output json 2>&1) || break
  765. else
  766. page_json=$(aws sagemaker list-cluster-events \
  767. --cluster-name "$CLUSTER" --region "$REGION" \
  768. --max-results 100 \
  769. --cli-read-timeout 30 --output json 2>&1) || break
  770. fi
  771. if echo "$page_json" | grep -qi "AccessDenied\|not authorized"; then
  772. denied=1
  773. break
  774. fi
  775. combined=$(python3 -c "
  776. import sys, json
  777. try:
  778. prev = json.loads(sys.argv[1])
  779. page = json.loads(sys.argv[2])
  780. except json.JSONDecodeError:
  781. # Malformed page response — stop paginating; caller falls through on break.
  782. sys.exit(2)
  783. prev.extend(page.get('ClusterEventSummaries', []))
  784. print(json.dumps(prev))
  785. print(page.get('NextToken',''))
  786. " "$merged" "$page_json" 2>/dev/null) || break
  787. merged=$(printf '%s\n' "$combined" | sed -n '1p')
  788. token=$(printf '%s\n' "$combined" | sed -n '2p')
  789. i=$((i+1))
  790. [[ -z "$token" ]] && break
  791. done
  792. if (( denied )); then
  793. echo "__AUTH_DENIED__"
  794. return 1
  795. fi
  796. python3 -c "import sys, json; print(json.dumps({'ClusterEventSummaries': json.loads(sys.argv[1])}))" "$merged" \
  797. 2>/dev/null || echo '{"ClusterEventSummaries":[]}'
  798. }
  799. EVENTS_JSON=$(fetch_cluster_events_cd)
  800. if [[ "$EVENTS_JSON" == "__AUTH_DENIED__" ]]; then
  801. warn "list-cluster-events" "IAM permission denied — add sagemaker:ListClusterEvents to your role"
  802. EVENTS_JSON='{"ClusterEventSummaries":[]}'
  803. fi
  804. EVENT_COUNT=$(echo "$EVENTS_JSON" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('ClusterEventSummaries',[])))" 2>/dev/null || echo 0)
  805. if [[ "$EVENT_COUNT" -eq 0 ]]; then
  806. info "No cluster events found"
  807. if [[ "$ORCHESTRATOR" == "Slurm" ]]; then
  808. info "(Cluster events may not be available for HyperPod Slurm clusters)"
  809. fi
  810. else
  811. echo "$EVENTS_JSON" | python3 -c "
  812. import sys, json
  813. events = json.load(sys.stdin).get('ClusterEventSummaries', [])
  814. # Issue pattern mapping
  815. ISSUE_PATTERNS = {
  816. 'EFA health checks': 'EFA health check failure → references/cluster-diagnostics-detail.md § A',
  817. 'Insufficient capacity': 'Capacity error → references/cluster-diagnostics-detail.md § B',
  818. 'No subnets in the capacity': 'AZ/subnet mismatch → references/cluster-diagnostics-detail.md § B',
  819. 'Lifecycle scripts did not run': 'Lifecycle script failure → references/cluster-diagnostics-detail.md § C',
  820. 'Lifecycle scripts execution timed out': 'Lifecycle script timeout → references/cluster-diagnostics-detail.md § C',
  821. 'network misconfiguration': 'Network misconfiguration → references/cluster-diagnostics-detail.md § A + § B',
  822. 'hardware failure': 'Hardware failure → delegate to node-debugger',
  823. 'Failed to provision': 'Provisioning failure → references/cluster-diagnostics-detail.md § B or § E',
  824. 'replace': 'Node replacement activity → references/cluster-diagnostics-detail.md § G',
  825. 'reboot': 'Node reboot activity → references/cluster-diagnostics-detail.md § G',
  826. }
  827. for e in events[:20]:
  828. ts = str(e.get('EventTime', '?'))[:19]
  829. etype = e.get('EventType', '?')
  830. msg = e.get('Message', '?')[:120]
  831. print(f' [{ts}] {etype}: {msg}')
  832. msg_lower = (e.get('Message','') or '').lower()
  833. for pattern, hint in ISSUE_PATTERNS.items():
  834. if pattern.lower() in msg_lower:
  835. print(f' [ISSUE] {hint}')
  836. break
  837. " 2>/dev/null
  838. fi
  839. header "4. VPC & Security Group Configuration"
  840. SUBNET_IDS=$(echo "$CLUSTER_JSON" | python3 -c "
  841. import sys,json
  842. d=json.load(sys.stdin)
  843. print(' '.join(d.get('VpcConfig',{}).get('Subnets',[])))
  844. " 2>/dev/null || echo "")
  845. SG_IDS=$(echo "$CLUSTER_JSON" | python3 -c "
  846. import sys,json
  847. d=json.load(sys.stdin)
  848. print(' '.join(d.get('VpcConfig',{}).get('SecurityGroupIds',[])))
  849. " 2>/dev/null || echo "")
  850. if [[ -z "$SUBNET_IDS" ]]; then
  851. warn "VpcConfig" "No VpcConfig found in cluster"
  852. else
  853. info "Subnets: $SUBNET_IDS"
  854. info "Security Groups: $SG_IDS"
  855. IFS=' ' read -ra _subnet_ids_arr <<< "$SUBNET_IDS"
  856. SUBNET_JSON=$(aws ec2 describe-subnets \
  857. --subnet-ids "${_subnet_ids_arr[@]}" \
  858. --region "$REGION" \
  859. --cli-read-timeout 30 \
  860. --output json 2>&1) || {
  861. SUB_ERR="$SUBNET_JSON"
  862. if echo "$SUB_ERR" | grep -qi "AccessDenied\|UnauthorizedOperation\|not authorized"; then
  863. warn "describe-subnets" "IAM permission denied — add ec2:DescribeSubnets to your role"
  864. fi
  865. SUBNET_JSON='{"Subnets":[]}'
  866. }
  867. _SUBNET_CHECK=$(echo "$SUBNET_JSON" | python3 -c "
  868. import sys, json
  869. subnets = json.load(sys.stdin).get('Subnets', [])
  870. vpcs = set()
  871. for s in subnets:
  872. sid = s.get('SubnetId', '?')
  873. vpc = s.get('VpcId', '?')
  874. az = s.get('AvailabilityZone', '?')
  875. free = s.get('AvailableIpAddressCount', 0)
  876. flag = ' LOW IPs' if free < 10 else ''
  877. print(f' {sid}: VPC={vpc} AZ={az} FreeIPs={free}{flag}')
  878. vpcs.add(vpc)
  879. if len(vpcs) > 1:
  880. print('MULTI_VPC=true')
  881. print('VPC_LIST=' + ','.join(vpcs))
  882. else:
  883. print('MULTI_VPC=false')
  884. v = vpcs.pop() if vpcs else '?'
  885. print('VPC_ID=' + v)
  886. " 2>/dev/null || echo "")
  887. while IFS= read -r line; do
  888. if [[ "$line" == "MULTI_VPC=true" ]]; then
  889. fail "Subnet VPC alignment" "Subnets are in DIFFERENT VPCs — all must be in the same VPC"
  890. add_issue "Subnets in different VPCs → references/cluster-diagnostics-detail.md § B (Capacity & AZ)" "P0"
  891. fi
  892. if [[ "$line" != MULTI_VPC=* && "$line" != VPC_ID=* && "$line" != VPC_LIST=* ]]; then
  893. echo "$line"
  894. fi
  895. done <<< "$_SUBNET_CHECK"
  896. # SG self-referencing rules are an EFA requirement.
  897. # shellcheck disable=SC2086 # intentional word-split on space-separated SG IDs
  898. for SG in $SG_IDS; do
  899. SG_RESULT=$(aws ec2 describe-security-groups \
  900. --group-ids "$SG" \
  901. --region "$REGION" \
  902. --cli-read-timeout 30 \
  903. --output json 2>&1)
  904. if echo "$SG_RESULT" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
  905. warn "describe-security-groups" "IAM permission denied for $SG — SG check skipped"
  906. continue
  907. fi
  908. SG_JSON="${SG_RESULT}"
  909. [[ -z "$SG_JSON" || "$SG_JSON" == *"error"* ]] && SG_JSON='{"SecurityGroups":[]}'
  910. _SG_CHECK=$(echo "$SG_JSON" | check_sg_self_ref "$SG")
  911. while IFS= read -r line; do
  912. [[ -z "$line" ]] && continue
  913. level=$(echo "$line" | cut -d: -f1)
  914. msg=$(echo "$line" | cut -d: -f2-)
  915. case "$level" in
  916. PASS) pass "$msg" ;;
  917. FAIL) fail "$msg"
  918. if echo "$msg" | grep -q "Inbound self-ref MISSING"; then
  919. add_issue "Security group $SG inbound self-ref MISSING → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
  920. elif echo "$msg" | grep -q "Outbound self-ref MISSING"; then
  921. add_issue "Security group $SG outbound self-ref MISSING → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
  922. elif echo "$msg" | grep -q "Outbound 0.0.0.0/0 missing"; then
  923. add_issue "Security group $SG outbound 0.0.0.0/0 MISSING → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
  924. else
  925. add_issue "Security group $SG rule missing → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
  926. fi
  927. ;;
  928. WARN) warn "$msg" ;;
  929. SKIP) info "$msg" ;;
  930. esac
  931. done <<< "$_SG_CHECK"
  932. done
  933. fi
  934. header "4b. Instance Quotas"
  935. INSTANCE_TYPES=$(echo "$CLUSTER_JSON" | python3 -c "
  936. import sys,json
  937. d=json.load(sys.stdin)
  938. types=set(g.get('InstanceType','') for g in d.get('InstanceGroups',[]))
  939. print(' '.join(t for t in types if t))
  940. " 2>/dev/null || echo "")
  941. if [[ -n "$INSTANCE_TYPES" ]]; then
  942. # One paginated list-service-quotas call, cached across all instance types.
  943. # The API is account/region rate-limited and throttles if called per-type.
  944. QUOTA_ALL=""
  945. QUOTA_ERR=""
  946. _next=""
  947. for _pg in 1 2 3 4 5; do
  948. if [[ -n "$_next" ]]; then
  949. _raw=$(aws service-quotas list-service-quotas \
  950. --service-code sagemaker --region "$REGION" \
  951. --cli-read-timeout 15 --starting-token "$_next" \
  952. --output json 2>&1 || true)
  953. else
  954. _raw=$(aws service-quotas list-service-quotas \
  955. --service-code sagemaker --region "$REGION" \
  956. --cli-read-timeout 15 \
  957. --output json 2>&1 || true)
  958. fi
  959. # Order matters: test for specific errors first, then fall through to
  960. # generic "not JSON" check, so throttled responses don't get misclassified.
  961. if echo "$_raw" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
  962. QUOTA_ERR="denied"; break
  963. elif echo "$_raw" | grep -qiE "TooManyRequestsException|ThrottlingException|RequestLimitExceeded|exceeded the rate"; then
  964. QUOTA_ERR="throttled"; break
  965. elif ! echo "$_raw" | head -c 1 | grep -q '{'; then
  966. QUOTA_ERR="api-error"; break
  967. fi
  968. _pg_quotas=$(echo "$_raw" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps(d.get('Quotas',[])))" 2>/dev/null || echo "[]")
  969. if [[ "$_pg_quotas" != "[]" ]]; then
  970. if [[ -z "$QUOTA_ALL" ]]; then
  971. QUOTA_ALL="$_pg_quotas"
  972. else
  973. QUOTA_ALL=$(python3 -c "import sys,json; a=json.loads(sys.argv[1]); b=json.loads(sys.argv[2]); print(json.dumps(a+b))" "$QUOTA_ALL" "$_pg_quotas")
  974. fi
  975. fi
  976. _next=$(echo "$_raw" | python3 -c "import sys,json; print(json.load(sys.stdin).get('NextToken','') or '')" 2>/dev/null || echo "")
  977. [[ -z "$_next" ]] && break
  978. done
  979. case "$QUOTA_ERR" in
  980. denied) warn "list-service-quotas" "IAM permission denied — quota check skipped" ;;
  981. throttled) warn "list-service-quotas" "Throttled — quota check skipped (retry later)" ;;
  982. api-error) warn "list-service-quotas" "API call failed — quota check skipped" ;;
  983. esac
  984. if [[ -n "$QUOTA_ALL" && -z "$QUOTA_ERR" ]]; then
  985. for ITYPE in $INSTANCE_TYPES; do
  986. QUOTA_VAL=$(python3 -c "
  987. import sys, json
  988. quotas = json.loads(sys.argv[1])
  989. itype = sys.argv[2]
  990. # Match quotas that reference the instance type AND HyperPod
  991. matches = [q for q in quotas if itype in q.get('QuotaName','') and 'HyperPod' in q.get('QuotaName','')]
  992. if matches:
  993. q = matches[0]
  994. print(f\"{q.get('QuotaName','?')}: {int(q.get('Value',0))}\")
  995. else:
  996. print('NOT_FOUND')
  997. " "$QUOTA_ALL" "$ITYPE" 2>/dev/null || echo "NOT_FOUND")
  998. if [[ "$QUOTA_VAL" == "NOT_FOUND" ]]; then
  999. info "Quota for $ITYPE: not found in the SageMaker quota list (check Service Quotas console)"
  1000. else
  1001. info "Quota: $QUOTA_VAL"
  1002. fi
  1003. done
  1004. fi
  1005. else
  1006. info "No instance types found in cluster config"
  1007. fi
  1008. if [[ "$ORCHESTRATOR" == "EKS" && -n "$EKS_NAME" ]]; then
  1009. header "5. EKS Configuration"
  1010. EKS_AUTH=$(aws eks describe-cluster \
  1011. --name "$EKS_NAME" \
  1012. --region "$REGION" \
  1013. --query 'cluster.accessConfig.authenticationMode' \
  1014. --output text 2>/dev/null || echo "unknown")
  1015. if [[ "$EKS_AUTH" == "CONFIG_MAP" ]]; then
  1016. warn "EKS auth mode" "CONFIG_MAP-only — access entries require API or API_AND_CONFIG_MAP"
  1017. add_issue "EKS auth mode is CONFIG_MAP — access entries unavailable until switched (see EKS access-entries docs) → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P2"
  1018. elif [[ "$EKS_AUTH" == "API" || "$EKS_AUTH" == "API_AND_CONFIG_MAP" ]]; then
  1019. pass "EKS auth mode" "$EKS_AUTH"
  1020. else
  1021. warn "EKS auth mode" "Could not determine ($EKS_AUTH)"
  1022. fi
  1023. # Check access entries for current identity. AWS CLI paginates JSON output by
  1024. # token, so paginate explicitly to handle accounts with many principals.
  1025. info "Current IAM identity: $CALLER_ARN"
  1026. fetch_all_access_entries() {
  1027. local merged='[]' token='' page_json combined i=0
  1028. while (( i < 20 )); do
  1029. if [[ -n "$token" ]]; then
  1030. page_json=$(aws eks list-access-entries --cluster-name "$EKS_NAME" --region "$REGION" \
  1031. --next-token "$token" --output json 2>/dev/null) || break
  1032. else
  1033. page_json=$(aws eks list-access-entries --cluster-name "$EKS_NAME" --region "$REGION" \
  1034. --output json 2>/dev/null) || break
  1035. fi
  1036. combined=$(python3 -c "
  1037. import sys, json
  1038. prev = json.loads(sys.argv[1])
  1039. page = json.loads(sys.argv[2])
  1040. prev.extend(page.get('accessEntries', []))
  1041. print(json.dumps(prev))
  1042. print(page.get('nextToken',''))
  1043. " "$merged" "$page_json" 2>/dev/null) || break
  1044. merged=$(printf '%s\n' "$combined" | sed -n '1p')
  1045. token=$(printf '%s\n' "$combined" | sed -n '2p')
  1046. i=$((i+1))
  1047. [[ -z "$token" ]] && break
  1048. done
  1049. echo "$merged"
  1050. }
  1051. ACCESS_ENTRIES=$(fetch_all_access_entries)
  1052. [[ -z "$ACCESS_ENTRIES" ]] && ACCESS_ENTRIES='[]'
  1053. ENTRY_COUNT=$(echo "$ACCESS_ENTRIES" | python3 -c "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0)
  1054. info "Access entries: $ENTRY_COUNT configured"
  1055. # Strip session name for role-based ARNs
  1056. CALLER_BASE=$(echo "$CALLER_ARN" | python3 -c "
  1057. import sys
  1058. arn = sys.stdin.read().strip()
  1059. # Convert assumed-role ARN to role ARN for matching
  1060. # arn:aws:sts::ACCOUNT:assumed-role/ROLE/SESSION -> arn:aws:iam::ACCOUNT:role/ROLE
  1061. if ':assumed-role/' in arn:
  1062. parts = arn.split(':')
  1063. role_path = parts[-1].replace('assumed-role/', 'role/')
  1064. role_path = '/'.join(role_path.split('/')[:2]) # remove session name
  1065. parts[-1] = role_path
  1066. parts[2] = 'iam'
  1067. parts[3] = '' # IAM ARNs have no region
  1068. print(':'.join(parts))
  1069. else:
  1070. print(arn)
  1071. " 2>/dev/null || echo "$CALLER_ARN")
  1072. HAS_ACCESS=$(echo "$ACCESS_ENTRIES" | CALLER_BASE_ENV="$CALLER_BASE" python3 -c "
  1073. import sys, json, os
  1074. entries = json.load(sys.stdin)
  1075. caller = os.environ['CALLER_BASE_ENV']
  1076. found = any(caller in str(e) for e in entries)
  1077. print('true' if found else 'false')
  1078. " 2>/dev/null || echo "false")
  1079. if [[ "$HAS_ACCESS" == "true" ]]; then
  1080. pass "EKS access entry" "current identity has an access entry"
  1081. else
  1082. warn "EKS access entry" "current identity ($CALLER_BASE) may not have an access entry — kubectl may fail"
  1083. add_issue "Current IAM identity may lack EKS access → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
  1084. fi
  1085. if command -v kubectl &>/dev/null; then
  1086. KUBECTL_TEST=$(kubectl cluster-info 2>&1 || true)
  1087. if echo "$KUBECTL_TEST" | grep -q "Kubernetes control plane\|running at"; then
  1088. pass "kubectl connectivity" "can reach EKS API server"
  1089. if kubectl get namespace aws-hyperpod &>/dev/null 2>&1; then
  1090. pass "aws-hyperpod namespace" "exists"
  1091. else
  1092. warn "aws-hyperpod namespace" "missing → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)"
  1093. fi
  1094. # Node count. Note: `wc -l` never fails; avoid `|| echo 0` which would produce "0\n0".
  1095. K8S_NODE_COUNT=$(kubectl get nodes --no-headers 2>/dev/null | wc -l | tr -d ' ')
  1096. K8S_NODE_COUNT=${K8S_NODE_COUNT:-0}
  1097. info "Kubernetes nodes visible: $K8S_NODE_COUNT"
  1098. if [[ "$K8S_NODE_COUNT" -eq 0 && "$TOTAL_NODES" -gt 0 ]]; then
  1099. warn "K8s nodes" "0 K8s nodes but $TOTAL_NODES HyperPod nodes — nodes may not have registered with EKS"
  1100. add_issue "Nodes not visible in kubectl → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P1"
  1101. fi
  1102. HEALTH_LABELS=$(kubectl get nodes -o custom-columns='NODE:.metadata.name,HEALTH:.metadata.labels.sagemaker\.amazonaws\.com/node-health-status' --no-headers 2>/dev/null || true)
  1103. if [[ -n "$HEALTH_LABELS" ]]; then
  1104. UNHEALTHY_K8S=$(echo "$HEALTH_LABELS" | grep -v "<none>" | grep -viE "Schedulable$" || true)
  1105. if [[ -n "$UNHEALTHY_K8S" ]]; then
  1106. warn "EKS node health labels" "non-schedulable nodes detected:"
  1107. echo "$UNHEALTHY_K8S" | while IFS= read -r line; do info " $line"; done
  1108. add_issue "EKS nodes with health issues → delegate to hyperpod-node-debugger skill; references/cluster-diagnostics-detail.md § G (Node Replacement)" "P1"
  1109. else
  1110. pass "EKS node health labels" "all nodes schedulable"
  1111. fi
  1112. fi
  1113. # Dangling node detection — nodes visible in EKS but not in HyperPod list
  1114. # (or vice versa). Happens after failed scale-up, rollback, or orphaned
  1115. # kubelet registrations.
  1116. if [[ "$K8S_NODE_COUNT" -gt 0 && "$TOTAL_NODES" -gt 0 ]]; then
  1117. HP_INSTANCES=$(echo "$NODE_LIST" | python3 -c "
  1118. import sys,json
  1119. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  1120. for n in nodes:
  1121. iid=n.get('InstanceId','')
  1122. if iid: print(iid)
  1123. " 2>/dev/null | sort -u)
  1124. EKS_INSTANCES=$(kubectl get nodes -l sagemaker.amazonaws.com/compute-type=hyperpod \
  1125. -o jsonpath='{range .items[*]}{.spec.providerID}{"\n"}{end}' 2>/dev/null \
  1126. | awk -F/ '{print $NF}' | grep -E '^i-' | sort -u || true)
  1127. if [[ -n "$HP_INSTANCES" && -n "$EKS_INSTANCES" ]]; then
  1128. DANGLING=$(comm -13 <(echo "$HP_INSTANCES") <(echo "$EKS_INSTANCES"))
  1129. ORPHANED=$(comm -23 <(echo "$HP_INSTANCES") <(echo "$EKS_INSTANCES"))
  1130. if [[ -n "$DANGLING" ]]; then
  1131. warn "Dangling nodes" "visible in EKS but not in HyperPod ($(echo "$DANGLING" | wc -l))"
  1132. echo "$DANGLING" | head -5 | while IFS= read -r iid; do info " EKS-only: $iid"; done
  1133. add_issue "Dangling EKS nodes (present in kubectl, absent from list-cluster-nodes) → references/cluster-diagnostics-detail.md § K (Dangling Nodes & Cleanup)" "P1"
  1134. fi
  1135. if [[ -n "$ORPHANED" ]]; then
  1136. warn "Orphaned HyperPod nodes" "visible in HyperPod but not in EKS ($(echo "$ORPHANED" | wc -l))"
  1137. echo "$ORPHANED" | head -5 | while IFS= read -r iid; do info " HyperPod-only: $iid"; done
  1138. add_issue "HyperPod nodes not registered in EKS → references/cluster-diagnostics-detail.md § E (Cluster Provisioning); delegate to hyperpod-node-debugger" "P1"
  1139. fi
  1140. [[ -z "$DANGLING" && -z "$ORPHANED" ]] && pass "Node reconciliation" "EKS and HyperPod views match"
  1141. fi
  1142. fi
  1143. # EKS add-on health — VPC CNI, CoreDNS, kube-proxy failures break pod networking.
  1144. # Add-on count is small in practice (<10) so a single page of 100 is always sufficient.
  1145. if [[ -n "$EKS_NAME" ]]; then
  1146. ADDON_JSON=$(aws eks list-addons --cluster-name "$EKS_NAME" --region "$REGION" \
  1147. --max-results 100 --output json 2>/dev/null || echo '{"addons":[]}')
  1148. ADDON_NAMES=$(echo "$ADDON_JSON" | python3 -c "
  1149. import sys,json
  1150. print('\n'.join(json.load(sys.stdin).get('addons',[])))
  1151. " 2>/dev/null)
  1152. DEGRADED_ADDONS=""
  1153. while IFS= read -r addon; do
  1154. [[ -z "$addon" ]] && continue
  1155. A_STATUS=$(aws eks describe-addon --cluster-name "$EKS_NAME" --addon-name "$addon" \
  1156. --region "$REGION" --query 'addon.status' --output text 2>/dev/null || echo "UNKNOWN")
  1157. if [[ "$A_STATUS" != "ACTIVE" && "$A_STATUS" != "UPDATING" ]]; then
  1158. DEGRADED_ADDONS+="$addon($A_STATUS) "
  1159. fi
  1160. done <<< "$ADDON_NAMES"
  1161. if [[ -n "$DEGRADED_ADDONS" ]]; then
  1162. warn "EKS add-ons" "not ACTIVE: $DEGRADED_ADDONS"
  1163. add_issue "EKS add-on(s) degraded: $DEGRADED_ADDONS → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
  1164. else
  1165. [[ -n "$ADDON_NAMES" ]] && pass "EKS add-ons" "$(echo "$ADDON_NAMES" | wc -l) add-on(s) ACTIVE"
  1166. fi
  1167. fi
  1168. # aws-auth ConfigMap legacy check — deprecated but still load-bearing if cluster auth mode
  1169. # is API_AND_CONFIG_MAP or CONFIG_MAP. Misconfigured entries here can shadow access entries.
  1170. if [[ -n "$EKS_NAME" ]]; then
  1171. AUTH_MODE=$(aws eks describe-cluster --name "$EKS_NAME" --region "$REGION" \
  1172. --query 'cluster.accessConfig.authenticationMode' --output text 2>/dev/null || echo "")
  1173. if [[ "$AUTH_MODE" == "CONFIG_MAP" || "$AUTH_MODE" == "API_AND_CONFIG_MAP" ]]; then
  1174. if kubectl -n kube-system get configmap aws-auth >/dev/null 2>&1; then
  1175. AUTH_ENTRIES=$(kubectl -n kube-system get configmap aws-auth -o jsonpath='{.data.mapRoles}' 2>/dev/null | grep -c "^" || true)
  1176. AUTH_ENTRIES=${AUTH_ENTRIES:-0}
  1177. info "aws-auth ConfigMap: $AUTH_ENTRIES mapRoles entries (auth mode: $AUTH_MODE)"
  1178. if [[ "$AUTH_MODE" == "API_AND_CONFIG_MAP" ]]; then
  1179. warn "aws-auth ConfigMap" "both ConfigMap and access entries in use — ConfigMap entries can shadow access entries; recommend migrating to API-only mode"
  1180. fi
  1181. fi
  1182. fi
  1183. fi
  1184. else
  1185. warn "kubectl connectivity" "cannot reach EKS API — check kubeconfig and access entries"
  1186. add_issue "kubectl cannot reach EKS → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
  1187. fi
  1188. else
  1189. info "kubectl not installed — skipping Kubernetes checks"
  1190. fi
  1191. else
  1192. header "5. Slurm Checks"
  1193. info "Orchestrator: Slurm"
  1194. # Warn/issue emitted in section 1; this branch is the PASS-only confirmation.
  1195. if [[ "$NODE_RECOVERY" == *"Automatic"* ]] && [[ "$NODE_RECOVERY" != *"None"* ]]; then
  1196. pass "NodeRecovery" "enabled on all instance groups"
  1197. fi
  1198. if command -v session-manager-plugin &>/dev/null && [[ -n "$CLUSTER_ID" ]]; then
  1199. header "5b. Slurm Controller Health (via SSM)"
  1200. HEAD_NODE_ID=$(echo "$NODE_LIST" | python3 -c "
  1201. import sys,json
  1202. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  1203. for n in nodes:
  1204. g=n.get('InstanceGroupName','').lower()
  1205. if any(x in g for x in ['controller','head','master','login']):
  1206. print(n.get('InstanceId',''))
  1207. break
  1208. else:
  1209. if nodes:
  1210. print(nodes[0].get('InstanceId',''))
  1211. " 2>/dev/null || echo "")
  1212. if [[ -n "$HEAD_NODE_ID" ]]; then
  1213. HEAD_GROUP=$(echo "$NODE_LIST" | HEAD_NODE_ID_ENV="$HEAD_NODE_ID" python3 -c "
  1214. import sys,json,os
  1215. target_id = os.environ['HEAD_NODE_ID_ENV']
  1216. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  1217. for n in nodes:
  1218. if n.get('InstanceId','') == target_id:
  1219. print(n.get('InstanceGroupName',''))
  1220. break
  1221. " 2>/dev/null || echo "")
  1222. if [[ -z "$HEAD_GROUP" ]]; then
  1223. warn "Controller node" "could not resolve instance-group name — SSM check skipped"
  1224. HEAD_NODE_ID=""
  1225. fi
  1226. fi
  1227. if [[ -n "$HEAD_NODE_ID" ]]; then
  1228. SSM_TARGET="sagemaker-cluster:${CLUSTER_ID}_${HEAD_GROUP}-${HEAD_NODE_ID}"
  1229. info "Controller node: $HEAD_NODE_ID ($HEAD_GROUP)"
  1230. info "SSM target: $SSM_TARGET"
  1231. _slurm_nonce=$(date +%s%N 2>/dev/null || echo "$RANDOM")
  1232. # Validate nonce is numeric to prevent injection in remote command
  1233. if [[ ! "$_slurm_nonce" =~ ^[0-9]+$ ]]; then
  1234. _slurm_nonce="$$"
  1235. fi
  1236. SLURM_SH=$(cat <<EOF
  1237. scontrol show config >/dev/null 2>&1
  1238. if [ \$? -eq 0 ]; then echo SLURM_OK_${_slurm_nonce}; else echo SLURM_DOWN_${_slurm_nonce}; fi
  1239. echo NODES_START_${_slurm_nonce}
  1240. sinfo -o '%N %T %30E' --noheader 2>/dev/null | head -20
  1241. echo NODES_END_${_slurm_nonce}
  1242. echo JOBS_START_${_slurm_nonce}
  1243. squeue -o '%i %j %T %R' --noheader 2>/dev/null | grep -iE 'COMPLETING|CONFIGURING|PENDING' | head -10 || true
  1244. echo JOBS_END_${_slurm_nonce}
  1245. echo MUNGE_${_slurm_nonce}
  1246. systemctl is-active munge 2>/dev/null || echo munge_inactive
  1247. echo END_${_slurm_nonce}
  1248. EOF
  1249. )
  1250. STDOUT=$(ssm_run_on_node "$HEAD_NODE_ID" "$HEAD_GROUP" "$SLURM_SH" || echo "")
  1251. if [[ -n "$STDOUT" ]]; then
  1252. if echo "$STDOUT" | grep -q "SLURM_OK_${_slurm_nonce}"; then
  1253. pass "slurmctld" "responsive"
  1254. elif echo "$STDOUT" | grep -q "SLURM_DOWN_${_slurm_nonce}"; then
  1255. fail "slurmctld" "not responding — all Slurm operations blocked"
  1256. add_issue "slurmctld down on controller → references/cluster-operations.md § 8 Slurm — controller operations" "P0"
  1257. fi
  1258. SLURM_DOWN_NODES=$(echo "$STDOUT" | sed -n "/^NODES_START_${_slurm_nonce}\$/,/^NODES_END_${_slurm_nonce}\$/p" | grep -v "^NODES_" | grep -iE "down|drain|fail" || true)
  1259. if [[ -n "$SLURM_DOWN_NODES" ]]; then
  1260. warn "Slurm nodes with issues:"
  1261. echo "$SLURM_DOWN_NODES" | while IFS= read -r line; do info " $line"; done
  1262. S_DOWN_COUNT=$(echo "$SLURM_DOWN_NODES" | grep -c . ; :)
  1263. S_DOWN_COUNT=${S_DOWN_COUNT:-0}
  1264. add_issue "$S_DOWN_COUNT Slurm node(s) down/drained → references/cluster-diagnostics-detail.md § G (Node Replacement); delegate to hyperpod-node-debugger" "P1"
  1265. else
  1266. pass "Slurm nodes" "all idle/alloc/mixed"
  1267. fi
  1268. STUCK_JOBS=$(echo "$STDOUT" | sed -n "/^JOBS_START_${_slurm_nonce}\$/,/^JOBS_END_${_slurm_nonce}\$/p" | grep -v "^JOBS_" || true)
  1269. if [[ -n "$STUCK_JOBS" ]]; then
  1270. warn "Stuck Slurm jobs detected:"
  1271. echo "$STUCK_JOBS" | while IFS= read -r line; do info " $line"; done
  1272. add_issue "Stuck Slurm jobs → references/cluster-operations.md § 8 Slurm — controller operations" "P1"
  1273. fi
  1274. if echo "$STDOUT" | sed -n "/^MUNGE_${_slurm_nonce}\$/,/^END_${_slurm_nonce}\$/p" | grep -q "munge_inactive"; then
  1275. fail "munge" "authentication service not running — Slurm auth will fail"
  1276. add_issue "munge service inactive on controller → references/cluster-operations.md § 8 Slurm — controller operations" "P0"
  1277. fi
  1278. else
  1279. info "Could not get output from SSM on controller — check ssm:StartSession permission, session-manager-plugin, or node reachability"
  1280. fi
  1281. else
  1282. info "Could not identify controller node from node list"
  1283. fi
  1284. else
  1285. info "SSM plugin not available — Slurm checks require SSM access to controller"
  1286. info "Install SSM plugin to enable Slurm health checks"
  1287. fi
  1288. fi
  1289. header "6. SSM Readiness"
  1290. if command -v session-manager-plugin &>/dev/null; then
  1291. if SSM_VERSION=$(session-manager-plugin --version 2>/dev/null); then
  1292. pass "SSM plugin installed" "version: $SSM_VERSION"
  1293. else
  1294. warn "SSM plugin" "installed but --version failed — plugin may be corrupt"
  1295. add_issue "SSM plugin installed but broken → references/cluster-diagnostics-detail.md § F (SSM Connectivity)" "P1"
  1296. fi
  1297. else
  1298. warn "SSM plugin" "not installed — required for node access (install session-manager-plugin)"
  1299. add_issue "SSM plugin not installed → references/cluster-diagnostics-detail.md § F (SSM Connectivity)" "P2"
  1300. fi
  1301. if [[ -n "$CLUSTER_ID" && "$TOTAL_NODES" -gt 0 ]]; then
  1302. FIRST_NODE=$(echo "$NODE_LIST" | python3 -c "
  1303. import sys, json
  1304. nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
  1305. if nodes:
  1306. n = nodes[0]
  1307. nid = n.get('InstanceId', '?')
  1308. group = n.get('InstanceGroupName', '?')
  1309. print(f'{group}-{nid}')
  1310. " 2>/dev/null || echo "")
  1311. if [[ -n "$FIRST_NODE" ]]; then
  1312. info "SSM target format: sagemaker-cluster:${CLUSTER_ID}_${FIRST_NODE}"
  1313. info "To connect: aws ssm start-session --target sagemaker-cluster:${CLUSTER_ID}_${FIRST_NODE} --region $REGION"
  1314. fi
  1315. fi
  1316. if [[ -n "$SUBNET_IDS" ]]; then
  1317. header "6b. VPC Endpoints"
  1318. FIRST_SUBNET=$(echo "$SUBNET_IDS" | awk '{print $1}')
  1319. VPC_FOR_ENDPOINTS=$(aws ec2 describe-subnets \
  1320. --subnet-ids "$FIRST_SUBNET" \
  1321. --region "$REGION" \
  1322. --cli-read-timeout 15 \
  1323. --query 'Subnets[0].VpcId' \
  1324. --output text 2>/dev/null || echo "")
  1325. if [[ -n "$VPC_FOR_ENDPOINTS" && "$VPC_FOR_ENDPOINTS" != "None" ]]; then
  1326. EP_RESULT=$(aws ec2 describe-vpc-endpoints \
  1327. --filters "Name=vpc-id,Values=$VPC_FOR_ENDPOINTS" \
  1328. --region "$REGION" \
  1329. --cli-read-timeout 15 \
  1330. --query "VpcEndpoints[?State==\`available\`].ServiceName" \
  1331. --output text 2>&1)
  1332. if echo "$EP_RESULT" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
  1333. warn "describe-vpc-endpoints" "IAM permission denied — VPC endpoint check skipped"
  1334. EP_RESULT=""
  1335. fi
  1336. ENDPOINTS="${EP_RESULT}"
  1337. # s3 → Lifecycle scripts (S3 bucket download path)
  1338. # ssm/ssmmessages/ec2messages → SSM connectivity (§ F)
  1339. for SVC in s3 ssm ssmmessages ec2messages; do
  1340. if echo "$ENDPOINTS" | grep -qE "(^|[.])${SVC}($|[[:space:]])"; then
  1341. pass "VPC endpoint: $SVC"
  1342. else
  1343. warn "VPC endpoint: $SVC" "not found — required only if the cluster subnet has no NAT/IGW path out"
  1344. case "$SVC" in
  1345. s3) add_issue "VPC endpoint not found for s3 → references/cluster-diagnostics-detail.md § C (Lifecycle Scripts)" "P2" ;;
  1346. ssm|ssmmessages|ec2messages)
  1347. add_issue "VPC endpoint not found for $SVC → references/cluster-diagnostics-detail.md § F (SSM Connectivity)" "P2" ;;
  1348. esac
  1349. fi
  1350. done
  1351. else
  1352. info "Could not determine VPC ID for endpoint check"
  1353. fi
  1354. fi
  1355. header "7. CloudWatch Logs"
  1356. if [[ -n "$CLUSTER_ID" ]]; then
  1357. # CW log groups follow /aws/sagemaker/Clusters/<CLUSTER_NAME>/<CLUSTER_ID>,
  1358. # where <CLUSTER_NAME> is the human-readable name (not the ARN short-id).
  1359. CLUSTER_NAME_FOR_LOGS=$(echo "$CLUSTER_JSON" | python3 -c "
  1360. import sys, json
  1361. try:
  1362. d = json.load(sys.stdin)
  1363. n = d.get('ClusterName', '')
  1364. print(n if n else '')
  1365. except Exception:
  1366. print('')
  1367. " 2>/dev/null)
  1368. # Fall back to the value the caller supplied, unless it looks like an ARN.
  1369. if [[ -z "$CLUSTER_NAME_FOR_LOGS" ]]; then
  1370. if [[ "$CLUSTER" == arn:aws:* ]]; then
  1371. CLUSTER_NAME_FOR_LOGS="$CLUSTER_ID" # best-effort; will probe the prefix below
  1372. else
  1373. CLUSTER_NAME_FOR_LOGS="$CLUSTER"
  1374. fi
  1375. fi
  1376. LOG_GROUP="/aws/sagemaker/Clusters/${CLUSTER_NAME_FOR_LOGS}/${CLUSTER_ID}"
  1377. LOG_RESULT=$(aws logs describe-log-groups \
  1378. --log-group-name-prefix "$LOG_GROUP" \
  1379. --region "$REGION" \
  1380. --query 'logGroups[0].logGroupName' \
  1381. --output text 2>&1)
  1382. if echo "$LOG_RESULT" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
  1383. warn "describe-log-groups" "IAM permission denied — CloudWatch log check skipped"
  1384. LOG_RESULT="None"
  1385. fi
  1386. LOG_EXISTS="${LOG_RESULT:-None}"
  1387. if [[ "$LOG_EXISTS" != "None" && -n "$LOG_EXISTS" ]]; then
  1388. pass "CloudWatch log group" "$LOG_GROUP"
  1389. # Use the server-side prefix filter; clusters with hundreds of nodes have
  1390. # hundreds of streams and the default first-page result truncates.
  1391. count_log_streams_by_prefix() {
  1392. local prefix="$1"
  1393. local merged='[]' token='' page_json combined i=0
  1394. while (( i < 20 )); do
  1395. if [[ -n "$token" ]]; then
  1396. page_json=$(aws logs describe-log-streams \
  1397. --log-group-name "$LOG_GROUP" --region "$REGION" \
  1398. --log-stream-name-prefix "$prefix" --limit 50 --next-token "$token" \
  1399. --output json 2>/dev/null) || break
  1400. else
  1401. page_json=$(aws logs describe-log-streams \
  1402. --log-group-name "$LOG_GROUP" --region "$REGION" \
  1403. --log-stream-name-prefix "$prefix" --limit 50 \
  1404. --output json 2>/dev/null) || break
  1405. fi
  1406. combined=$(python3 -c "
  1407. import sys, json
  1408. prev = json.loads(sys.argv[1])
  1409. page = json.loads(sys.argv[2])
  1410. prev.extend(s.get('logStreamName','') for s in page.get('logStreams', []))
  1411. print(json.dumps(prev))
  1412. print(page.get('nextToken',''))
  1413. " "$merged" "$page_json" 2>/dev/null) || break
  1414. merged=$(printf '%s\n' "$combined" | sed -n '1p')
  1415. token=$(printf '%s\n' "$combined" | sed -n '2p')
  1416. i=$((i+1))
  1417. [[ -z "$token" ]] && break
  1418. done
  1419. echo "$merged" | python3 -c "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0
  1420. }
  1421. LC_COUNT=$(count_log_streams_by_prefix "LifecycleConfig")
  1422. HM_COUNT=$(count_log_streams_by_prefix "SagemakerHealthMonitoringAgent")
  1423. info "Lifecycle log streams: $LC_COUNT"
  1424. info "Health monitoring log streams: $HM_COUNT"
  1425. if [[ "$LC_COUNT" -eq 0 && "$CLUSTER_STATUS" != "Creating" ]]; then
  1426. warn "Lifecycle logs" "no lifecycle log streams found — scripts may not have run"
  1427. fi
  1428. else
  1429. warn "CloudWatch log group" "not found: $LOG_GROUP"
  1430. info "Logs may not be available if cluster creation failed early"
  1431. info "Check IAM execution role has CloudWatch Logs write permissions"
  1432. add_issue "CloudWatch log group not found → references/cluster-diagnostics-detail.md § C (Lifecycle Scripts)" "P2"
  1433. fi
  1434. fi
  1435. echo ""
  1436. echo -e "${BOLD}========================================${NC}"
  1437. echo -e "${BOLD} DIAGNOSTIC SUMMARY ${NC}"
  1438. echo -e "${BOLD}========================================${NC}"
  1439. echo ""
  1440. echo -e " Cluster: ${BOLD}${CLUSTER}${NC} (${ORCHESTRATOR})"
  1441. echo -e " Status: ${CLUSTER_STATUS}"
  1442. echo -e " Results: ${RED}${CRITICAL_FAILURES} critical${NC} | ${YELLOW}${WARNINGS} warnings${NC}"
  1443. echo -e " Mode: READ-ONLY (no changes made; each [FAIL] points to a references section)"
  1444. echo ""
  1445. if [[ ${#ISSUES_FOUND[@]} -gt 0 ]]; then
  1446. echo -e "${BOLD} Issues Found (prioritized):${NC}"
  1447. for priority in P0 P1 P2; do
  1448. has_priority=false
  1449. for issue in "${ISSUES_FOUND[@]}"; do
  1450. if [[ "$issue" == "${priority}|"* ]]; then
  1451. if ! "$has_priority"; then
  1452. case "$priority" in
  1453. P0) echo -e " ${RED}${BOLD}[$priority — Fix Immediately]${NC}" ;;
  1454. P1) echo -e " ${YELLOW}${BOLD}[$priority — Fix Soon]${NC}" ;;
  1455. P2) echo -e " ${BOLD}[$priority — Informational]${NC}" ;;
  1456. esac
  1457. has_priority=true
  1458. fi
  1459. echo -e " → ${issue#*|}"
  1460. fi
  1461. done
  1462. done
  1463. echo ""
  1464. fi
  1465. if [[ $CRITICAL_FAILURES -eq 0 && $WARNINGS -eq 0 ]]; then
  1466. echo -e " ${GREEN}${BOLD}All cluster-level checks passed.${NC}"
  1467. echo " If issues persist, try:"
  1468. echo " - hyperpod-node-debugger skill for per-node issues"
  1469. echo " - hyperpod-nccl skill for NCCL/training issues"
  1470. elif [[ $CRITICAL_FAILURES -eq 0 ]]; then
  1471. echo -e " ${YELLOW}${BOLD}No critical issues, but $WARNINGS warning(s) found.${NC}"
  1472. echo " Review [WARN] items above."
  1473. else
  1474. echo -e " ${RED}${BOLD}$CRITICAL_FAILURES critical issue(s) found.${NC}"
  1475. echo " Fix [FAIL] items above. See SKILL.md for detailed resolution steps."
  1476. fi
  1477. echo ""
  1478. exit "$([[ $CRITICAL_FAILURES -eq 0 ]] && echo 0 || echo 1)"
plugins/sagemaker-ai/skills/hyperpod-nccl/scripts/nccl-diagnose.sh · Bundled script · 2,604 lines · 3 findings marked
  1. #!/usr/bin/env bash
  2. # nccl-diagnose.sh — read-only NCCL diagnostic for SageMaker HyperPod.
  3. # Supports both EKS and Slurm orchestrators (auto-detected).
  4. # Hardware checks run on cluster nodes via SSM, not locally.
  5. #
  6. # This script never modifies cluster state. It collects diagnostic signals and
  7. # attaches a reference pointer (→ references/<file>.md § <section>) to each
  8. # finding. The calling skill (hyperpod-nccl) reads this output alongside the
  9. # referenced sections to guide the user through the remediation.
  10. #
  11. # USAGE:
  12. # bash nccl-diagnose.sh [OPTIONS]
  13. #
  14. # OPTIONS:
  15. # --cluster <name> HyperPod cluster name (required)
  16. # --region <region> AWS region (required)
  17. # --orchestrator <eks|slurm> Force orchestrator (default: auto-detect)
  18. # --namespace <ns> [EKS] K8s namespace to scope (default: all)
  19. # --job <job-name> [EKS] Specific job to diagnose
  20. # --node <instance-id> Specific node instance ID for SSM checks
  21. # --sample-nodes <N> How many nodes to SSM into (default: 3)
  22. # --verbose Show extra debug output
  23. # --no-color Disable ANSI colors (also auto-off when not a TTY)
  24. # --help Show this help
  25. #
  26. # ARCHITECTURE:
  27. # LOCAL checks (run on this machine):
  28. # - AWS API calls: cluster status, SG rules, cluster events, node list
  29. # - kubectl calls: K8s node readiness, pod status, logs, NetworkPolicies
  30. # ON-NODE checks (run via SSM on actual cluster compute nodes):
  31. # - GPU health (nvidia-smi, XID errors, NVLink)
  32. # - EFA / libfabric availability
  33. # - NCCL library presence
  34. # - Network interfaces and MTU
  35. # - Memory / /dev/shm / memlock limits
  36. # - Active training processes
  37. # - dmesg hardware errors
  38. # SCALE strategy for 100s of nodes:
  39. # - AWS API checks cover ALL nodes cheaply via list-cluster-nodes
  40. # - K8s checks cover ALL nodes cheaply via kubectl
  41. # - SSM hardware checks sample --sample-nodes (default: 3) compute nodes
  42. # - CloudWatch log analysis covers ALL nodes at scale (no per-node SSM needed)
  43. #
  44. # EXAMPLES:
  45. # bash nccl-diagnose.sh --cluster my-cluster --region us-east-1
  46. # bash nccl-diagnose.sh --cluster my-cluster --region us-east-1 \
  47. # --namespace nccl-test --job my-job --sample-nodes 5
  48. # bash nccl-diagnose.sh --cluster my-cluster --region us-east-1 \
  49. # --node i-0123456789abcdef0
  50. #
  51. # EXIT CODES:
  52. # 0 No critical (P0/P1) issues; P2 informational findings are allowed.
  53. # 1 One or more critical issues, or a fatal prerequisite is missing.
  54. # 2 Invalid argument.
  55. set -euo pipefail
  56. _TEMP_FILES=()
  57. cleanup() {
  58. # Guard against empty-array + set -u on older bash (4.2 on AL2).
  59. [[ ${#_TEMP_FILES[@]} -gt 0 ]] && rm -f "${_TEMP_FILES[@]}" 2>/dev/null || true
  60. }
  61. trap cleanup EXIT
  62. # Auto-disable colors when stdout is not a TTY or TERM=dumb (agent-piped output).
  63. if [ -t 1 ] && [ "${TERM:-}" != "dumb" ]; then
  64. RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
  65. BLUE='\033[0;34m'; BOLD='\033[1m'; RESET='\033[0m'
  66. else
  67. RED=''; GREEN=''; YELLOW=''; BLUE=''; BOLD=''; RESET=''
  68. fi
  69. info() { echo -e "${BLUE}[INFO]${RESET} $*"; }
  70. success() { echo -e "${GREEN}[PASS]${RESET} $*"; }
  71. warn() { echo -e "${YELLOW}[WARN]${RESET} $*"; }
  72. error() { echo -e "${RED}[FAIL]${RESET} $*"; }
  73. header() { echo -e "\n${BOLD}${BLUE}═══════════════════════════════════════════════${RESET}"
  74. echo -e "${BOLD}${BLUE} $*${RESET}"
  75. echo -e "${BOLD}${BLUE}═══════════════════════════════════════════════${RESET}"; }
  76. section() { echo -e "\n${BOLD}-- $* --${RESET}"; }
  77. debug() { $VERBOSE && echo -e "[DEBUG] $*" >&2 || true; }
  78. CLUSTER_NAME=""
  79. REGION="${AWS_DEFAULT_REGION:-}"
  80. ORCHESTRATOR=""
  81. NAMESPACE=""
  82. JOB_NAME=""
  83. NODE_ID=""
  84. SAMPLE_NODES=3
  85. VERBOSE=false
  86. ISSUES_FOUND=0
  87. ISSUE_DETAILS=()
  88. add_issue_detail() {
  89. local priority="${2:-P1}"
  90. ISSUE_DETAILS+=("${priority}|$1")
  91. }
  92. K8S_CONNECTED=false
  93. SSM_CLUSTER_ID=""
  94. SSM_NODES=()
  95. usage() {
  96. # --help exits 0; invalid invocation exits 2 via usage 2.
  97. grep "^# USAGE:" -A 40 "$0" | grep "^#" | sed 's/^# \?//' | head -25
  98. exit "${1:-0}"
  99. }
  100. while [[ $# -gt 0 ]]; do
  101. case "$1" in
  102. --cluster) [[ $# -lt 2 ]] && { error "--cluster needs a value"; exit 2; }
  103. [[ ! "$2" =~ ^(arn:aws[a-z-]*:sagemaker:[a-z0-9-]+:[0-9]{12}:cluster/[a-z0-9]{12}|[a-zA-Z0-9]([-a-zA-Z0-9]{0,62}))$ ]] && { error "--cluster must be a valid HyperPod cluster name or ARN (got '$2')"; exit 2; }
  104. CLUSTER_NAME="$2"; shift 2 ;;
  105. --region) [[ $# -lt 2 ]] && { error "--region needs a value"; exit 2; }
  106. [[ ! "$2" =~ ^[a-z]{2}-[a-z]+-[0-9]+$ ]] && { error "--region must be a valid AWS region (got '$2')"; exit 2; }
  107. REGION="$2"; shift 2 ;;
  108. --orchestrator) [[ $# -lt 2 ]] && { error "--orchestrator needs a value"; exit 2; }
  109. [[ "$2" != "eks" && "$2" != "slurm" ]] && { error "--orchestrator must be 'eks' or 'slurm' (got '$2')"; exit 2; }
  110. ORCHESTRATOR="$2"; shift 2 ;;
  111. --namespace) [[ $# -lt 2 ]] && { error "--namespace needs a value"; exit 2; }
  112. [[ ! "$2" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] && { error "--namespace must be a valid K8s namespace (got '$2')"; exit 2; }
  113. NAMESPACE="$2"; shift 2 ;;
  114. --job) [[ $# -lt 2 ]] && { error "--job needs a value"; exit 2; }
  115. [[ ! "$2" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] && { error "--job must be a valid K8s name (got '$2')"; exit 2; }
  116. JOB_NAME="$2"; shift 2 ;;
  117. --node) [[ $# -lt 2 ]] && { error "--node needs a value"; exit 2; }
  118. [[ ! "$2" =~ ^i-[0-9a-f]{8,17}$ ]] && { error "--node must be an EC2 instance ID (got '$2')"; exit 2; }
  119. NODE_ID="$2"; shift 2 ;;
  120. --sample-nodes) [[ $# -lt 2 ]] && { error "--sample-nodes needs a value"; exit 2; }; SAMPLE_NODES="$2"; shift 2 ;;
  121. --verbose) VERBOSE=true; shift ;;
  122. --no-color) RED=''; GREEN=''; YELLOW=''; BLUE=''; BOLD=''; RESET=''; shift ;;
  123. --help|-h) usage 0 ;;
  124. *) echo "Unknown option: $1" >&2; usage 2 ;;
  125. esac
  126. done
  127. [[ -z "$CLUSTER_NAME" ]] && { error "Missing required: --cluster"; exit 1; }
  128. [[ -z "$REGION" ]] && { error "--region is required (or set AWS_DEFAULT_REGION before running)"; exit 2; }
  129. if ! [[ "$SAMPLE_NODES" =~ ^[0-9]+$ ]] || [[ "$SAMPLE_NODES" -lt 1 ]]; then
  130. error "--sample-nodes must be a positive integer (got: '$SAMPLE_NODES')"
  131. exit 1
  132. fi
  133. if [[ "$SAMPLE_NODES" -gt 50 ]]; then
  134. warn "--sample-nodes=$SAMPLE_NODES is very high (max recommended: 50). Capping at 50."
  135. SAMPLE_NODES=50
  136. fi
  137. # Paginate a sagemaker list-* call. Usage:
  138. # sagemaker_list_paginated list-cluster-nodes ClusterNodeSummaries [extra args...]
  139. # Returns {"<SummaryKey>": [...]} on stdout. Caps at 20 000 items; emits a
  140. # stderr warning if truncated. Returns an empty result on AccessDenied.
  141. sagemaker_list_paginated() {
  142. local api="$1" summary_key="$2"
  143. shift 2
  144. local merged='[]' token='' page_json combined i=0
  145. local max_pages=200
  146. while (( i < max_pages )); do
  147. local page_args=(--cluster-name "$CLUSTER_NAME" --region "$REGION" \
  148. --max-results 100 --cli-read-timeout 30 --output json "$@")
  149. # Validate token format before sending — avoid BadRequest on garbage.
  150. if [[ -n "$token" ]]; then
  151. if [[ "$token" =~ ^[a-zA-Z0-9/+]*={0,2}$ ]] && [[ -n "$token" ]]; then
  152. page_args+=(--next-token "$token")
  153. else
  154. break
  155. fi
  156. fi
  157. page_json=$(aws sagemaker "$api" "${page_args[@]}" 2>&1) || break
  158. if echo "$page_json" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized"; then
  159. break
  160. fi
  161. # Merge via stdin (NUL-delimited) to avoid ARG_MAX truncation at ~500
  162. # entries. summary_key stays in argv since it's small.
  163. combined=$(printf '%s\0%s' "$merged" "$page_json" | python3 -c "
  164. import sys, json
  165. blob = sys.stdin.buffer.read()
  166. try:
  167. a, b = blob.split(b'\0', 1)
  168. prev = json.loads(a)
  169. page = json.loads(b)
  170. except (json.JSONDecodeError, ValueError):
  171. sys.exit(2)
  172. prev.extend(page.get(sys.argv[1], []))
  173. print(json.dumps(prev))
  174. print(page.get('NextToken', ''))
  175. " "$summary_key" 2>/dev/null) || break
  176. merged=$(printf '%s\n' "$combined" | sed -n '1p')
  177. token=$(printf '%s\n' "$combined" | sed -n '2p')
  178. i=$((i+1))
  179. [[ -z "$token" ]] && break
  180. done
  181. if (( i == max_pages )) && [[ -n "$token" ]]; then
  182. echo "WARN: sagemaker_list_paginated($api): truncated at ${max_pages} pages (~$((max_pages*100)) items). Result may be incomplete for very large clusters." >&2
  183. fi
  184. printf '%s' "$merged" | python3 -c "
  185. import sys, json
  186. try:
  187. print(json.dumps({sys.argv[1]: json.loads(sys.stdin.read())}))
  188. except json.JSONDecodeError:
  189. print('{\"%s\":[]}' % sys.argv[1])
  190. " "$summary_key" 2>/dev/null || echo "{\"$summary_key\":[]}"
  191. }
  192. detect_orchestrator() {
  193. if [[ -n "$ORCHESTRATOR" ]]; then
  194. info "Orchestrator forced: $ORCHESTRATOR"; return
  195. fi
  196. header "Detecting Orchestrator Type"
  197. local orch_type
  198. orch_type=$(aws sagemaker describe-cluster \
  199. --cluster-name "$CLUSTER_NAME" --region "$REGION" \
  200. --query 'Orchestrator' --output text 2>/dev/null || echo "")
  201. if echo "$orch_type" | grep -qi "eks\|kubernetes"; then
  202. ORCHESTRATOR="eks"
  203. elif echo "$orch_type" | grep -qi "slurm"; then
  204. ORCHESTRATOR="slurm"
  205. elif kubectl cluster-info &>/dev/null 2>&1; then
  206. ORCHESTRATOR="eks"; info "Auto-detected: EKS (kubectl responds)"
  207. elif command -v sinfo &>/dev/null && sinfo &>/dev/null 2>&1; then
  208. ORCHESTRATOR="slurm"; info "Auto-detected: Slurm (sinfo responds)"
  209. elif command -v squeue &>/dev/null; then
  210. ORCHESTRATOR="slurm"; info "Auto-detected: Slurm (squeue found)"
  211. else
  212. ORCHESTRATOR="eks"
  213. warn "Could not auto-detect orchestrator — defaulting to 'eks'"
  214. warn "Override with: --orchestrator slurm"
  215. fi
  216. success "Orchestrator: $(printf '%s' "$ORCHESTRATOR" | tr '[:lower:]' '[:upper:]')"
  217. }
  218. check_prerequisites() {
  219. header "Checking Prerequisites"
  220. local missing=()
  221. local tool_path
  222. for tool in aws jq python3 unbuffer; do
  223. if tool_path=$(command -v "$tool" 2>/dev/null) && [[ -n "$tool_path" ]]; then
  224. success "$tool: $tool_path"
  225. else
  226. error "$tool NOT found — required"
  227. missing+=("$tool")
  228. fi
  229. done
  230. if [[ "$ORCHESTRATOR" == "eks" ]]; then
  231. if tool_path=$(command -v kubectl 2>/dev/null) && [[ -n "$tool_path" ]]; then
  232. success "kubectl: $tool_path"
  233. else
  234. error "kubectl NOT found — required for EKS"
  235. missing+=("kubectl")
  236. fi
  237. elif [[ "$ORCHESTRATOR" == "slurm" ]]; then
  238. local slurm_ok=false
  239. for t in sinfo squeue scontrol; do
  240. command -v "$t" &>/dev/null && { success "$t found (Slurm CLI OK)"; slurm_ok=true; break; }
  241. done
  242. $slurm_ok || warn "Slurm CLI not found locally — will use SSM for Slurm commands"
  243. fi
  244. if [[ ${#missing[@]} -gt 0 ]]; then
  245. error "Install: ${missing[*]}"
  246. # unbuffer ships in the `expect` package.
  247. if printf '%s\n' "${missing[@]}" | grep -qx unbuffer; then
  248. error " unbuffer: 'yum install expect' / 'apt install expect' / 'brew install expect'"
  249. fi
  250. exit 1
  251. fi
  252. if aws sts get-caller-identity --region "$REGION" &>/dev/null; then
  253. local id
  254. id=$(aws sts get-caller-identity --region "$REGION" --query 'Arn' --output text)
  255. success "AWS credentials: $id"
  256. else
  257. error "AWS credentials invalid or expired"; exit 1
  258. fi
  259. # Inspect both stdout (node list) and stderr (error message).
  260. # Empty stdout with non-zero exit is an auth / transport failure, not
  261. # a healthy cluster with zero nodes.
  262. if [[ "$ORCHESTRATOR" == "eks" ]]; then
  263. local kubectl_out kubectl_err tmpfile
  264. tmpfile=$(mktemp /tmp/kubectl-check-XXXXXX.err)
  265. _TEMP_FILES+=("$tmpfile")
  266. kubectl_out=$(kubectl get nodes --no-headers 2>"$tmpfile" || true)
  267. kubectl_err=$(cat "$tmpfile" 2>/dev/null || echo "")
  268. rm -f "$tmpfile"
  269. debug "kubectl stdout: '$kubectl_out'"
  270. debug "kubectl stderr: '$kubectl_err'"
  271. if echo "$kubectl_err" | grep -qiE \
  272. "Unauthorized|forbidden|You must be logged in|certificate|no configuration|Unable to connect|server.*refused"; then
  273. error "kubectl NOT authenticated to EKS cluster"
  274. error " $(echo "$kubectl_err" | head -1)"
  275. warn " K8s checks (2, 2b, 5, 5b, 6, 7, 9) will be SKIPPED"
  276. K8S_CONNECTED=false
  277. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  278. add_issue_detail "kubectl not authenticated to EKS cluster → references/operations.md § 3 SSM target format (HyperPod)" "P1"
  279. elif echo "$kubectl_err" | grep -qiE \
  280. "connection refused|no such host|dial tcp|context deadline exceeded|EOF"; then
  281. error "kubectl cannot reach EKS API server → references/operations.md § 1 Getting cluster names (kubeconfig setup)"
  282. error " $(echo "$kubectl_err" | head -1)"
  283. warn " K8s checks (2, 2b, 5, 5b, 6, 7, 9) will be SKIPPED — check VPN/network connectivity"
  284. K8S_CONNECTED=false
  285. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  286. add_issue_detail "kubectl cannot reach EKS API server → references/operations.md § 1 Getting cluster names (kubeconfig setup)" "P1"
  287. elif [[ -z "$kubectl_out" && -z "$kubectl_err" ]]; then
  288. warn "kubectl returned no output — kubeconfig may point to wrong cluster"
  289. warn " → references/operations.md § 1 Getting cluster names"
  290. K8S_CONNECTED=true # Allow K8s checks — cluster may simply have no nodes yet
  291. elif [[ -n "$kubectl_err" && -z "$kubectl_out" ]]; then
  292. error "kubectl error: $(echo "$kubectl_err" | head -1)"
  293. warn " K8s checks (2, 2b, 5, 5b, 6, 7, 9) will be SKIPPED"
  294. K8S_CONNECTED=false
  295. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  296. add_issue_detail "kubectl error — K8s checks skipped → references/operations.md § 1 Getting cluster names (kubeconfig setup)" "P1"
  297. else
  298. local node_count
  299. node_count=$(echo "$kubectl_out" | wc -l | tr -d ' ')
  300. success "kubectl authenticated — $node_count node(s) visible"
  301. K8S_CONNECTED=true
  302. fi
  303. fi
  304. }
  305. check_cluster_health() {
  306. header "Check 1: HyperPod Cluster Health"
  307. local cluster_json
  308. cluster_json=$(aws sagemaker describe-cluster \
  309. --cluster-name "$CLUSTER_NAME" --region "$REGION" \
  310. --output json 2>&1) || {
  311. if echo "$cluster_json" | grep -qiE "ResourceNotFound|Cluster with name .* not found|ValidationException"; then
  312. error "Cluster '$CLUSTER_NAME' not found in region '$REGION'"
  313. echo "$cluster_json" | head -3
  314. echo ""
  315. echo "Available clusters in $REGION:"
  316. aws sagemaker list-clusters --region "$REGION" \
  317. --query 'ClusterSummaries[*].{Name:ClusterName,Status:ClusterStatus}' \
  318. --output table 2>/dev/null || echo " (unable to list clusters — check IAM)"
  319. exit 1
  320. fi
  321. if echo "$cluster_json" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
  322. warn "Permission denied: sagemaker:DescribeCluster — check IAM policy"
  323. fi
  324. cluster_json="{}"
  325. }
  326. local cluster_state
  327. cluster_state=$(echo "$cluster_json" | python3 -c \
  328. "import sys,json; print(json.load(sys.stdin).get('ClusterStatus','UNKNOWN'))" 2>/dev/null \
  329. || echo "UNKNOWN")
  330. case "$cluster_state" in
  331. InService)
  332. success "Cluster status: $cluster_state" ;;
  333. UNKNOWN|None|"")
  334. warn "Cluster status: could not retrieve"
  335. warn " Ensure --cluster is the HyperPod cluster name and IAM has sagemaker:DescribeCluster" ;;
  336. Creating|Updating|RollingBack|SystemUpdating)
  337. warn "Cluster status: $cluster_state (operation in progress — NCCL checks may be partial)"
  338. add_issue_detail "Cluster in transient state $cluster_state — rerun after it completes → hyperpod-cluster-debugger skill if it stays stuck" "P2" ;;
  339. Deleting|DeleteFailed)
  340. error "Cluster status: $cluster_state (cluster is being torn down)"
  341. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  342. add_issue_detail "Cluster is ${cluster_state} → hyperpod-cluster-debugger skill" "P0" ;;
  343. Failed|ClusterMaintenanceRollbackFailed)
  344. error "Cluster status: $cluster_state (expected: InService)"
  345. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  346. add_issue_detail "Cluster status ${cluster_state} → hyperpod-cluster-debugger skill" "P0" ;;
  347. *)
  348. warn "Cluster status: $cluster_state (unrecognized state)"
  349. add_issue_detail "Unrecognized cluster state '${cluster_state}' → hyperpod-cluster-debugger skill" "P1" ;;
  350. esac
  351. # NodeRecovery — affects whether failed nodes are auto-replaced.
  352. # Prefer top-level NodeRecovery (the canonical location); InstanceGroups[*].NodeRecovery
  353. # is null when cluster-level setting is applied, so per-group-only reads always return 'Unknown'.
  354. local node_recovery
  355. node_recovery=$(echo "$cluster_json" | python3 -c "
  356. import sys,json
  357. d=json.load(sys.stdin)
  358. top=d.get('NodeRecovery')
  359. if top:
  360. print(top)
  361. else:
  362. igs = d.get('InstanceGroups',[])
  363. modes = sorted({ig.get('NodeRecovery') for ig in igs if ig.get('NodeRecovery')})
  364. print(','.join(modes) if modes else 'Unknown')
  365. " 2>/dev/null || echo "Unknown")
  366. if echo "$node_recovery" | grep -q "Automatic"; then
  367. success "NodeRecovery: $node_recovery (auto-repair enabled)"
  368. elif echo "$node_recovery" | grep -qi "^Unknown$"; then
  369. info "NodeRecovery: could not retrieve (needs sagemaker:DescribeCluster)"
  370. elif echo "$node_recovery" | grep -qi "^None$"; then
  371. warn "NodeRecovery: None — failed nodes won't auto-replace → references/operations.md § 6 HyperPod node health labels (EKS)"
  372. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  373. add_issue_detail "NodeRecovery disabled (set to 'None') — failed nodes won't auto-replace → references/operations.md § 6 HyperPod node health labels (EKS)" "P2"
  374. else
  375. warn "NodeRecovery: $node_recovery — failed nodes won't auto-replace → references/operations.md § 6 HyperPod node health labels (EKS)"
  376. fi
  377. # All instance groups — count nodes per group, surface any unhealthy count.
  378. # Paginated because clusters >50 nodes would otherwise be diagnosed on a partial sample.
  379. local node_summary
  380. node_summary=$(sagemaker_list_paginated list-cluster-nodes ClusterNodeSummaries)
  381. local node_output
  382. node_output=$(echo "$node_summary" | python3 -c "
  383. import sys,json
  384. nodes = json.load(sys.stdin).get('ClusterNodeSummaries',[])
  385. total = len(nodes)
  386. by_status = {}
  387. for n in nodes:
  388. s = n.get('InstanceStatus',{}).get('Status','Unknown')
  389. by_status[s] = by_status.get(s,0) + 1
  390. print(f' Total nodes: {total}')
  391. for s,c in sorted(by_status.items()):
  392. tag = '[PASS]' if s == 'Running' else '[FAIL]'
  393. print(f' {tag} {s}: {c}')
  394. failed = [n for n in nodes if n.get('InstanceStatus',{}).get('Status') not in ('Running','Pending')]
  395. for n in failed[:10]:
  396. msg = n.get('InstanceStatus',{}).get('Message','')
  397. print(f' -> {n[\"InstanceId\"]} ({n[\"InstanceGroupName\"]}): {msg[:120]}')
  398. print(f'FAILED_COUNT={len(failed)}')
  399. " 2>/dev/null || echo "FAILED_COUNT=0")
  400. local fc
  401. fc=$(echo "$node_output" | grep "^FAILED_COUNT=" | cut -d= -f2 || echo 0)
  402. # `|| true` on grep — no-match returns 1 and pipefail aborts the function.
  403. echo "$node_output" | { grep -v "^FAILED_COUNT=" || true; } | while IFS= read -r line; do
  404. if echo "$line" | grep -q "\[FAIL\]"; then
  405. error "$line"
  406. else
  407. echo "$line"
  408. fi
  409. done
  410. if [[ "${fc:-0}" -gt 0 ]]; then
  411. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  412. add_issue_detail "${fc} node(s) in failed/non-Running state → hyperpod-node-debugger skill" "P1"
  413. fi
  414. # Pre-flight: if the cluster has no GPU/EFA-capable instance groups, NCCL
  415. # diagnostics don't apply — exit clearly instead of emitting mixed INFO/SKIP.
  416. local gpu_groups
  417. gpu_groups=$(echo "$cluster_json" | python3 -c "
  418. import sys, json, re
  419. d = json.load(sys.stdin)
  420. igs = d.get('InstanceGroups', [])
  421. gpu_efa_re = re.compile(r'^ml\.(p4d|p4de|p5|p5e|p5en|p6|trn1|trn2|g5\.48xlarge|g6\.48xlarge|g6e\.48xlarge)', re.I)
  422. matches = [ig.get('InstanceGroupName','?') + ':' + ig.get('InstanceType','?')
  423. for ig in igs if gpu_efa_re.match(ig.get('InstanceType',''))]
  424. print('|'.join(matches))
  425. " 2>/dev/null || echo "")
  426. if [[ -z "$gpu_groups" ]] && [[ "$(echo "$cluster_json" | python3 -c 'import sys,json; print(len(json.load(sys.stdin).get("InstanceGroups",[])))' 2>/dev/null)" -gt 0 ]]; then
  427. warn "No GPU/EFA-capable instance groups in this cluster — NCCL is not applicable"
  428. warn " NCCL is only meaningful on multi-GPU instances with EFA (p4d/p4de/p5/p5e/p5en/p6/trn1/trn2/g5.48xlarge/g6.48xlarge/g6e.48xlarge)"
  429. warn " The rest of the diagnostic will still run, but most checks will return INFO/SKIP on CPU-only fleets"
  430. fi
  431. }
  432. check_cluster_events() {
  433. header "Check 3: Cluster Events (infrastructure signals)"
  434. # HyperPod cluster events report infrastructure-level state only:
  435. # lifecycle, bootstrap, EFA health-check, capacity, replacement, reboot,
  436. # software update. They do NOT carry NCCL / GPU / training-level signals —
  437. # those come from pod logs, CloudWatch, and on-node probes (checks 6–8).
  438. # ListClusterEvents response shape: array under `Events` with fields
  439. # EventId / ClusterArn / ClusterName / InstanceGroupName / ResourceType /
  440. # EventTime / Description (verified live; no Severity field).
  441. local events_json
  442. events_json=$(sagemaker_list_paginated list-cluster-events Events)
  443. local events
  444. events=$(echo "$events_json" | python3 -c "
  445. import sys, json
  446. summaries = json.load(sys.stdin).get('Events', [])
  447. proj = [{'Time': e.get('EventTime',''),
  448. 'Grp': e.get('InstanceGroupName','') or e.get('ResourceType',''),
  449. 'Msg': e.get('Description','') or ''} for e in summaries]
  450. print(json.dumps(proj))
  451. " 2>/dev/null || echo "[]")
  452. local infra_events
  453. infra_events=$(echo "$events" | python3 -c "
  454. import sys, json
  455. data = json.load(sys.stdin)
  456. # Match real HyperPod event messages that could block or degrade distributed training.
  457. keywords = [
  458. 'efa health checks', # 'EFA health checks did not run successfully'
  459. 'lifecycle script', # 'Lifecycle scripts did not run successfully' / 'execution timed out'
  460. 'bootstrap failed', # 'Instance bootstrap failed likely because of customer network misconfiguration'
  461. 'network misconfiguration', # appears in bootstrap-failed events
  462. 'insufficient capacity', # 'Insufficient capacity' / 'No subnets in the capacity AZ'
  463. 'failed to provision', # provisioning events
  464. 'hardware failure', # rare; surfaces via events when SMHP detects
  465. 'replacement', # node replacement activity
  466. 'reboot', # node reboot activity
  467. 'rollback', # AMI upgrade rollback
  468. ]
  469. found = [e for e in data if any(k in e.get('Msg','').lower() for k in keywords)]
  470. for e in found[:20]:
  471. print(f\"[{e.get('Grp','?')}] {str(e.get('Time','?'))[:19]} | {e.get('Msg','?')[:140]}\")
  472. print(f'COUNT={len(found)}')
  473. " 2>/dev/null || echo "COUNT=0")
  474. local count
  475. count=$(echo "$infra_events" | grep "^COUNT=" | cut -d= -f2 || echo 0)
  476. local lines
  477. lines=$(echo "$infra_events" | grep -v "^COUNT=" || true)
  478. if [[ -z "$lines" || "${count:-0}" -eq 0 ]]; then
  479. success "No infrastructure events that would block NCCL"
  480. if [[ "$ORCHESTRATOR" == "slurm" ]]; then
  481. info "(Cluster events may not be populated for HyperPod Slurm clusters — rely on pod-/job-log checks instead.)"
  482. fi
  483. else
  484. warn "Infrastructure events potentially affecting NCCL (last 100):"
  485. echo "$lines" | while IFS= read -r line; do
  486. if echo "$line" | grep -qiE "error|fail|timeout|rollback"; then
  487. error " $line"
  488. else
  489. warn " $line"
  490. fi
  491. done
  492. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  493. add_issue_detail "Infrastructure-level events found — review and cross-reference with cluster-debugger if root-cause is cluster-wide → references/debugging-guide.md (match event text to section)" "P1"
  494. fi
  495. }
  496. check_security_groups() {
  497. header "Check 4: Security Group Rules (EFA / NCCL Communication)"
  498. local cluster_json
  499. cluster_json=$(aws sagemaker describe-cluster \
  500. --cluster-name "$CLUSTER_NAME" --region "$REGION" \
  501. --output json 2>/dev/null || echo "{}")
  502. # DescribeCluster.VpcConfig returns SecurityGroupIds + Subnets (not SubnetIds).
  503. # VpcId is not on VpcConfig; derive from a subnet if needed downstream.
  504. local sgs subnets
  505. sgs=$(echo "$cluster_json" | python3 -c "
  506. import sys,json
  507. d=json.load(sys.stdin)
  508. print(','.join(d.get('VpcConfig',{}).get('SecurityGroupIds',[])))
  509. " 2>/dev/null || echo "")
  510. subnets=$(echo "$cluster_json" | python3 -c "
  511. import sys,json
  512. d=json.load(sys.stdin)
  513. print(','.join(d.get('VpcConfig',{}).get('Subnets',[])))
  514. " 2>/dev/null || echo "")
  515. info "SGs: ${sgs:-none} | Subnets: ${subnets:-none}"
  516. if [[ -z "$sgs" ]]; then
  517. warn "No security groups in cluster VPC config — cannot verify NCCL rules"
  518. warn " (DescribeCluster may need sagemaker:DescribeCluster permission)"
  519. return
  520. fi
  521. IFS=',' read -ra sg_list <<< "$sgs"
  522. for sg in "${sg_list[@]}"; do
  523. [[ -z "$sg" ]] && continue
  524. section "SG: $sg"
  525. local sg_json
  526. sg_json=$(aws ec2 describe-security-groups \
  527. --group-ids "$sg" --region "$REGION" \
  528. --query 'SecurityGroups[0]' --output json 2>&1) || {
  529. if echo "$sg_json" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
  530. warn "Permission denied: ec2:DescribeSecurityGroups — check IAM policy"
  531. fi
  532. sg_json="{}"
  533. }
  534. local self_in self_out all_out
  535. read -r self_in self_out all_out < <(echo "$sg_json" | python3 -c "
  536. import sys,json
  537. sg=json.load(sys.stdin)
  538. gid=sg.get('GroupId','')
  539. def has_self(rules):
  540. return any(any(p.get('GroupId')==gid for p in r.get('UserIdGroupPairs',[])) for r in rules)
  541. def has_all_out(rules):
  542. return any(r.get('IpProtocol')=='-1' and any(x.get('CidrIp')=='0.0.0.0/0' for x in r.get('IpRanges',[])) for r in rules)
  543. print('YES' if has_self(sg.get('IpPermissions',[])) else 'NO',
  544. 'YES' if has_self(sg.get('IpPermissionsEgress',[])) else 'NO',
  545. 'YES' if has_all_out(sg.get('IpPermissionsEgress',[])) else 'NO')
  546. " 2>/dev/null || echo "UNKNOWN UNKNOWN UNKNOWN")
  547. if [[ "$self_in" == "YES" ]]; then
  548. success " Inbound self-reference: PRESENT (inter-node communication OK)"
  549. else
  550. error " Inbound self-reference: MISSING — NCCL inter-node comm WILL FAIL"
  551. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  552. add_issue_detail "SG $sg missing inbound self-referencing rule → references/operations.md § 8 NCCL-specific remediations (Security group self-reference)" "P0"
  553. fi
  554. if [[ "$self_out" == "YES" ]]; then
  555. success " Outbound self-reference: PRESENT (EFA traffic OK)"
  556. else
  557. error " Outbound self-reference: MISSING — EFA traffic WILL FAIL"
  558. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  559. add_issue_detail "SG $sg missing outbound self-referencing rule → references/operations.md § 8 NCCL-specific remediations (Security group self-reference)" "P0"
  560. fi
  561. if [[ "$all_out" == "YES" ]]; then
  562. success " Outbound 0.0.0.0/0: PRESENT (API/internet OK)"
  563. else
  564. warn " Outbound 0.0.0.0/0: MISSING — may block SageMaker/S3 API calls"
  565. fi
  566. done
  567. }
  568. check_k8s_nodes() {
  569. header "Check 2: Kubernetes Node Readiness"
  570. local raw_nodes total not_ready
  571. raw_nodes=$(kubectl get nodes --no-headers 2>/dev/null || true)
  572. total=$(echo "$raw_nodes" | awk 'NF{c++} END{print c+0}')
  573. not_ready=$(echo "$raw_nodes" | { grep -vE " Ready" || true; } | awk 'NF{c++} END{print c+0}')
  574. info "Total K8s nodes: $total"
  575. if [[ "$not_ready" -eq 0 ]]; then
  576. success "All $total nodes are Ready"
  577. else
  578. error "$not_ready/$total nodes NOT Ready"
  579. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  580. add_issue_detail "$not_ready/$total K8s nodes not Ready → hyperpod-node-debugger skill" "P1"
  581. echo "$raw_nodes" | { grep -vE " Ready" || true; } | while read -r line; do
  582. error " Not Ready: $line"
  583. done
  584. fi
  585. section "HyperPod Health Labels (all nodes)"
  586. # Uses the 4 documented node-health-status values plus deep-health-check-status
  587. local health_output
  588. health_output=$(kubectl get nodes -o json 2>/dev/null | python3 -c "
  589. import sys, json
  590. data = json.load(sys.stdin)
  591. issues = 0
  592. for node in data.get('items', []):
  593. name = node['metadata']['name']
  594. labels = node['metadata'].get('labels', {})
  595. health = labels.get('sagemaker.amazonaws.com/node-health-status', '')
  596. deep = labels.get('sagemaker.amazonaws.com/deep-health-check-status', '')
  597. fault_type = labels.get('sagemaker.amazonaws.com/fault-types', '')
  598. fault_reason = labels.get('sagemaker.amazonaws.com/fault-reasons', '')
  599. ok = health in ('', 'Schedulable') and deep in ('', 'Passed') and not fault_type
  600. tag = '[PASS]' if ok else '[FAIL]'
  601. if not ok:
  602. issues += 1
  603. line = f' {tag} {name}: health={health or \"(none)\"}'
  604. if deep: line += f' deep={deep}'
  605. if fault_type: line += f' fault={fault_type}'
  606. print(line)
  607. if health == 'Unschedulable':
  608. print(' -> Running deep health checks (~2h), temporarily unavailable')
  609. elif health == 'UnschedulablePendingReplacement':
  610. print(' -> Failed health checks — needs replacement (NodeRecovery=Automatic will auto-replace)')
  611. elif health == 'UnschedulablePendingReboot':
  612. print(' -> Unhealthy — rebooting to re-run health checks')
  613. if deep == 'InProgress': print(' -> Deep health check in progress')
  614. elif deep == 'Failed': print(' -> Deep health check FAILED — node will be replaced')
  615. if fault_type: print(f' -> Fault: {fault_type} | {fault_reason}')
  616. print(f'ISSUES={issues}')
  617. " 2>/dev/null || echo "ISSUES=0")
  618. local health_issues
  619. health_issues=$(echo "$health_output" | grep "^ISSUES=" | cut -d= -f2 || echo 0)
  620. echo "$health_output" | { grep -v "^ISSUES=" || true; } | while IFS= read -r line; do
  621. echo -e "$line"
  622. done
  623. # Using `if` instead of `[[ ... ]] && ...` — the short-circuit form returns
  624. # non-zero when the test is false, which aborts the script under `set -e`
  625. # and silently skips every remaining check (pods, env vars, hardware).
  626. if [[ "${health_issues:-0}" -gt 0 ]]; then
  627. ISSUES_FOUND=$((ISSUES_FOUND + health_issues))
  628. fi
  629. }
  630. check_pod_status() {
  631. local ns_flag ns_label
  632. if [[ -n "$NAMESPACE" ]]; then
  633. ns_flag=(-n "$NAMESPACE")
  634. ns_label="'$NAMESPACE'"
  635. else
  636. ns_flag=(-A)
  637. ns_label="all namespaces"
  638. fi
  639. header "Check 5: Pod / Job Status ($ns_label)"
  640. local job_filter=()
  641. [[ -n "$JOB_NAME" ]] && job_filter=(-l "job-name=$JOB_NAME")
  642. # `${arr[@]+"${arr[@]}"}` — expand only if defined; plain `${arr[@]}`
  643. # trips `set -u` on empty arrays under bash 4.2 (AL2 default).
  644. local pods_json
  645. pods_json=$(kubectl get pods "${ns_flag[@]}" ${job_filter[@]+"${job_filter[@]}"} -o json 2>/dev/null \
  646. || echo '{"items":[]}')
  647. local pod_output
  648. pod_output=$(python3 -c "
  649. import sys, json
  650. d = json.load(sys.stdin)
  651. items = d['items']
  652. total = len(items)
  653. failed = [p for p in items if p.get('status',{}).get('phase') in ('Failed','Unknown')]
  654. pending = [p for p in items if p.get('status',{}).get('phase') == 'Pending']
  655. crashes = []
  656. for p in items:
  657. for cs in p.get('status',{}).get('containerStatuses',[]):
  658. if cs.get('restartCount',0)>2 or cs.get('state',{}).get('waiting',{}).get('reason') \
  659. in ('CrashLoopBackOff','OOMKilled','Error'):
  660. crashes.append(p)
  661. break
  662. print(f'TOTAL={total}')
  663. print(f'FAILED={len(failed)}')
  664. print(f'PENDING={len(pending)}')
  665. print(f'CRASH={len(crashes)}')
  666. for p in failed[:5]:
  667. name = p['metadata']['name']
  668. ns = p['metadata']['namespace']
  669. msg = p.get('status',{}).get('message','')[:150]
  670. print(f'FAILED_POD={ns}/{name}: {msg}')
  671. for p in pending[:5]:
  672. name = p['metadata']['name']
  673. ns = p['metadata']['namespace']
  674. for c in p.get('status',{}).get('conditions',[]):
  675. if c.get('status')=='False':
  676. print(f'PENDING_POD={ns}/{name}: {c.get(\"message\",\"\")[:120]}')
  677. for p in crashes[:5]:
  678. name = p['metadata']['name']
  679. ns = p['metadata']['namespace']
  680. for cs in p.get('status',{}).get('containerStatuses',[]):
  681. r = cs.get('state',{}).get('waiting',{}).get('reason','CrashLoop')
  682. print(f'CRASH_POD={ns}/{name}: {r} restarts={cs.get(\"restartCount\",0)}')
  683. break
  684. " <<< "$pods_json" 2>/dev/null || echo "TOTAL=0
  685. FAILED=0
  686. PENDING=0
  687. CRASH=0")
  688. # Parse counts outside of pipe to avoid subshell variable loss
  689. local p_total p_failed p_pending p_crash
  690. p_total=$(echo "$pod_output" | grep "^TOTAL=" | cut -d= -f2 || echo 0)
  691. p_failed=$(echo "$pod_output" | grep "^FAILED=" | cut -d= -f2 || echo 0)
  692. p_pending=$(echo "$pod_output" | grep "^PENDING=" | cut -d= -f2 || echo 0)
  693. p_crash=$(echo "$pod_output" | grep "^CRASH=" | cut -d= -f2 || echo 0)
  694. info " Total pods: ${p_total:-0}"
  695. if [[ "${p_failed:-0}" -gt 0 ]]; then
  696. error " Failed/Unknown pods: $p_failed"; ISSUES_FOUND=$((ISSUES_FOUND+1)); add_issue_detail "$p_failed Failed/Unknown pod(s) → references/debugging-guide.md § 20 Pending / CrashLoopBackOff / Init-Container Failures" "P1"
  697. else
  698. success " No failed pods"
  699. fi
  700. if [[ "${p_pending:-0}" -gt 0 ]]; then
  701. warn " Pending pods: $p_pending"; ISSUES_FOUND=$((ISSUES_FOUND+1)); add_issue_detail "$p_pending Pending pod(s) → references/debugging-guide.md § 20 Pending / CrashLoopBackOff / Init-Container Failures" "P1"
  702. else
  703. success " No pending pods"
  704. fi
  705. if [[ "${p_crash:-0}" -gt 0 ]]; then
  706. error " CrashLoop/OOM pods: $p_crash"; ISSUES_FOUND=$((ISSUES_FOUND+1)); add_issue_detail "$p_crash CrashLoopBackOff/OOM pod(s) → references/debugging-guide.md § 20 Pending / CrashLoopBackOff / Init-Container Failures" "P1"
  707. else
  708. success " No crashloop pods"
  709. fi
  710. # `|| true` — grep returns 1 on no-match; with `pipefail` that kills the
  711. # whole function, silently skipping the rest of the diagnostic.
  712. echo "$pod_output" | { grep "^FAILED_POD=" || true; } | while IFS= read -r line; do error " ${line#FAILED_POD=}"; done
  713. echo "$pod_output" | { grep "^PENDING_POD=" || true; } | while IFS= read -r line; do warn " ${line#PENDING_POD=}"; done
  714. echo "$pod_output" | { grep "^CRASH_POD=" || true; } | while IFS= read -r line; do error " ${line#CRASH_POD=}"; done
  715. }
  716. # Checks EKS-specific prerequisites that cause NCCL failures before training starts:
  717. # - Headless service for MASTER_ADDR DNS resolution
  718. # - Init container failures blocking training containers
  719. # - /dev/shm volume mount (K8s default 64MB is too small for NCCL)
  720. check_nccl_infra_prereqs() {
  721. header "Check 5b: NCCL Infrastructure Prerequisites"
  722. local ns_flag ns_label
  723. if [[ -n "$NAMESPACE" ]]; then
  724. ns_flag=(-n "$NAMESPACE")
  725. ns_label="'$NAMESPACE'"
  726. else
  727. ns_flag=(-A)
  728. ns_label="all namespaces"
  729. fi
  730. local job_filter=()
  731. [[ -n "$JOB_NAME" ]] && job_filter=(-l "job-name=$JOB_NAME")
  732. # MASTER_ADDR DNS resolution requires a headless service (ClusterIP: None)
  733. # Without it, pods get DNS like "10-0-1-5.default.pod.cluster.local" which
  734. # doesn't resolve from other pods → rendezvous timeout
  735. section "Headless Service (MASTER_ADDR DNS)"
  736. local headless_svcs
  737. headless_svcs=$(kubectl get svc "${ns_flag[@]}" -o json 2>/dev/null | python3 -c "
  738. import sys, json
  739. data = json.load(sys.stdin)
  740. found = []
  741. for svc in data.get('items', []):
  742. spec = svc.get('spec', {})
  743. if spec.get('clusterIP') == 'None':
  744. name = svc['metadata']['name']
  745. ns = svc['metadata']['namespace']
  746. sel = spec.get('selector', {})
  747. found.append(f'{ns}/{name} selector={sel}')
  748. print(f'COUNT={len(found)}')
  749. for f in found[:10]:
  750. print(f)
  751. " 2>/dev/null || echo "COUNT=0")
  752. local hl_count
  753. hl_count=$(echo "$headless_svcs" | grep "^COUNT=" | cut -d= -f2 || echo 0)
  754. if [[ "${hl_count:-0}" -gt 0 ]]; then
  755. success "Headless service(s) found (${hl_count}) — MASTER_ADDR DNS can resolve"
  756. echo "$headless_svcs" | { grep -v "^COUNT=" || true; } | while IFS= read -r line; do
  757. [[ -n "$line" ]] && info " $line"
  758. done
  759. else
  760. warn "No headless services found in $ns_label"
  761. warn " If MASTER_ADDR uses a hostname, DNS resolution will fail"
  762. warn " Example: spec.clusterIP: None, spec.selector: {app: my-training-job}"
  763. fi
  764. # Init containers must complete before training container starts.
  765. # Common failures: S3 data download, config fetch, health check wait
  766. section "Init Container Status"
  767. local init_issues
  768. init_issues=$(kubectl get pods "${ns_flag[@]}" ${job_filter[@]+"${job_filter[@]}"} -o json 2>/dev/null | python3 -c "
  769. import sys, json
  770. data = json.load(sys.stdin)
  771. issues = 0
  772. for pod in data.get('items', []):
  773. name = pod['metadata']['name']
  774. ns = pod['metadata']['namespace']
  775. for ics in pod.get('status', {}).get('initContainerStatuses', []):
  776. state = ics.get('state', {})
  777. if 'waiting' in state:
  778. reason = state['waiting'].get('reason', '')
  779. msg = state['waiting'].get('message', '')[:100]
  780. if reason in ('CrashLoopBackOff', 'Error', 'ImagePullBackOff', 'ErrImagePull'):
  781. print(f'FAIL:{ns}/{name}: init container \"{ics[\"name\"]}\" {reason}: {msg}')
  782. issues += 1
  783. elif 'terminated' in state and state['terminated'].get('exitCode', 0) != 0:
  784. reason = state['terminated'].get('reason', 'Error')
  785. print(f'FAIL:{ns}/{name}: init container \"{ics[\"name\"]}\" exited {state[\"terminated\"][\"exitCode\"]}: {reason}')
  786. issues += 1
  787. print(f'ISSUES={issues}')
  788. " 2>/dev/null || echo "ISSUES=0")
  789. local init_count
  790. init_count=$(echo "$init_issues" | grep "^ISSUES=" | cut -d= -f2 || echo 0)
  791. if [[ "${init_count:-0}" -gt 0 ]]; then
  792. error " $init_count init container failure(s) — training containers cannot start"
  793. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  794. add_issue_detail "$init_count failed init container(s) blocking training → references/debugging-guide.md § 20 Pending / CrashLoopBackOff / Init-Container Failures" "P1"
  795. echo "$init_issues" | { grep "^FAIL:" || true; } | while IFS= read -r line; do
  796. error " ${line#FAIL:}"
  797. done
  798. else
  799. success "No init container failures"
  800. fi
  801. # K8s default /dev/shm = 64MB. NCCL needs ≥1GB. Without emptyDir mount,
  802. # training gets "failed to extend /dev/shm/nccl-*" or SIGBUS.
  803. section "/dev/shm Volume Mount"
  804. if [[ -n "$JOB_NAME" ]]; then
  805. local ns="${NAMESPACE:-default}"
  806. local shm_check
  807. shm_check=$(kubectl get pods -n "$ns" -l "job-name=$JOB_NAME" -o json 2>/dev/null | python3 -c "
  808. import sys, json
  809. data = json.load(sys.stdin)
  810. pods = data.get('items', [])
  811. if not pods:
  812. print('NO_PODS')
  813. else:
  814. pod = pods[0]
  815. vols = pod.get('spec', {}).get('volumes', [])
  816. has_dshm = any(
  817. v.get('emptyDir', {}).get('medium') == 'Memory'
  818. for v in vols
  819. if any(vm.get('mountPath') == '/dev/shm'
  820. for c in pod.get('spec', {}).get('containers', [])
  821. for vm in c.get('volumeMounts', [])
  822. if vm.get('name') == v.get('name'))
  823. )
  824. if has_dshm:
  825. print('OK')
  826. else:
  827. print('MISSING')
  828. " 2>/dev/null || echo "UNKNOWN")
  829. case "$shm_check" in
  830. OK) success "/dev/shm mounted as emptyDir Memory — NCCL shared memory OK" ;;
  831. MISSING) warn "/dev/shm NOT mounted as emptyDir Memory (K8s default = 64MB)"
  832. warn " NCCL will fail with 'failed to extend /dev/shm/nccl-*' or Bus error"
  833. warn " volumes: [{name: dshm, emptyDir: {medium: Memory, sizeLimit: '10Gi'}}]"
  834. warn " volumeMounts: [{name: dshm, mountPath: /dev/shm}]"
  835. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  836. add_issue_detail "/dev/shm not mounted as emptyDir Memory → references/debugging-guide.md § 17 RDMA Memory Registration Failure" "P1" ;;
  837. NO_PODS) info "No pods found for job '$JOB_NAME' — /dev/shm check skipped" ;;
  838. *) info "/dev/shm mount status unknown" ;;
  839. esac
  840. else
  841. info "/dev/shm check requires --job flag (skipped)"
  842. fi
  843. }
  844. analyze_nccl_logs() {
  845. header "Check 6: NCCL Log Pattern Analysis"
  846. local job_filter=()
  847. [[ -n "$JOB_NAME" ]] && job_filter=(-l "job-name=$JOB_NAME")
  848. local pod_entries
  849. if [[ -n "$NAMESPACE" ]]; then
  850. pod_entries=$(kubectl get pods -n "$NAMESPACE" ${job_filter[@]+"${job_filter[@]}"} --no-headers 2>/dev/null \
  851. | awk -v ns="$NAMESPACE" '{print ns"/"$1}' || echo "")
  852. else
  853. pod_entries=$(kubectl get pods -A ${job_filter[@]+"${job_filter[@]}"} --no-headers 2>/dev/null \
  854. | awk '{print $1"/"$2}' \
  855. | grep -vE "^(kube-system|kube-public|kube-node-lease|aws-hyperpod)/" || true)
  856. fi
  857. if [[ -z "$pod_entries" ]]; then
  858. info "No workload pods found to analyze logs"
  859. return
  860. fi
  861. local NCCL_PAT_KEYS=(
  862. "Timeout waiting for"
  863. "Connection refused"
  864. "Address already in use"
  865. "NCCL WARN Connect to"
  866. "network is unreachable"
  867. "Error in Store"
  868. "DistStoreError"
  869. "RendezvousConnectionError"
  870. "RendezvousTimeout"
  871. "Name or service not known"
  872. "getaddrinfo failed"
  873. "Watchdog timeout"
  874. "unhandled system error"
  875. "unhandled cuda error"
  876. "peer access is not supported"
  877. "NCCL WARN Cuda failure"
  878. "fi_getinfo failed"
  879. "NCCL_OFI_RDMA"
  880. "Call to ibv_reg_mr failed"
  881. "NET/OFI Using TCP"
  882. "Failed to load NCCL"
  883. "libnccl-net.so"
  884. "OOMKilled"
  885. "CUDA out of memory"
  886. "cudaMalloc failed"
  887. "failed to extend /dev/shm"
  888. "Bus error"
  889. "NCCL function not found"
  890. "Incompatible NCCL version"
  891. "Could not find interface"
  892. "world_size mismatch"
  893. "doesn't have NCCL built in"
  894. "CUDA_VISIBLE_DEVICES"
  895. "unlink shared memory"
  896. "Call to ncclCommAbort"
  897. "MNNVL topology"
  898. "ENOMEM"
  899. "invalid alignment"
  900. )
  901. local NCCL_PAT_VALS=(
  902. "TIMEOUT_RENDEZVOUS:Rendezvous timed out — peer ranks not responding"
  903. "CONN_REFUSED:TCP refused — check MASTER_ADDR/MASTER_PORT"
  904. "PORT_CONFLICT:Port already in use — change MASTER_PORT"
  905. "CONNECT_FAIL:NCCL peer connection failed — check SG/NetworkPolicy"
  906. "NET_UNREACHABLE:Network unreachable — VPC/routing issue"
  907. "STORE_ERR:Distributed store error — usually rendezvous timeout"
  908. "STORE_ERR:Distributed store error (PyTorch 2.x) — usually rendezvous timeout"
  909. "RDZV_CONN_ERR:Torch elastic rendezvous connection failed — check MASTER_ADDR DNS + SG"
  910. "RDZV_TIMEOUT:Torch elastic rendezvous timed out — peers not reachable"
  911. "DNS_FAIL:DNS resolution failed for MASTER_ADDR — check headless service or /etc/hosts"
  912. "DNS_FAIL:DNS resolution failed — headless service missing or CoreDNS issue"
  913. "WATCHDOG_TIMEOUT:AllReduce watchdog expired — straggler or OOM"
  914. "SYSTEM_ERROR:NCCL system error — GPU/EFA hardware issue"
  915. "CUDA_ERROR:CUDA runtime error — GPU driver crash or hardware fault"
  916. "P2P_FAIL:GPU peer access blocked — ACS enabled or IOMMU misconfigured"
  917. "CUDA_ERROR:CUDA failure inside NCCL — GPU hardware or driver issue"
  918. "EFA_INIT_FAIL:EFA libfabric init failed — EFA not available or wrong NCCL_SOCKET_IFNAME"
  919. "OFI_ERROR:aws-ofi-nccl plugin error — check EFA driver and OFI NCCL version"
  920. "RDMA_REG_FAIL:EFA/RDMA memory registration failed — memlock limit too low"
  921. "EFA_TCP_FALLBACK:NCCL fell back to TCP instead of EFA — 10-100x slower than expected"
  922. "NCCL_LOAD_FAIL:Failed to load NCCL library — libnccl.so missing or LD_LIBRARY_PATH wrong"
  923. "OFI_LOAD_FAIL:Failed to load aws-ofi-nccl plugin — libnccl-net.so not found"
  924. "OOM_KILL:Container killed (OOM) — reduce batch size or increase memory limit"
  925. "CUDA_OOM:GPU out of memory — reduce batch size or model size"
  926. "CUDA_OOM:GPU cudaMalloc failed — reduce batch size or model size"
  927. "SHM_FULL:NCCL shared memory /dev/shm full — mount emptyDir with 10Gi sizeLimit"
  928. "SHM_FULL:/dev/shm too small or SIGBUS — mount emptyDir with 10Gi sizeLimit"
  929. "NCCL_VERSION_MISMATCH:NCCL version mismatch across nodes — mixed container images"
  930. "NCCL_VERSION_MISMATCH:NCCL version mismatch across nodes — mixed container images"
  931. "IFACE_NOT_FOUND:NCCL_SOCKET_IFNAME points to missing interface"
  932. "WORLD_SIZE_MISMATCH:WORLD_SIZE doesn't match running process count"
  933. "NCCL_NOT_BUILT:PyTorch compiled without NCCL — rebuild with USE_NCCL=1 or use AWS DLC image"
  934. "CUDA_VIS_DEV:CUDA_VISIBLE_DEVICES misconfigured — GPUs not visible to training process"
  935. "SHM_STALE:Stale NCCL shared memory from previous run — systemd RemoveIPC=yes or manual cleanup"
  936. "NCCL_COMM_ABORT:NCCL communicator aborted — check for straggler node or hardware fault"
  937. "MNNVL_TOPO_FAIL:NCCL MNNVL topology search failed — memlock=unlimited + stack=unlimited causes 2MB thread stack; fix: ulimit -l 8388608 -s 8192"
  938. "ENOMEM:Memory registration/allocation failed — check memlock limits and available GPU memory"
  939. "CUDA_ALIGN_ERR:CUDA memory alignment error — possible driver/NCCL version incompatibility"
  940. )
  941. local issues_in_logs=false
  942. while IFS= read -r entry; do
  943. local ns pod
  944. ns="${entry%%/*}"; pod="${entry#*/}"
  945. section "Logs: $ns/$pod"
  946. local logs
  947. # Use --tail=500 to catch patterns even in longer outputs.
  948. # For Failed/Error pods, also check --previous (logs from the crashed container instance).
  949. local pod_phase
  950. pod_phase=$(kubectl get pod -n "$ns" "$pod" -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
  951. logs=$(kubectl logs -n "$ns" "$pod" --tail=500 2>/dev/null || echo "")
  952. if [[ -z "$logs" ]]; then
  953. logs=$(kubectl logs -n "$ns" "$pod" --previous --tail=500 2>/dev/null || echo "")
  954. elif [[ "$pod_phase" == "Failed" ]]; then
  955. local prev_logs
  956. prev_logs=$(kubectl logs -n "$ns" "$pod" --previous --tail=500 2>/dev/null || echo "")
  957. [[ -n "$prev_logs" ]] && logs="${logs}"$'\n'"${prev_logs}"
  958. fi
  959. if [[ -z "$logs" ]]; then
  960. info " No logs available"
  961. continue
  962. fi
  963. local _pi
  964. for _pi in "${!NCCL_PAT_KEYS[@]}"; do
  965. local pattern="${NCCL_PAT_KEYS[$_pi]}"
  966. if echo "$logs" | grep -qi "$pattern"; then
  967. local meaning="${NCCL_PAT_VALS[$_pi]}"
  968. local code="${meaning%%:*}"
  969. local desc="${meaning#*:}"
  970. error " DETECTED [$code]: $desc"
  971. echo "$logs" | { grep -i "$pattern" || true; } | tail -3 | while IFS= read -r logline; do
  972. echo -e " ${YELLOW}> $logline${RESET}"
  973. done
  974. issues_in_logs=true
  975. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  976. add_issue_detail "NCCL log pattern [$code] in pod $pod: $desc → references/error-patterns-quick-ref.md" "P1"
  977. fi
  978. done
  979. if echo "$logs" | grep -qiE "BASELINE TEST PASSED|AllReduce SUCCESS|Training complete"; then
  980. success " Pod $pod: completed successfully"
  981. fi
  982. done <<< "$pod_entries"
  983. $issues_in_logs || success "No NCCL error patterns found in pod logs"
  984. }
  985. check_nccl_env_vars() {
  986. header "Check 7: NCCL Environment Variable Audit"
  987. local job_filter=()
  988. [[ -n "$JOB_NAME" ]] && job_filter=(-l "job-name=$JOB_NAME")
  989. local ns="${NAMESPACE:-default}"
  990. local first_pod
  991. first_pod=$(kubectl get pods -n "$ns" ${job_filter[@]+"${job_filter[@]}"} --no-headers 2>/dev/null \
  992. | grep -E " Running " | head -1 | awk '{print $1}' || echo "")
  993. if [[ -z "$first_pod" ]]; then
  994. info "No Running pods found for env var audit (only meaningful during active training)"
  995. return
  996. fi
  997. info "Checking env vars in Running pod: $ns/$first_pod"
  998. local pod_env
  999. pod_env=$(kubectl exec -n "$ns" "$first_pod" -- env 2>/dev/null || echo "")
  1000. if [[ -z "$pod_env" ]]; then
  1001. warn "Could not exec into $first_pod"
  1002. return
  1003. fi
  1004. # Capture Python output; the sentinel line feeds issue accounting below.
  1005. local env_audit_out env_warn_count
  1006. env_audit_out=$(python3 - <<'PYEOF' "$pod_env"
  1007. import sys
  1008. pod_env = sys.argv[1] if len(sys.argv) > 1 else ""
  1009. env_map = {}
  1010. for line in pod_env.strip().split('\n'):
  1011. if '=' in line:
  1012. k, _, v = line.partition('=')
  1013. env_map[k.strip()] = v.strip()
  1014. # (rec_value, severity, description)
  1015. # severity WARN = counts as issue; INFO = advisory only
  1016. checks = {
  1017. 'MASTER_ADDR': (None, 'WARN', 'Must be rank-0 pod hostname/IP'),
  1018. 'MASTER_PORT': ('29500', 'WARN', 'Must match across all ranks'),
  1019. 'WORLD_SIZE': (None, 'WARN', 'Must equal total processes'),
  1020. 'RANK': (None, 'WARN', 'Must be unique 0..WORLD_SIZE-1'),
  1021. 'NCCL_SOCKET_IFNAME': ('^lo,docker,efa,veth,virbr', 'WARN', 'Exclude non-VPC interfaces (loopback/docker/EFA control/veth)'),
  1022. 'NCCL_TIMEOUT': ('1200', 'WARN', 'Default 600s too short for large clusters'),
  1023. 'FI_PROVIDER': ('efa', 'INFO', 'Set to efa on EFA instances; omit for CPU-only'),
  1024. 'FI_EFA_USE_DEVICE_RDMA': ('1', 'INFO', 'Required for full EFA RDMA performance'),
  1025. 'NCCL_DEBUG': ('WARN', 'INFO', 'Enable for diagnostics (use WARN not INFO in prod)'),
  1026. }
  1027. print(" {:<28} {:<22} {}".format('Variable','Value','Status'))
  1028. print(" " + "-"*68)
  1029. warn_count = 0
  1030. for var,(rec,sev,desc) in checks.items():
  1031. val = env_map.get(var)
  1032. if val:
  1033. print(f" [SET] {var:<26} {val:<22}")
  1034. elif sev == 'WARN':
  1035. warn_count += 1
  1036. print(f" [WARN] {var:<26} {'(not set)':<22} <- {desc}")
  1037. else:
  1038. print(f" [INFO] {var:<26} {'(not set)':<22} <- {desc}")
  1039. nccl_debug = env_map.get('NCCL_DEBUG', '')
  1040. if nccl_debug.upper() == 'INFO':
  1041. warn_count += 1
  1042. print("\n [WARN] NCCL_DEBUG=INFO detected in production job — verbose logging adds runtime overhead; set to WARN for production")
  1043. elif nccl_debug.upper() == 'TRACE':
  1044. warn_count += 1
  1045. print("\n [WARN] NCCL_DEBUG=TRACE detected — TRACE prints replayable trace info on every NCCL call (per the NCCL env-var doc); large overhead and gigabytes of logs per rank, set to WARN immediately")
  1046. # NCCL_TIMEOUT value validation (formula: nodes * 5 + 600)
  1047. nccl_timeout_str = env_map.get('NCCL_TIMEOUT', '')
  1048. world_size_str = env_map.get('WORLD_SIZE', '0')
  1049. try:
  1050. world_size = int(world_size_str)
  1051. except ValueError:
  1052. world_size = 0
  1053. if nccl_timeout_str and world_size > 0:
  1054. try:
  1055. nccl_timeout = int(nccl_timeout_str)
  1056. recommended = world_size * 5 + 600
  1057. if nccl_timeout < recommended:
  1058. warn_count += 1
  1059. print(f"\n [WARN] NCCL_TIMEOUT={nccl_timeout}s may be too low for {world_size} ranks (recommended >= {recommended}s)")
  1060. except ValueError:
  1061. pass
  1062. # Large cluster checks (256+ nodes)
  1063. if world_size > 256:
  1064. warn_count += 1
  1065. print(f"\n [WARN] WORLD_SIZE={world_size} (large cluster) — verify memlock and stack ulimits")
  1066. if warn_count == 0:
  1067. print("\n [PASS] All critical NCCL env vars configured")
  1068. else:
  1069. print(f"\n [WARN] {warn_count} critical NCCL env var(s) not set or misconfigured")
  1070. # Sentinel line consumed by the caller — DO NOT remove.
  1071. print(f"__WARN_COUNT__={warn_count}")
  1072. PYEOF
  1073. )
  1074. echo "$env_audit_out" | grep -v '^__WARN_COUNT__='
  1075. env_warn_count=$(echo "$env_audit_out" | grep '^__WARN_COUNT__=' | cut -d= -f2)
  1076. if [[ "${env_warn_count:-0}" =~ ^[0-9]+$ ]] && (( env_warn_count > 0 )); then
  1077. ISSUES_FOUND=$((ISSUES_FOUND + env_warn_count))
  1078. add_issue_detail "${env_warn_count} NCCL env var issue(s) in pod ${ns}/${first_pod} → references/operations.md § 5 NCCL environment variable reference" "P1"
  1079. fi
  1080. }
  1081. # EFA device plugin + NCCL version consistency. kubectl-only, no active job needed.
  1082. check_efa_k8s() {
  1083. header "Check 2b: EFA K8s Device Plugin & NCCL Version Consistency"
  1084. # Without this DaemonSet, pods can't request vpc.amazonaws.com/efa resources
  1085. # and EFA interfaces won't be mounted into training containers.
  1086. local efa_ds
  1087. efa_ds=$(kubectl get daemonset -A 2>/dev/null | grep -iE "efa|aws-efa" | head -3 || echo "")
  1088. if [[ -n "$efa_ds" ]]; then
  1089. success "EFA device plugin DaemonSet found:"
  1090. echo "$efa_ds" | while IFS= read -r line; do info " $line"; done
  1091. else
  1092. # Missing plugin is a FAIL only if any pod requests vpc.amazonaws.com/efa.
  1093. local ns_flag=(); if [[ -n "$NAMESPACE" ]]; then ns_flag=(-n "$NAMESPACE"); else ns_flag=(-A); fi
  1094. local efa_requested
  1095. efa_requested=$(kubectl get pods "${ns_flag[@]}" -o json 2>/dev/null | python3 -c "
  1096. import sys, json
  1097. data = json.load(sys.stdin)
  1098. for pod in data.get('items', []):
  1099. for c in pod.get('spec', {}).get('containers', []):
  1100. lims = c.get('resources', {}).get('limits', {})
  1101. if 'vpc.amazonaws.com/efa' in lims:
  1102. ns = pod['metadata']['namespace']
  1103. name = pod['metadata']['name']
  1104. count = lims['vpc.amazonaws.com/efa']
  1105. print(f' {ns}/{name}: requests {count} EFA interface(s)')
  1106. " 2>/dev/null || echo "")
  1107. if [[ -n "$efa_requested" ]]; then
  1108. error "Pods request EFA resources but EFA device plugin DaemonSet NOT found!"
  1109. error " EFA interfaces will NOT be mounted into training containers"
  1110. echo "$efa_requested"
  1111. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  1112. add_issue_detail "EFA device plugin DaemonSet missing → references/operations.md § 5 NCCL environment variable reference / references/debugging-guide.md § 6 EFA Configuration" "P0"
  1113. else
  1114. info "EFA device plugin not detected (OK if no pods request vpc.amazonaws.com/efa)"
  1115. fi
  1116. fi
  1117. # Mixed NCCL versions across nodes → 'NCCL function not found' at init.
  1118. # Two independent probes:
  1119. # - torch.cuda.nccl.version(): works only if PyTorch is installed.
  1120. # - libnccl.so on disk: authoritative — this is what actually loads at
  1121. # runtime, works for any image (PyTorch, JAX, raw NCCL, custom).
  1122. if [[ -n "$JOB_NAME" ]]; then
  1123. section "NCCL Version Consistency (job: $JOB_NAME)"
  1124. local ns="${NAMESPACE:-default}"
  1125. local job_pods
  1126. job_pods=$(kubectl get pods -n "$ns" -l "job-name=$JOB_NAME" --no-headers 2>/dev/null \
  1127. | grep -E " Running " | awk '{print $1}' | head -4 || echo "")
  1128. if [[ -z "$job_pods" ]]; then
  1129. info "No Running pods in job '$JOB_NAME' — version check skipped"
  1130. else
  1131. # Read-only probe: find libnccl.so*, extract embedded version string,
  1132. # fall back to SONAME filename parsing when `strings` is unavailable.
  1133. # Variables below are expanded inside the remote pod via `kubectl exec
  1134. # sh -c`, NOT locally — the quoted heredoc prevents local expansion.
  1135. local lib_probe
  1136. lib_probe=$(cat <<'REMOTE_PROBE'
  1137. NCCL_LIB=$(find /usr/local/cuda/lib64 /usr/lib /usr/lib64 /usr/lib/x86_64-linux-gnu /opt/nccl/lib /opt/amazon/ofi-nccl/lib -maxdepth 3 -name "libnccl.so*" -type f 2>/dev/null | head -1)
  1138. if [ -z "$NCCL_LIB" ]; then echo "not-found"; exit 0; fi
  1139. VER=$(strings "$NCCL_LIB" 2>/dev/null | grep -oE "NCCL version [0-9]+\.[0-9]+\.[0-9]+" | head -1 | sed "s/NCCL version //")
  1140. [ -z "$VER" ] && VER=$(basename "$(readlink -f "$NCCL_LIB")" 2>/dev/null | grep -oE "[0-9]+\.[0-9]+\.[0-9]+" | head -1)
  1141. [ -z "$VER" ] && VER="present-no-version"
  1142. echo "$VER"
  1143. REMOTE_PROBE
  1144. )
  1145. local torch_versions=()
  1146. local lib_versions=()
  1147. for pod in $job_pods; do
  1148. local tver lver
  1149. tver=$(kubectl exec -n "$ns" "$pod" -- \
  1150. python3 -c "import torch; print(torch.cuda.nccl.version())" 2>/dev/null \
  1151. || echo "unavailable")
  1152. lver=$(kubectl exec -n "$ns" "$pod" -- sh -c "$lib_probe" 2>/dev/null \
  1153. || echo "unavailable")
  1154. info " $pod: torch.nccl=$tver libnccl.so=$lver"
  1155. torch_versions+=("$tver")
  1156. lib_versions+=("$lver")
  1157. done
  1158. local unique_torch unique_lib
  1159. unique_torch=$(printf '%s\n' "${torch_versions[@]}" | grep -v unavailable | sort -u | wc -l | tr -d ' ')
  1160. unique_lib=$(printf '%s\n' "${lib_versions[@]}" \
  1161. | grep -vE "unavailable|not-found|present-no-version" | sort -u | wc -l | tr -d ' ')
  1162. if [[ "$unique_torch" -gt 1 ]]; then
  1163. error "NCCL VERSION MISMATCH (torch.cuda.nccl.version) across pods — will cause 'NCCL function not found' at init!"
  1164. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  1165. add_issue_detail "NCCL version mismatch across pods (torch) → references/debugging-guide.md § 10 NCCL Version Mismatch" "P1"
  1166. fi
  1167. if [[ "$unique_lib" -gt 1 ]]; then
  1168. error "libnccl.so VERSION MISMATCH across pods — mixed NCCL libraries will cause symbol errors at init!"
  1169. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  1170. add_issue_detail "libnccl.so version mismatch across pods → references/debugging-guide.md § 10 NCCL Version Mismatch" "P1"
  1171. fi
  1172. if [[ "$unique_torch" -le 1 ]] && [[ "$unique_lib" -le 1 ]]; then
  1173. if [[ "$unique_torch" -eq 1 ]] || [[ "$unique_lib" -eq 1 ]]; then
  1174. success "NCCL version consistent across ${#lib_versions[@]} pod(s)"
  1175. else
  1176. info "NCCL version unavailable (neither torch nor libnccl.so could be probed)"
  1177. fi
  1178. fi
  1179. fi
  1180. fi
  1181. }
  1182. check_network_policies() {
  1183. header "Check 9: Kubernetes NetworkPolicy Scan"
  1184. local np_flag np_label
  1185. if [[ -n "$NAMESPACE" ]]; then
  1186. np_flag=(-n "$NAMESPACE")
  1187. np_label="'$NAMESPACE'"
  1188. else
  1189. np_flag=(-A)
  1190. np_label="all namespaces"
  1191. fi
  1192. local policies
  1193. policies=$(kubectl get networkpolicy "${np_flag[@]}" 2>/dev/null || echo "")
  1194. if [[ -z "$policies" ]] || echo "$policies" | grep -q "No resources found"; then
  1195. success "No NetworkPolicies in $np_label — all traffic allowed"
  1196. return
  1197. fi
  1198. # Informational — only raise a finding when the per-policy scan below
  1199. # identifies one that actually blocks all ingress/egress. Narrow allow-list
  1200. # policies (e.g. operator-scoped ingress) are common and not a defect.
  1201. info "NetworkPolicies found in $np_label — review each for NCCL impact:"
  1202. echo "$policies"
  1203. local scope_flag
  1204. local scope_flag=()
  1205. if [[ -n "$NAMESPACE" ]]; then scope_flag=(-n "$NAMESPACE"); else scope_flag=(-A); fi
  1206. kubectl get networkpolicy "${scope_flag[@]}" -o json 2>/dev/null | python3 -c "
  1207. import sys, json
  1208. data = json.load(sys.stdin)
  1209. for pol in data.get('items', []):
  1210. name = pol['metadata']['name']
  1211. ns = pol['metadata']['namespace']
  1212. spec = pol.get('spec', {})
  1213. types = spec.get('policyTypes', [])
  1214. ingress = spec.get('ingress', [])
  1215. egress = spec.get('egress', [])
  1216. print(f' Policy: {ns}/{name} | Types: {types}')
  1217. if 'Ingress' in types and not ingress:
  1218. print(f' [FAIL] BLOCKS ALL INBOUND — will break NCCL rendezvous and AllReduce!')
  1219. if 'Egress' in types and not egress:
  1220. print(f' [FAIL] BLOCKS ALL OUTBOUND — will break NCCL communication!')
  1221. if ('Ingress' not in types) and ('Egress' not in types):
  1222. print(f' [INFO] Policy has no policyTypes — acts as allow-all')
  1223. " 2>/dev/null
  1224. local scope_flag2
  1225. local scope_flag2=()
  1226. if [[ -n "$NAMESPACE" ]]; then
  1227. scope_flag2=(-n "$NAMESPACE")
  1228. else
  1229. scope_flag2=(-A)
  1230. fi
  1231. local blocking_list
  1232. blocking_list=$(kubectl get networkpolicy "${scope_flag2[@]}" -o json 2>/dev/null | python3 -c "
  1233. import sys, json
  1234. try:
  1235. data = json.load(sys.stdin)
  1236. except json.JSONDecodeError:
  1237. # kubectl returned non-JSON (empty stdin, error text, or version-skew output).
  1238. # Skip this check rather than aborting the overall diagnostic run.
  1239. sys.exit(0)
  1240. for pol in data.get('items', []):
  1241. name = pol['metadata']['name']
  1242. ns = pol['metadata']['namespace']
  1243. spec = pol.get('spec', {})
  1244. types = spec.get('policyTypes', [])
  1245. ingress = spec.get('ingress', [])
  1246. egress = spec.get('egress', [])
  1247. blocks = ('Ingress' in types and not ingress) or ('Egress' in types and not egress)
  1248. if blocks:
  1249. print(f'{ns}/{name}')
  1250. " 2>/dev/null || echo "")
  1251. if [[ -n "$blocking_list" ]]; then
  1252. while IFS= read -r bp; do
  1253. [[ -z "$bp" ]] && continue
  1254. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  1255. add_issue_detail "Blocking NetworkPolicy $bp may prevent NCCL traffic → references/operations.md § 8 NCCL-specific remediations (NetworkPolicy)" "P1"
  1256. done <<< "$blocking_list"
  1257. fi
  1258. }
  1259. # Populates SSM_CLUSTER_ID and SSM_NODES array (up to SAMPLE_NODES entries).
  1260. # Each entry is "INSTANCE_ID GROUP_NAME".
  1261. # Prefers worker/compute nodes over controller/head nodes.
  1262. # Respects --node <INSTANCE_ID> if provided.
  1263. resolve_cluster_nodes_for_ssm() {
  1264. SSM_CLUSTER_ID=""
  1265. SSM_NODES=()
  1266. local cluster_arn
  1267. cluster_arn=$(aws sagemaker describe-cluster \
  1268. --cluster-name "$CLUSTER_NAME" --region "$REGION" \
  1269. --query 'ClusterArn' --output text 2>/dev/null || echo "")
  1270. if [[ -z "$cluster_arn" || "$cluster_arn" == "None" ]]; then
  1271. debug "resolve_cluster_nodes_for_ssm: describe-cluster returned empty ARN"
  1272. return 1
  1273. fi
  1274. SSM_CLUSTER_ID=$(echo "$cluster_arn" | awk -F'/' '{print $NF}')
  1275. local nodes_json
  1276. nodes_json=$(sagemaker_list_paginated list-cluster-nodes ClusterNodeSummaries)
  1277. if [[ -n "$NODE_ID" ]]; then
  1278. local grp
  1279. grp=$(echo "$nodes_json" | python3 -c "
  1280. import sys, json
  1281. target = sys.argv[1]
  1282. nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
  1283. for n in nodes:
  1284. if n.get('InstanceId') == target:
  1285. print(n.get('InstanceGroupName','worker'))
  1286. break
  1287. " "$NODE_ID" 2>/dev/null | head -1)
  1288. [[ -z "$grp" ]] && grp="worker"
  1289. SSM_NODES=("$NODE_ID $grp")
  1290. return 0
  1291. fi
  1292. local all_nodes
  1293. all_nodes=$(echo "$nodes_json" | python3 -c "
  1294. import sys, json
  1295. print(json.dumps(json.load(sys.stdin).get('ClusterNodeSummaries', [])))
  1296. " 2>/dev/null || echo '[]')
  1297. # For NCCL diagnostics, hardware probes (nvidia-smi, fi_info -p efa,
  1298. # neuron-ls) only produce meaningful signal on GPU / accelerator nodes.
  1299. # Prioritize by type: GPU/Neuron first, other Running compute next, then
  1300. # fall back to any Running node so the script still reports on a cluster
  1301. # that has only CPU nodes.
  1302. local picked
  1303. picked=$(echo "$all_nodes" | python3 -c "
  1304. import sys, json
  1305. nodes = json.load(sys.stdin)
  1306. sample = $SAMPLE_NODES
  1307. # Instance-type prefixes that carry NVIDIA GPUs or AWS Trainium/Inferentia.
  1308. # A node's instance type shows up in ClusterNodeSummaries as e.g. 'ml.p5.48xlarge'.
  1309. GPU_PREFIXES = ('ml.p3', 'ml.p3dn', 'ml.p4d', 'ml.p4de', 'ml.p5', 'ml.p5e',
  1310. 'ml.p5en', 'ml.p6', 'ml.g4dn', 'ml.g5', 'ml.g6', 'ml.g6e', 'ml.g7e')
  1311. NEURON_PREFIXES = ('ml.trn1', 'ml.trn2', 'ml.inf2')
  1312. ACCEL_PREFIXES = GPU_PREFIXES + NEURON_PREFIXES
  1313. def is_utility_group(name):
  1314. n = (name or '').lower()
  1315. return any(x in n for x in ('controller', 'head', 'master'))
  1316. def itype(n):
  1317. return n.get('InstanceType', '') or ''
  1318. running = [n for n in nodes if n.get('InstanceStatus', {}).get('Status', '') == 'Running']
  1319. # Tier 1: running + accelerator type + not a controller group
  1320. tier1 = [n for n in running if itype(n).startswith(ACCEL_PREFIXES) and not is_utility_group(n.get('InstanceGroupName', ''))]
  1321. # Tier 2: running + non-controller (may be CPU-only compute)
  1322. tier2 = [n for n in running if n not in tier1 and not is_utility_group(n.get('InstanceGroupName', ''))]
  1323. # Tier 3: anything else running (utility / controller nodes, last resort)
  1324. tier3 = [n for n in running if n not in tier1 and n not in tier2]
  1325. results = []
  1326. for n in tier1 + tier2 + tier3:
  1327. if len(results) >= sample:
  1328. break
  1329. results.append(n['InstanceId'] + ' ' + n['InstanceGroupName'])
  1330. for r in results:
  1331. print(r)
  1332. " 2>/dev/null || echo "")
  1333. if [[ -z "$picked" ]]; then
  1334. debug "resolve_cluster_nodes_for_ssm: no Running nodes found"
  1335. return 1
  1336. fi
  1337. while IFS= read -r line; do
  1338. [[ -n "$line" ]] && SSM_NODES+=("$line")
  1339. done <<< "$picked"
  1340. return 0
  1341. }
  1342. # Usage: _ssm_run INSTANCE_ID GROUP_NAME CLUSTER_ID SCRIPT_BODY
  1343. # Returns the stdout of the remote script, or empty on failure.
  1344. _ssm_run() {
  1345. local instance_id="$1"
  1346. local group_name="$2"
  1347. local cluster_id="$3"
  1348. local script_body="$4"
  1349. # Validate inputs before interpolating into the SSM target string.
  1350. [[ -z "$instance_id" || -z "$group_name" || -z "$cluster_id" || -z "$script_body" ]] && return 1
  1351. [[ ! "$instance_id" =~ ^i-[0-9a-f]{8,17}$ ]] && return 1
  1352. [[ ! "$group_name" =~ ^[A-Za-z0-9._-]+$ ]] && return 1
  1353. [[ ! "$cluster_id" =~ ^[A-Za-z0-9._-]+$ ]] && return 1
  1354. local target="sagemaker-cluster:${cluster_id}_${group_name}-${instance_id}"
  1355. local tmpfile
  1356. tmpfile=$(mktemp "${TMPDIR:-/tmp}/nccl-ssm-XXXXXX.json") || return 1
  1357. chmod 600 "$tmpfile" 2>/dev/null || true
  1358. _TEMP_FILES+=("$tmpfile")
  1359. # AWS-StartNonInteractiveCommand collapses newlines in a single command
  1360. # element, so embed the multi-line script as a base64 payload.
  1361. local cmd_b64
  1362. cmd_b64=$(printf '%s' "$script_body" | base64 | tr -d '\n') || { rm -f "$tmpfile"; return 1; }
  1363. local remote="bash -c \"echo $cmd_b64 | base64 -d | bash\""
  1364. python3 -c "import json,sys; print(json.dumps({'command':[sys.argv[1]]}))" "$remote" > "$tmpfile" 2>/dev/null || { rm -f "$tmpfile"; return 1; }
  1365. # session-manager-plugin races to close before flushing its last stdout
  1366. # block; `unbuffer` (from the `expect` package) gives it a PTY and avoids
  1367. # the resulting "Cannot perform start session: EOF". Required — see the
  1368. # prerequisite check at script startup.
  1369. # Retry transient SSM session errors (EOF, throttling, i/o timeout).
  1370. # Do not retry AccessDenied / UnauthorizedOperation — permanent IAM denials.
  1371. local out attempt=0
  1372. while (( attempt < 5 )); do
  1373. out=$(unbuffer timeout 180 aws ssm start-session \
  1374. --target "$target" \
  1375. --region "$REGION" \
  1376. --document-name AWS-StartNonInteractiveCommand \
  1377. --parameters "file://$tmpfile" 2>&1 || echo "")
  1378. # Fatal (don't retry) — permanent IAM or agent state.
  1379. if echo "$out" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized to perform|TargetNotConnected"; then
  1380. break
  1381. fi
  1382. if ! echo "$out" | grep -qiE "Cannot perform start session|EOF$|SessionManagerPlugin is not found|i/o timeout|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable"; then
  1383. break
  1384. fi
  1385. attempt=$((attempt + 1))
  1386. sleep $((attempt * 3))
  1387. done
  1388. rm -f "$tmpfile"
  1389. # Strip SSM session banners and the echoed base64 command line.
  1390. echo "$out" | grep -vE '^(Starting session with SessionId:|Exiting session with sessionId:|\s*$)' \
  1391. | grep -vE "^(bash -c \"echo [A-Za-z0-9+/=]+ \| base64 -d \| bash\"|echo '[A-Za-z0-9+/=]+'|[A-Za-z0-9+/=]{40,}={0,2})[[:space:]]*\|?[[:space:]]*base64?[[:space:]]*-?d?[[:space:]]*\|?[[:space:]]*bash\"?\$" || true
  1392. }
  1393. # Self-contained bash script executed on each HyperPod compute node via SSM.
  1394. # Covers GPU, EFA, NCCL library, network, memory, and process health.
  1395. IFS= read -r -d '' _NODE_DIAG_SCRIPT <<'NODE_SCRIPT' || true
  1396. #!/bin/bash
  1397. # HyperPod NCCL Node Hardware Diagnostics
  1398. # Runs ON the compute node via SSM — NOT on the local machine.
  1399. export PATH="/opt/amazon/efa/bin:/usr/local/cuda/bin:$PATH"
  1400. echo "=== NODE DIAGNOSTICS ==="
  1401. echo "Host: $(hostname)"
  1402. echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  1403. echo "Kernel: $(uname -r)"
  1404. # Instance type via IMDS (v2)
  1405. IMDS_TOKEN=$(curl -sf -m 3 -X PUT "http://169.254.169.254/latest/api/token" \
    • Dismissed after review: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
    • Line 1,547 on GitHub
  1406. -H "X-aws-ec2-metadata-token-ttl-seconds: 60" 2>/dev/null || echo "")
  1407. if [ -n "$IMDS_TOKEN" ]; then
  1408. INSTANCE_TYPE=$(curl -sf -m 3 -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  1409. "http://169.254.169.254/latest/meta-data/instance-type" 2>/dev/null || echo "unknown")
  1410. AZ=$(curl -sf -m 3 -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  1411. "http://169.254.169.254/latest/meta-data/placement/availability-zone" 2>/dev/null || echo "unknown")
  1412. else
  1413. INSTANCE_TYPE="unknown"
  1414. AZ="unknown"
  1415. fi
  1416. echo "Instance: ${INSTANCE_TYPE} | AZ: ${AZ}"
  1417. echo ""
  1418. echo "--- GPU ---"
  1419. # Require both the binary AND at least one GPU visible. nvidia-smi is preinstalled
  1420. # on some non-GPU instance types (t3/c5) but returns "No devices were found" —
  1421. # reporting that as [FAIL] would be a false positive on controllers/logins.
  1422. if command -v nvidia-smi &>/dev/null && nvidia-smi -L 2>/dev/null | grep -q "^GPU"; then
  1423. nvidia-smi --query-gpu=index,name,driver_version,memory.used,memory.total,temperature.gpu,utilization.gpu \
  1424. --format=csv,noheader 2>/dev/null \
  1425. && echo "" \
  1426. || echo "[FAIL] nvidia-smi query failed"
  1427. # XID errors indicate hardware faults that will cause NCCL to abort.
  1428. # Modern A100/H100 drivers log XIDs to dmesg but NOT to nvidia-smi -q,
  1429. # so check both sources — verified on-hardware with A100 driver 580.126
  1430. # where an injected XID 31 appeared in dmesg but was invisible to -q.
  1431. XID_DMESG=$(dmesg 2>/dev/null | grep -E 'NVRM: Xid' | tail -5)
  1432. XID_SMI=$(nvidia-smi -q 2>/dev/null | grep -E '^[[:space:]]*Xid' | head -5)
  1433. if [ -n "$XID_DMESG" ] || [ -n "$XID_SMI" ]; then
  1434. echo "[FAIL] GPU XID ERRORS DETECTED (hardware fault — NCCL will abort):"
  1435. [ -n "$XID_DMESG" ] && echo "$XID_DMESG"
  1436. [ -n "$XID_SMI" ] && echo "$XID_SMI"
  1437. else
  1438. echo "[PASS] No GPU XID errors"
  1439. fi
  1440. # Only surface nonzero ECC counts. 'ECC Errors' section header and
  1441. # 'Uncorrectable ... : 0' lines fire on every healthy GPU.
  1442. ECC=$(nvidia-smi -q 2>/dev/null | awk '
  1443. /Uncorrectable/ { if ($NF ~ /^[0-9]+$/ && $NF+0 > 0) print }
  1444. ' | head -5)
  1445. [ -n "$ECC" ] && echo "[FAIL] GPU uncorrectable ECC errors detected: $ECC" || echo "[PASS] No ECC errors"
  1446. # Row-remap state — marginal GPU memory. Pending rows need a reset to finalize;
  1447. # Failed means exceeded remap capacity (bad memory). Silent degrader that
  1448. # default DCGM medium + memtest in some driver versions miss entirely.
  1449. REMAP=$(nvidia-smi --query-remapped-rows=gpu_bus_id,remapped_rows.pending,remapped_rows.failure \
  1450. --format=csv,noheader 2>/dev/null)
  1451. if [ -n "$REMAP" ]; then
  1452. PENDING_SUM=$(echo "$REMAP" | awk -F, '{gsub(/ /,""); s+=$2} END {print s+0}')
  1453. FAILED_COUNT=$(echo "$REMAP" | awk -F, '{gsub(/ /,""); if ($3=="Yes" || $3=="1") c++} END {print c+0}')
  1454. if [ "$FAILED_COUNT" -gt 0 ]; then
  1455. echo "[FAIL] GPU row-remap FAILED on $FAILED_COUNT device(s) — bad memory, replace GPU"
  1456. elif [ "$PENDING_SUM" -gt 0 ]; then
  1457. echo "[FAIL] GPU row-remap PENDING ($PENDING_SUM row(s)) — marginal memory; reset/reboot to finalize"
  1458. echo " If pending persists across reboots, firmware may be stuck — replace GPU"
  1459. else
  1460. echo "[PASS] GPU row-remap: no pending or failed rows"
  1461. fi
  1462. fi
  1463. # DCGM health — complements XID/ECC above. Parse Fail/Warn verdicts only
  1464. # (Pass is not authoritative on DCGM <= 3.3.9 due to memtest bug).
  1465. if command -v dcgmi >/dev/null 2>&1; then
  1466. DCGM_OUT=$(dcgmi health --check -j 2>/dev/null || dcgmi health --check 2>/dev/null || echo "")
  1467. if echo "$DCGM_OUT" | grep -qiE '"overall_health"\s*:\s*"(Fail|Warn)"|HEALTH_RESULT_FAIL|HEALTH_RESULT_WARN|Health Monitor Report.*(Fail|Warn)'; then
  1468. echo "[FAIL] DCGM health check reports Fail/Warn — inspect with 'dcgmi health --check'"
  1469. fi
  1470. fi
  1471. # DCGM nvvs log presence — HyperPod deep-health-check writes here.
  1472. if [ -d /var/log/nvidia-dcgm ]; then
  1473. NVVS_LATEST=$(find /var/log/nvidia-dcgm -maxdepth 1 -name 'nvvs*.log' -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -1 | awk '{print $2}')
  1474. if [ -n "$NVVS_LATEST" ]; then
  1475. if tail -n 200 "$NVVS_LATEST" 2>/dev/null | grep -qiE 'row ?remap.*(pending|fail)|FAIL: |Error: '; then
  1476. echo "[FAIL] DCGM nvvs log contains failure / row-remap signals: $NVVS_LATEST"
  1477. fi
  1478. fi
  1479. fi
  1480. # NVLink — important for p4d/p5 multi-GPU NCCL bandwidth.
  1481. # Output format across driver versions:
  1482. # - 'Link N: X GB/s' (active, driver 470+)
  1483. # - 'Link N: Active' (older drivers)
  1484. # - 'error'/'fail'/'inactive' keywords when degraded
  1485. NVLINK=$(nvidia-smi nvlink --status 2>/dev/null | head -200)
  1486. if echo "$NVLINK" | grep -qiE "error|fail|inactive"; then
  1487. echo "[FAIL] NVLink errors/inactive links detected (replace node):"
  1488. echo "$NVLINK" | grep -iE "error|fail|inactive"
  1489. else
  1490. ACTIVE_COUNT=$(echo "$NVLINK" | grep -cE "Link [0-9]+:[[:space:]]+([0-9]+ GB/s|Active)" || true)
  1491. if [ "${ACTIVE_COUNT:-0}" -gt 0 ]; then
  1492. echo "[PASS] NVLink: $ACTIVE_COUNT active link(s)"
  1493. else
  1494. echo "[INFO] NVLink not available (expected on single-GPU or non-NVLink instances)"
  1495. fi
  1496. fi
  1497. # GPU P2P topology — critical for intra-node NCCL AllReduce performance
  1498. echo ""
  1499. echo "--- GPU P2P Topology (nvidia-smi topo) ---"
  1500. nvidia-smi topo -m 2>/dev/null | head -25 | while IFS= read -r line; do
  1501. if echo "$line" | grep -qiE "NV[0-9]|NVLink"; then
  1502. echo " [PASS] $line"
  1503. elif echo "$line" | grep -qiE "PIX|PXB|PHB|SOC"; then
  1504. echo " [WARN] $line <- PCIe path (slower than NVLink)"
  1505. else
  1506. echo " [INFO] $line"
  1507. fi
  1508. done
  1509. # PCI ACS — intercepts GPU Direct P2P → 10-50x slower intra-node AllReduce or hang
  1510. echo ""
  1511. echo "--- PCI ACS (Access Control Services) ---"
  1512. if command -v lspci &>/dev/null; then
  1513. ACS_ENABLED=$(lspci -vvv 2>/dev/null | grep -A20 "PCI bridge\|Root Port\|Upstream Port" \
  1514. | grep "ACSCtl:" | { grep -c "SrcValid+" 2>/dev/null; true; })
  1515. if [ "$ACS_ENABLED" -gt 0 ] 2>/dev/null; then
  1516. echo "[FAIL] ACS enabled on $ACS_ENABLED PCI bridge(s) — GPU Direct P2P blocked!"
  1517. echo " Symptom: 'NCCL WARN P2P not supported between dev X and dev Y'"
  1518. echo " Impact: 10-50x slower intra-node AllReduce"
  1519. else
  1520. echo "[PASS] ACS not enabled on PCI bridges — GPU Direct P2P unobstructed"
  1521. fi
  1522. else
  1523. echo "[INFO] lspci not available — install pciutils to check ACS"
  1524. fi
  1525. IOMMU=$(dmesg 2>/dev/null | grep -iE "iommu.*enabled|dmar.*enabled" | head -2 || \
  1526. grep -oE "intel_iommu=[^ ]+|iommu=[^ ]+" /proc/cmdline 2>/dev/null | head -1 || echo "")
  1527. if [ -n "$IOMMU" ]; then
  1528. echo "[WARN] IOMMU may be enabled: $IOMMU"
  1529. echo " On baremetal: disable VT-d/IOMMU in BIOS for best GPU Direct P2P"
  1530. echo " In VMs: normal — use ATS on network adapters"
  1531. else
  1532. echo "[PASS] IOMMU: not detected as enabled"
  1533. fi
  1534. [ "${NCCL_P2P_DISABLE:-0}" = "1" ] && \
  1535. echo "[WARN] NCCL_P2P_DISABLE=1 set — workaround active, performance degraded" || true
  1536. # nvidia-peermem — GPU Direct RDMA to NIC (required for EFA↔GPU on p4d/p5)
  1537. echo ""
  1538. echo "--- nvidia-peermem (GPU Direct RDMA) ---"
  1539. if lsmod 2>/dev/null | grep -q "nvidia_peermem\|nv_peer_mem"; then
  1540. echo "[PASS] nvidia-peermem loaded — GPU Direct RDMA to EFA/NIC enabled"
  1541. else
  1542. # Kernel 5.12+ uses DMA-BUF instead of nvidia-peermem.
  1543. KVER_MAJOR=$(uname -r | cut -d. -f1)
  1544. KVER_MINOR=$(uname -r | cut -d. -f2)
  1545. if [ "$KVER_MAJOR" -gt 5 ] || { [ "$KVER_MAJOR" -eq 5 ] && [ "$KVER_MINOR" -ge 12 ]; } 2>/dev/null; then
  1546. echo "[INFO] nvidia-peermem not loaded; kernel $(uname -r) supports DMA-BUF (auto-detected)"
  1547. else
  1548. echo "[WARN] nvidia-peermem NOT loaded — EFA↔GPU copies go through CPU"
  1549. fi
  1550. fi
  1551. else
  1552. if command -v nvidia-smi &>/dev/null; then
  1553. echo "[INFO] nvidia-smi installed but no GPU devices visible — likely a CPU-only node (controller/login)"
  1554. else
  1555. echo "[INFO] nvidia-smi not found — CPU-only node or GPU driver not installed"
  1556. fi
  1557. fi
  1558. echo ""
  1559. echo "--- EFA ---"
  1560. if lsmod 2>/dev/null | grep -q "^efa "; then
  1561. EFA_MOD_VER=$(modinfo efa 2>/dev/null | grep "^version:" | awk '{print $2}' || echo "unknown")
  1562. echo "[PASS] EFA kernel module loaded (version: ${EFA_MOD_VER})"
  1563. else
  1564. EFA_DEVS=$(ls /dev/infiniband/uverbs* 2>/dev/null || echo "")
  1565. EFA_IFACES=$(ip -br link show 2>/dev/null | grep -cE "^efa" || echo 0)
  1566. if [ -n "$EFA_DEVS" ] || [ "$EFA_IFACES" -gt 0 ] 2>/dev/null; then
  1567. echo "[FAIL] EFA devices present but kernel module NOT loaded — NCCL EFA will fail"
  1568. else
  1569. echo "[INFO] EFA kernel module not loaded (expected on non-EFA instances)"
  1570. fi
  1571. fi
  1572. FI_CMD=""
  1573. command -v fi_info &>/dev/null && FI_CMD="fi_info"
  1574. [ -z "$FI_CMD" ] && [ -f /opt/amazon/efa/bin/fi_info ] && FI_CMD="/opt/amazon/efa/bin/fi_info"
  1575. if [ -n "$FI_CMD" ]; then
  1576. EFA_OUTPUT=$($FI_CMD -p efa 2>&1)
  1577. if echo "$EFA_OUTPUT" | grep -q "provider: efa"; then
  1578. EFA_COUNT=$(echo "$EFA_OUTPUT" | { grep -c "provider: efa" 2>/dev/null; true; })
  1579. echo "[PASS] EFA provider available: $EFA_COUNT interface(s)"
  1580. echo "$EFA_OUTPUT" | grep "device:" | head -5
  1581. # Validate EFA count against expected per-instance-type counts. A subset
  1582. # of NICs silently failing to attach is a top NCCL failure mode (training
  1583. # runs at reduced bandwidth with no error). Counts per AWS EC2 docs.
  1584. IMDS_TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
    • Dismissed after review: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
    • Line 1,743 on GitHub
  1585. -H "X-aws-ec2-metadata-token-ttl-seconds: 60" --connect-timeout 2 2>/dev/null || echo "")
  1586. if [ -n "$IMDS_TOKEN" ]; then
  1587. INST_TYPE=$(curl -s -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  1588. http://169.254.169.254/latest/meta-data/instance-type --connect-timeout 2 2>/dev/null || echo "")
  1589. # Counts only included where AWS publishes them in the EC2 EFA
  1590. # docs (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html).
  1591. # For other types the doc lists bandwidth but not card count, so we
  1592. # skip the check rather than guess.
  1593. case "$INST_TYPE" in
  1594. p5.48xlarge|p5e.48xlarge) EXPECTED_EFA=32 ;;
  1595. p5en.48xlarge) EXPECTED_EFA=16 ;;
  1596. p4d.24xlarge) EXPECTED_EFA=4 ;;
  1597. p6-b200.48xlarge) EXPECTED_EFA=8 ;;
  1598. p6-b300.48xlarge) EXPECTED_EFA=17 ;;
  1599. p6e-gb200.36xlarge) EXPECTED_EFA=17 ;;
  1600. *) EXPECTED_EFA=0 ;;
  1601. esac
  1602. if [ "$EXPECTED_EFA" -gt 0 ] 2>/dev/null; then
  1603. if [ "$EFA_COUNT" -lt "$EXPECTED_EFA" ] 2>/dev/null; then
  1604. echo "[FAIL] EFA count mismatch on ${INST_TYPE}: got ${EFA_COUNT}, expected ${EXPECTED_EFA}"
  1605. echo " A subset of NICs failed to attach — NCCL will run at reduced bandwidth"
  1606. else
  1607. echo "[PASS] EFA count matches ${INST_TYPE} expected value (${EXPECTED_EFA})"
  1608. fi
  1609. else
  1610. echo "[INFO] EFA count validation skipped — no expected value for ${INST_TYPE:-unknown}"
  1611. fi
  1612. fi
  1613. else
  1614. # Determine whether EFA is expected — absence on non-EFA instance types
  1615. # (t3, c5, controllers) is normal, not a failure.
  1616. INST_TYPE_CHECK="${INST_TYPE:-}"
  1617. case "$INST_TYPE_CHECK" in
  1618. p4d.*|p4de.*|p5.*|p5e.*|p5en.*|p6*|trn1.*|trn2.*)
  1619. echo "[FAIL] EFA provider NOT available on ${INST_TYPE_CHECK}"
  1620. echo " fi_info -p efa returned no results"
  1621. echo " Required for NCCL on this instance type — training will fall back to TCP (very slow)"
  1622. ;;
  1623. *)
  1624. echo "[INFO] EFA provider not available — expected on non-EFA instance type (${INST_TYPE_CHECK:-unknown})"
  1625. ;;
  1626. esac
  1627. fi
  1628. TCP_COUNT=$($FI_CMD -p tcp 2>/dev/null | { grep -c "provider: tcp" 2>/dev/null; true; })
  1629. LF_VER=$($FI_CMD --version 2>&1 | grep libfabric | sed 's/.*: //' | head -1)
  1630. echo " libfabric: ${LF_VER:-unknown} | TCP fallback endpoints: $TCP_COUNT"
  1631. else
  1632. echo "[INFO] fi_info not found — EFA tools not installed (OK for non-EFA instances)"
  1633. fi
  1634. [ -f /opt/amazon/efa_installed_packages ] && \
  1635. grep "# EFA installer version" /opt/amazon/efa_installed_packages | head -1 \
  1636. || echo "[INFO] /opt/amazon/efa_installed_packages not found"
  1637. # aws-ofi-nccl — bridges NCCL and EFA, required for GPU training on EFA instances
  1638. OFI_LIB=$(find /opt/amazon/ofi-nccl /usr/local/lib /usr/lib /opt/aws-ofi-nccl/lib \
  1639. -maxdepth 4 -name "libnccl-net.so" 2>/dev/null | head -1)
  1640. NCCL_NET_PLUGIN_ENV="${NCCL_NET_PLUGIN:-}"
  1641. if [ -n "$NCCL_NET_PLUGIN_ENV" ]; then
  1642. [ -f "$NCCL_NET_PLUGIN_ENV" ] && \
  1643. echo "[PASS] NCCL_NET_PLUGIN=$NCCL_NET_PLUGIN_ENV (file exists)" || \
  1644. echo "[FAIL] NCCL_NET_PLUGIN=$NCCL_NET_PLUGIN_ENV — FILE NOT FOUND! NCCL EFA will fail"
  1645. elif [ -n "$OFI_LIB" ]; then
  1646. echo "[PASS] aws-ofi-nccl plugin: $OFI_LIB"
  1647. else
  1648. # FAIL only if FI_PROVIDER=efa is set; otherwise informational.
  1649. [ "${FI_PROVIDER:-}" = "efa" ] && \
  1650. echo "[FAIL] FI_PROVIDER=efa but aws-ofi-nccl plugin not found — NCCL EFA will fail" || \
  1651. echo "[INFO] aws-ofi-nccl not found (required for EFA+NCCL; not needed for CPU-only)"
  1652. fi
  1653. # Hugepages — improve EFA/RDMA memory registration performance
  1654. HP_2M=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 2>/dev/null || echo 0)
  1655. HP_1G=$(cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages 2>/dev/null || echo 0)
  1656. if [ "$HP_2M" -gt 0 ] 2>/dev/null; then
  1657. HP_FREE=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages 2>/dev/null || echo 0)
  1658. echo "[PASS] 2MB hugepages: ${HP_2M} total, ${HP_FREE} free"
  1659. [ "$HP_FREE" -eq 0 ] && echo "[WARN] All hugepages in use — RDMA may have reduced performance"
  1660. elif [ "$HP_1G" -gt 0 ] 2>/dev/null; then
  1661. echo "[PASS] 1GB hugepages: ${HP_1G} allocated"
  1662. else
  1663. echo "[INFO] No hugepages configured (set vm.nr_hugepages=512 for optimal EFA RDMA)"
  1664. fi
  1665. echo ""
  1666. echo "--- NCCL ---"
  1667. NCCL_LIB=$(find /usr/local/cuda*/lib* /usr/lib /opt/nccl/lib 2>/dev/null \
  1668. -maxdepth 4 -name "libnccl.so*" 2>/dev/null | head -3)
  1669. if [ -n "$NCCL_LIB" ]; then
  1670. echo "[PASS] NCCL library found:"
  1671. echo "$NCCL_LIB" | while read -r l; do echo " $l"; done
  1672. else
  1673. echo "[INFO] NCCL library not found (install NCCL for distributed GPU training)"
  1674. fi
  1675. NCCL_HDR=$(find /usr/local/cuda*/include /usr/include /opt/nccl/include 2>/dev/null \
  1676. -maxdepth 3 -name "nccl.h" 2>/dev/null | head -1)
  1677. if [ -n "$NCCL_HDR" ]; then
  1678. NCCL_VER=$(grep -E "NCCL_MAJOR|NCCL_MINOR|NCCL_PATCH" "$NCCL_HDR" 2>/dev/null \
  1679. | awk '{print $3}' | tr '\n' '.' | sed 's/\.$//')
  1680. [ -n "$NCCL_VER" ] && echo " NCCL version: $NCCL_VER"
  1681. fi
  1682. CUDA_DRV=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | head -1 | tr -d ' ' || echo "")
  1683. if [ -n "$CUDA_DRV" ] && [ -n "$NCCL_VER" ]; then
  1684. DRV_MAJOR=$(echo "$CUDA_DRV" | cut -d. -f1)
  1685. NCCL_MAJOR=$(echo "$NCCL_VER" | cut -d. -f1)
  1686. NCCL_MINOR=$(echo "$NCCL_VER" | cut -d. -f2)
  1687. # NCCL 2.20+ requires CUDA driver >= 525
  1688. if { [ "$NCCL_MAJOR" -gt 2 ] || { [ "$NCCL_MAJOR" -eq 2 ] && [ "${NCCL_MINOR:-0}" -ge 20 ]; }; } && [ "$DRV_MAJOR" -gt 0 ] && [ "$DRV_MAJOR" -lt 525 ] 2>/dev/null; then
  1689. echo "[WARN] NCCL $NCCL_VER may require CUDA driver >= 525; found $CUDA_DRV"
  1690. echo " Symptom: 'NCCL function not found' on mixed-version nodes"
  1691. fi
  1692. fi
  1693. echo ""
  1694. echo "--- Network Interfaces ---"
  1695. ip -br addr show 2>/dev/null | while IFS= read -r line; do
  1696. IFACE=$(echo "$line" | awk '{print $1}')
  1697. STATE=$(echo "$line" | awk '{print $2}')
  1698. ADDR=$(echo "$line" | awk '{print $3}')
  1699. if echo "$IFACE" | grep -q "^lo"; then TYPE="loopback"
  1700. elif echo "$IFACE" | grep -qE "^efa|^rdma"; then TYPE="EFA device"
  1701. elif echo "$IFACE" | grep -qE "^ib[0-9]"; then TYPE="InfiniBand"
  1702. elif echo "$IFACE" | grep -qE "^eth|^ens|^enp|^en[0-9]"; then TYPE="VPC ENI"
  1703. elif echo "$IFACE" | grep -qE "^docker|^br-|^veth"; then TYPE="container bridge"
  1704. else TYPE="other"; fi
  1705. printf " %-18s %-8s %-20s (%s)\n" "$IFACE" "$STATE" "${ADDR:--}" "$TYPE"
  1706. done
  1707. echo ""
  1708. echo "--- MTU ---"
  1709. ip -br link show 2>/dev/null | grep -v "^lo" | while IFS= read -r line; do
  1710. IFACE=$(echo "$line" | awk '{print $1}')
  1711. MTU=$(ip link show "$IFACE" 2>/dev/null | grep -o "mtu [0-9]*" | awk '{print $2}')
  1712. [ -z "$MTU" ] && continue
  1713. if echo "$IFACE" | grep -qE "docker|br-|veth"; then echo " [INFO] $IFACE: MTU=$MTU (container bridge — OK)"
  1714. elif [ "$MTU" -ge 9000 ] 2>/dev/null; then echo " [PASS] $IFACE: MTU=$MTU (jumbo frames — optimal for EFA)"
  1715. else echo " [WARN] $IFACE: MTU=$MTU — expected 9001 for EFA/RDMA (fragmentation risk for large tensors)"; fi
  1716. done
  1717. echo ""
  1718. echo "--- Memory & Limits ---"
  1719. free -h
  1720. echo ""
  1721. SHM_SIZE=$(df -BG /dev/shm 2>/dev/null | tail -1 | awk '{print $2}' | tr -d 'G')
  1722. SHM_FS=$(df -T /dev/shm 2>/dev/null | tail -1 | awk '{print $2}' || echo "unknown")
  1723. if [ -n "$SHM_SIZE" ] && [ "$SHM_SIZE" -ge 1 ] 2>/dev/null; then
  1724. echo "[PASS] /dev/shm: ${SHM_SIZE}GB (fs: ${SHM_FS})"
  1725. [ "$SHM_SIZE" -lt 4 ] 2>/dev/null && \
  1726. echo "[WARN] /dev/shm ${SHM_SIZE}GB < 4GB — consider 4GB+ for large model training"
  1727. else
  1728. echo "[FAIL] /dev/shm: ${SHM_SIZE:-0}GB — NCCL needs ≥1GB (K8s default=64MB)"
  1729. echo " Symptom: 'failed to extend /dev/shm/nccl-*' or Bus error"
  1730. fi
  1731. [ "$SHM_FS" != "tmpfs" ] && [ "$SHM_FS" != "unknown" ] && \
  1732. echo "[WARN] /dev/shm fs type: $SHM_FS (expected tmpfs)"
  1733. MEMLOCK=$(ulimit -l 2>/dev/null || echo "unknown")
  1734. if [ "$MEMLOCK" = "0" ]; then
  1735. echo "[FAIL] memlock=0 — InfiniBand/EFA RDMA memory registration WILL FAIL"
  1736. echo " Symptom: 'NCCL WARN Call to ibv_reg_mr failed'"
  1737. elif [ -n "$MEMLOCK" ] && [ "$MEMLOCK" != "unlimited" ] && [ "$MEMLOCK" -ge 8388608 ] 2>/dev/null; then
  1738. echo "[PASS] memlock=${MEMLOCK}KB (≥8GB — OK)"
  1739. elif [ "$MEMLOCK" = "unlimited" ]; then
  1740. echo "[INFO] memlock=unlimited (OK for RDMA; see stack check below for libc quirk)"
  1741. else
  1742. echo "[INFO] memlock=${MEMLOCK}KB"
  1743. fi
  1744. # Stack size — GNU libc quirk: when memlock=unlimited, thread stack is reduced to 2MB.
  1745. # NCCL topology graph search (especially MNNVL on 256+ nodes) needs 8MB+ stack.
  1746. STACK=$(ulimit -s 2>/dev/null || echo "unknown")
  1747. if [ "$MEMLOCK" = "unlimited" ] && [ "$STACK" = "unlimited" ]; then
  1748. echo "[WARN] memlock=unlimited + stack=unlimited — GNU libc reduces NCCL thread stack to 2MB"
  1749. echo " NCCL MNNVL/large topology graph search needs 8MB+ and will fail"
  1750. elif [ "$STACK" = "unlimited" ]; then
  1751. echo "[PASS] stack=unlimited (memlock is bounded, so libc quirk does not apply)"
  1752. elif [ "$STACK" != "unknown" ] && [ "$STACK" -lt 4096 ] 2>/dev/null; then
  1753. echo "[FAIL] stack=${STACK}KB — too small for NCCL topology search (need ≥4096KB)"
  1754. else
  1755. echo "[PASS] stack=${STACK:-unknown}KB"
  1756. fi
  1757. # systemd RemoveIPC — deletes NCCL shm files when session ends (Slurm nodes)
  1758. # Strip comment lines first; many distros ship logind.conf with `#RemoveIPC=yes`
  1759. # as the documented default, which would false-WARN on a substring match.
  1760. if [ -f /etc/systemd/logind.conf ]; then
  1761. REMOVEIPC=$(grep -v '^[[:space:]]*#' /etc/systemd/logind.conf 2>/dev/null \
  1762. | grep -i "RemoveIPC" | tail -1 || echo "")
  1763. if [ -z "$REMOVEIPC" ]; then
  1764. echo "[WARN] RemoveIPC unset in /etc/systemd/logind.conf — defaults to 'yes' on RHEL/Amazon Linux"
  1765. echo " Symptom: 'unlink shared memory /dev/shm/nccl-* failed: No such file'"
  1766. elif echo "$REMOVEIPC" | grep -qi "yes\|true\|1"; then
  1767. echo "[WARN] systemd RemoveIPC=yes — NCCL shm files will be deleted at session end"
  1768. echo " Symptom: 'unlink shared memory /dev/shm/nccl-* failed: No such file'"
  1769. else
  1770. echo "[PASS] systemd RemoveIPC=no — NCCL shm files will not be deleted"
  1771. fi
  1772. fi
  1773. # cuMem NUMA (NCCL 2.23+)
  1774. NUMA_NODES=$(ls /sys/devices/system/node/ 2>/dev/null | { grep -c "^node[0-9]" 2>/dev/null; true; })
  1775. if [ "$NUMA_NODES" -gt 0 ] 2>/dev/null; then
  1776. echo "[PASS] NUMA topology: $NUMA_NODES node(s) visible (cuMem host alloc OK)"
  1777. else
  1778. echo "[WARN] NUMA topology not visible — cuMem host allocations may fail"
  1779. fi
  1780. echo ""
  1781. echo "--- NCCL RAS ---"
  1782. # RAS port is configurable via NCCL_RAS_ADDR. Probe whatever is in the
  1783. # training process's environment; skip if no candidate is found rather than
  1784. # hard-coding a port that may not match every NCCL build.
  1785. NC_CMD=$(command -v nc 2>/dev/null || command -v ncat 2>/dev/null || echo "")
  1786. if [ -n "$NC_CMD" ]; then
  1787. RAS_PID=$(pgrep -f "python|torchrun|mpirun" 2>/dev/null | head -1)
  1788. RAS_ADDR=""
  1789. if [ -n "$RAS_PID" ] && [ -r "/proc/$RAS_PID/environ" ]; then
  1790. RAS_ADDR=$(tr '\0' '\n' < "/proc/$RAS_PID/environ" 2>/dev/null \
  1791. | awk -F= '/^NCCL_RAS_ADDR=/{print $2}' | head -1)
  1792. fi
  1793. RAS_HOST="${RAS_ADDR%:*}"; RAS_PORT="${RAS_ADDR##*:}"
  1794. if [ -n "$RAS_PORT" ] && [ "$RAS_PORT" != "$RAS_ADDR" ]; then
  1795. RAS=$(echo "status" | timeout 3 $NC_CMD -w 2 "${RAS_HOST:-localhost}" "$RAS_PORT" 2>/dev/null || echo "")
  1796. if [ -n "$RAS" ]; then
  1797. echo "[PASS] NCCL RAS responding at ${RAS_HOST:-localhost}:${RAS_PORT}:"
  1798. echo "$RAS" | head -10
  1799. else
  1800. echo "[INFO] NCCL RAS port ${RAS_PORT} not responding — training job may not be using RAS, or RAS is disabled (NCCL_RAS_ENABLE=0)"
  1801. fi
  1802. else
  1803. echo "[INFO] NCCL_RAS_ADDR not set in any training process — skipping RAS probe (set NCCL_RAS_ADDR=<host>:<port> and re-run during training to enable)"
  1804. fi
  1805. else
  1806. echo "[INFO] nc/ncat not found — cannot probe NCCL RAS"
  1807. fi
  1808. echo ""
  1809. echo "--- Active Training Processes ---"
  1810. PROCS=$(ps aux 2>/dev/null | grep -E "python|torchrun|mpirun|nccl_test" | grep -v grep | head -10)
  1811. if [ -n "$PROCS" ]; then
  1812. echo "$PROCS"
  1813. else
  1814. echo "[INFO] No active training processes"
  1815. fi
  1816. echo ""
  1817. echo "--- Recent Hardware Errors (dmesg) ---"
  1818. DMESG=$(dmesg 2>/dev/null | grep -iE "xid|nvrm|efa|ib_core|rdma|correctable|uncorrectable|acs|iommu" \
  1819. | tail -20 || echo "")
  1820. if [ -n "$DMESG" ]; then
  1821. echo "$DMESG"
  1822. else
  1823. echo "[PASS] No hardware errors in dmesg"
  1824. fi
  1825. # iptables / nftables — host-level firewall rules that block NCCL
  1826. echo "--- Host Firewall (iptables/nftables) ---"
  1827. IPT_DROP=0
  1828. if command -v iptables &>/dev/null; then
  1829. IPT_DROP=$(iptables -L -n 2>/dev/null | grep -cE "DROP|REJECT" || echo 0)
  1830. if [ "$IPT_DROP" -gt 0 ] 2>/dev/null; then
  1831. echo "[WARN] iptables has $IPT_DROP DROP/REJECT rules — may block NCCL traffic"
  1832. iptables -L -n 2>/dev/null | grep -E "DROP|REJECT" | head -5
  1833. echo " Verify NCCL ports (29400-29500, RDMA) are not blocked"
  1834. else
  1835. echo "[PASS] iptables: no DROP/REJECT rules"
  1836. fi
  1837. elif command -v nft &>/dev/null; then
  1838. NFT_DROP=$(nft list ruleset 2>/dev/null | grep -cE "drop|reject" || echo 0)
  1839. if [ "$NFT_DROP" -gt 0 ] 2>/dev/null; then
  1840. echo "[WARN] nftables has $NFT_DROP drop/reject rules — may block NCCL traffic"
  1841. else
  1842. echo "[PASS] nftables: no drop/reject rules"
  1843. fi
  1844. else
  1845. echo "[INFO] iptables/nftables not found"
  1846. fi
  1847. echo ""
  1848. echo "--- Stale NCCL Shared Memory ---"
  1849. STALE_SHM=$(ls /dev/shm/nccl-* 2>/dev/null || echo "")
  1850. if [ -n "$STALE_SHM" ]; then
  1851. STALE_COUNT=$(echo "$STALE_SHM" | wc -l)
  1852. echo "[WARN] $STALE_COUNT stale NCCL shared memory file(s) found:"
  1853. echo "$STALE_SHM" | head -5
  1854. echo " From a previous training run — may cause 'file exists' errors"
  1855. else
  1856. echo "[PASS] No stale NCCL shared memory files"
  1857. fi
  1858. echo ""
  1859. # EFA Latency Check (fi_ping) — catches degraded EFA ports (straggler #1 cause)
  1860. echo "--- EFA Latency (fi_ping self-test) ---"
  1861. FI_PING_CMD=""
  1862. command -v fi_ping &>/dev/null && FI_PING_CMD="fi_ping"
  1863. [ -z "$FI_PING_CMD" ] && [ -f /opt/amazon/efa/bin/fi_ping ] && FI_PING_CMD="/opt/amazon/efa/bin/fi_ping"
  1864. if [ -n "$FI_PING_CMD" ]; then
  1865. # Self-ping on loopback — tests EFA stack without needing a second node
  1866. # A degraded EFA port shows high latency (>20us) even on self-ping
  1867. # Validate FI_PING_CMD is a known safe EFA binary path (not user-controlled)
  1868. if [[ ! "$FI_PING_CMD" =~ ^(/opt/amazon/efa/bin/fi_ping|fi_ping)$ ]]; then
  1869. echo "[SKIP] fi_ping path not recognised: $FI_PING_CMD"
  1870. else
  1871. # Try EFA provider first; if it succeeds, the result reflects EFA. If
  1872. # EFA isn't reachable on loopback (some kernels), fall back to TCP — but
  1873. # label it explicitly so a TCP latency isn't reported as if it were EFA.
  1874. # nosemgrep: ai.ai-best-practices.hooks-dns-exfiltration.hooks-dns-exfiltration.hooks-dns-exfiltration-generic -- FI_PING_CMD validated to known EFA binary path above; targets loopback 127.0.0.1
  1875. PING_OUT=$($FI_PING_CMD -p efa -I 10 127.0.0.1 2>/dev/null || echo "")
  1876. PROVIDER="efa"
  1877. if [ -z "$PING_OUT" ]; then
  1878. # nosemgrep: ai.ai-best-practices.hooks-dns-exfiltration.hooks-dns-exfiltration.hooks-dns-exfiltration-generic -- FI_PING_CMD validated above; loopback only
  1879. PING_OUT=$($FI_PING_CMD -p tcp -I 10 127.0.0.1 2>/dev/null || echo "")
  1880. PROVIDER="tcp"
  1881. fi
  1882. if [ -n "$PING_OUT" ]; then
  1883. LATENCY=$(echo "$PING_OUT" | grep -oE "[0-9]+\.[0-9]+ us" | tail -1 || echo "")
  1884. LAT_VAL=$(echo "$LATENCY" | grep -oE "[0-9]+" | head -1 || echo 0)
  1885. if [ "$PROVIDER" = "tcp" ]; then
  1886. # TCP loopback latency does NOT reflect EFA path health; an EFA
  1887. # straggler will not be visible here. Surface as INFO, not PASS/WARN.
  1888. echo "[INFO] fi_ping fell back to provider=tcp (EFA loopback unreachable) — latency=${LATENCY:-?}; this does NOT measure EFA path health"
  1889. echo " For EFA latency, run fi_ping/fi_pingpong between two real nodes (not loopback) — see references/performance-testing.md"
  1890. elif [ -n "$LATENCY" ]; then
  1891. if [ "$LAT_VAL" -gt 20 ] 2>/dev/null; then
  1892. echo "[WARN] fi_ping latency (provider=efa): $LATENCY (>20us — EFA port may be degraded; normal is 1-5us)"
  1893. echo " Impact: straggler AllReduce, training much slower than expected"
  1894. echo " Action: drain this node and replace via HyperPod API"
  1895. else
  1896. echo "[PASS] fi_ping latency (provider=efa): $LATENCY"
  1897. fi
  1898. else
  1899. echo "[INFO] fi_ping (provider=efa) ran but no latency value extracted"
  1900. echo "$PING_OUT" | tail -3
  1901. fi
  1902. else
  1903. echo "[INFO] fi_ping self-test skipped (no EFA/TCP provider reachable)"
  1904. fi
  1905. fi
  1906. else
  1907. echo "[INFO] fi_ping not found (install EFA tools for latency testing)"
  1908. fi
  1909. echo ""
  1910. echo "=== END NODE DIAGNOSTICS ==="
  1911. NODE_SCRIPT
  1912. # Strategy for 100s of nodes:
  1913. # 1. Resolve all Running compute nodes via HyperPod API (paginated)
  1914. # 2. Sample --sample-nodes (default 3) for SSM hardware checks
  1915. # 3. Each SSM call has a 60s timeout
  1916. # 4. Results show per-node summary; failures are highlighted
  1917. # 5. This check does NOT increment ISSUES_FOUND (hardware checks are advisory)
  1918. # unless a critical hardware fault is detected (XID errors, EFA fail on GPU instance)
  1919. check_node_hardware_via_ssm() {
  1920. header "Check 8: Node Hardware Checks (via SSM — runs ON cluster nodes)"
  1921. info "Resolving cluster nodes for SSM..."
  1922. if ! resolve_cluster_nodes_for_ssm; then
  1923. info "Could not resolve cluster nodes via HyperPod API"
  1924. info " (DescribeCluster needs sagemaker:DescribeCluster on this cluster)"
  1925. info " To check a specific node: --node <INSTANCE_ID>"
  1926. return
  1927. fi
  1928. if [[ ${#SSM_NODES[@]} -eq 0 ]]; then
  1929. info "No Running compute nodes found in cluster"
  1930. return
  1931. fi
  1932. local total_nodes
  1933. total_nodes="${#SSM_NODES[@]}"
  1934. info "Sampling $total_nodes node(s) for hardware checks (use --sample-nodes N for more)"
  1935. info "Cluster ID: $SSM_CLUSTER_ID"
  1936. local node_pass=0 node_warn=0 node_fail=0
  1937. for entry in "${SSM_NODES[@]}"; do
  1938. local instance_id group_name
  1939. instance_id=$(echo "$entry" | awk '{print $1}')
  1940. group_name=$(echo "$entry" | awk '{print $2}')
  1941. local target="sagemaker-cluster:${SSM_CLUSTER_ID}_${group_name}-${instance_id}"
  1942. section "Node: $instance_id ($group_name)"
  1943. info " SSM target: $target"
  1944. info " Connecting (timeout 60s)..."
  1945. local output
  1946. output=$(_ssm_run "$instance_id" "$group_name" "$SSM_CLUSTER_ID" "$_NODE_DIAG_SCRIPT")
  1947. # Detect SSM transport failures. Letting error text fall through as
  1948. # diagnostic output produces a misleading "0 [PASS]" finding.
  1949. if [[ -z "$output" ]] || echo "$output" | grep -qiE "SessionManagerPlugin|error.*session|not authorized|AccessDenied|Could not connect|^Cannot perform start session|EOF$|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable|TargetNotConnected"; then
  1950. warn " SSM connection failed for $instance_id → references/operations.md § 3 SSM target format (HyperPod)"
  1951. node_warn=$((node_warn + 1))
  1952. continue
  1953. fi
  1954. echo "$output"
  1955. local passes fails
  1956. passes=$(echo "$output" | { grep -c "\[PASS\]" 2>/dev/null; true; })
  1957. fails=$(echo "$output" | { grep -c "\[FAIL\]" 2>/dev/null; true; })
  1958. # Non-GPU / non-EFA nodes (controllers, logins, CPU families) sampled
  1959. # as a fallback. Flag as SKIP rather than PASS — a PASS on a node
  1960. # without GPU/EFA is meaningless for NCCL.
  1961. local is_non_gpu=false
  1962. if echo "$output" | grep -qE "^\[INFO\].*(CPU-only node|non-EFA instance|no GPU devices visible|nvidia-smi not found)"; then
  1963. if ! echo "$output" | grep -qE "^\[PASS\] EFA provider available|^\[PASS\] GPU row-remap"; then
  1964. is_non_gpu=true
  1965. fi
  1966. fi
  1967. if [[ "$fails" -gt 0 ]]; then
  1968. error " Node $instance_id: $fails hardware issue(s) detected — see above"
  1969. node_fail=$((node_fail + 1))
  1970. # XID errors or EFA fail on a GPU instance = cluster-level issue
  1971. if echo "$output" | grep -q "\[FAIL\] GPU XID"; then
  1972. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  1973. add_issue_detail "XID errors on GPU hardware ($instance_id) → references/operations.md § 8 NCCL-specific remediations (Node reboot / replacement); hyperpod-node-debugger skill" "P0"
  1974. elif echo "$output" | grep -q "\[FAIL\] EFA provider NOT"; then
  1975. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  1976. add_issue_detail "EFA provider failure on $instance_id → references/debugging-guide.md § 6 EFA Configuration / § 13 EFA TCP Fallback" "P0"
  1977. fi
  1978. elif $is_non_gpu; then
  1979. info " Node $instance_id: no GPU/EFA present — skipping (NCCL checks apply only to GPU/EFA compute nodes)"
  1980. node_warn=$((node_warn + 1))
  1981. else
  1982. success " Node $instance_id: hardware checks passed ($passes [PASS])"
  1983. node_pass=$((node_pass + 1))
  1984. fi
  1985. done
  1986. echo ""
  1987. info "Hardware check summary: $node_pass PASS | $node_warn UNREACHABLE | $node_fail FAIL"
  1988. if [[ "$node_fail" -gt 0 ]]; then
  1989. warn " $node_fail node(s) have hardware issues — check above for details"
  1990. warn " For ALL nodes: re-run with --sample-nodes <total> to check every node"
  1991. fi
  1992. if [[ "$node_warn" -gt 0 ]]; then
  1993. warn " $node_warn node(s) unreachable via SSM — verify SSM agent and IAM permissions"
  1994. fi
  1995. }
  1996. # CloudWatch covers ALL nodes at once without per-node SSM calls.
  1997. # This runs for EKS when K8S_CONNECTED=false (can't use kubectl logs).
  1998. check_cloudwatch_nccl_logs() {
  1999. header "Check 6b: NCCL Pattern Analysis via CloudWatch"
  2000. local cluster_arn cluster_id
  2001. cluster_arn=$(aws sagemaker describe-cluster \
  2002. --cluster-name "$CLUSTER_NAME" --region "$REGION" \
  2003. --query 'ClusterArn' --output text 2>/dev/null || echo "")
  2004. cluster_id=$(echo "$cluster_arn" | awk -F'/' '{print $NF}')
  2005. if [[ -z "$cluster_id" || "$cluster_id" == "None" ]]; then
  2006. info "Cluster ID unavailable — skipping CloudWatch log analysis"
  2007. return
  2008. fi
  2009. local log_group="/aws/sagemaker/Clusters/${CLUSTER_NAME}/${cluster_id}"
  2010. info "CloudWatch log group: $log_group"
  2011. local lg_exists
  2012. lg_exists=$(aws logs describe-log-groups \
  2013. --log-group-name-prefix "$log_group" --region "$REGION" \
  2014. --query 'logGroups[0].logGroupName' --output text 2>&1) || {
  2015. if echo "$lg_exists" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
  2016. warn "Permission denied: logs:DescribeLogGroups — check IAM policy"
  2017. fi
  2018. lg_exists=""
  2019. }
  2020. if [[ -z "$lg_exists" || "$lg_exists" == "None" ]]; then
  2021. info "CloudWatch log group not found — CloudWatch agent may not be configured"
  2022. info " Enable the CloudWatch agent in the cluster's lifecycle script (see operations.md § 4)"
  2023. return
  2024. fi
  2025. local start_time=$(( ($(date +%s) - 7200) * 1000 ))
  2026. local patterns=(
  2027. "NCCL WARN" "Watchdog timeout" "Timeout waiting for"
  2028. "fi_getinfo failed" "unhandled system error" "nccl error"
  2029. "Connection refused" "NCCL_OFI_RDMA"
  2030. )
  2031. local found_any=false
  2032. for pattern in "${patterns[@]}"; do
  2033. local matches
  2034. matches=$(aws logs filter-log-events \
  2035. --log-group-name "$log_group" \
  2036. --filter-pattern "\"$pattern\"" \
  2037. --start-time "$start_time" \
  2038. --region "$REGION" \
  2039. --query 'events[*].{t:timestamp,s:logStreamName,m:message}' \
  2040. --output json 2>/dev/null || echo "[]")
  2041. local count
  2042. count=$(echo "$matches" | python3 -c \
  2043. "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0)
  2044. if [[ "$count" -gt 0 ]]; then
  2045. error "CloudWatch: '$pattern' found $count time(s) in last 2h:"
  2046. echo "$matches" | python3 -c "
  2047. import sys,json,datetime
  2048. events=json.load(sys.stdin)[:5]
  2049. for e in events:
  2050. ts=datetime.datetime.utcfromtimestamp(e['t']//1000).strftime('%H:%M:%S')
  2051. stream=e['s'][:30]
  2052. msg=e['m'][:120].strip()
  2053. print(f' [{ts}] {stream}: {msg}')
  2054. " 2>/dev/null
  2055. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  2056. add_issue_detail "CloudWatch pattern '$pattern' found ${count} time(s) → references/error-patterns-quick-ref.md" "P1"
  2057. found_any=true
  2058. fi
  2059. done
  2060. $found_any || success "No NCCL error patterns in CloudWatch logs (last 2h)"
  2061. }
  2062. # Slurm: run command on head node via SSM (start-session, not send-command)
  2063. run_slurm_cmd_via_ssm() {
  2064. local cmd="$1"
  2065. if ! resolve_cluster_nodes_for_ssm; then
  2066. return 1
  2067. fi
  2068. # Paginate list-cluster-nodes so controller/head nodes in the last page
  2069. # of a large cluster aren't missed.
  2070. local all_nodes
  2071. all_nodes=$(sagemaker_list_paginated list-cluster-nodes ClusterNodeSummaries)
  2072. local head_entry
  2073. head_entry=$(echo "$all_nodes" | python3 -c "
  2074. import sys,json
  2075. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  2076. for n in nodes:
  2077. g=n.get('InstanceGroupName','').lower()
  2078. if any(x in g for x in ['controller','head','master']):
  2079. print(n['InstanceId'] + ' ' + n['InstanceGroupName'])
  2080. break
  2081. else:
  2082. for n in nodes:
  2083. if n.get('InstanceStatus',{}).get('Status') == 'Running':
  2084. print(n['InstanceId'] + ' ' + n['InstanceGroupName'])
  2085. break
  2086. " 2>/dev/null || echo "")
  2087. [[ -z "$head_entry" ]] && return 1
  2088. local iid grp
  2089. iid=$(echo "$head_entry" | awk '{print $1}')
  2090. grp=$(echo "$head_entry" | awk '{print $2}')
  2091. _ssm_run "$iid" "$grp" "$SSM_CLUSTER_ID" "$cmd"
  2092. }
  2093. check_slurm_nodes() {
  2094. header "Check 2 [Slurm]: Node States"
  2095. local sinfo_output=""
  2096. if command -v sinfo &>/dev/null; then
  2097. sinfo_output=$(sinfo -o "%N %T %30E" --noheader 2>/dev/null || echo "")
  2098. else
  2099. sinfo_output=$(run_slurm_cmd_via_ssm "sinfo -o '%N %T %30E' --noheader" || echo "")
  2100. fi
  2101. # Treat SSM transport errors as retrieval failures, not as healthy state.
  2102. # Without this, "Cannot perform start session: EOF" is non-empty and falls
  2103. # through the empty-check below → grep finds no "down" → misleading [PASS].
  2104. if echo "$sinfo_output" | grep -qiE "^(Cannot perform start session|SessionManagerPlugin is not found)|EOF$|TargetNotConnected|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable|AccessDenied|UnauthorizedOperation|not authorized to perform"; then
  2105. warn "Could not retrieve Slurm node states — SSM transient error after retries"
  2106. info " Rerun the diagnostic; if persistent, delegate to hyperpod-ssm skill for manual probe."
  2107. return
  2108. fi
  2109. if [[ -z "$sinfo_output" ]]; then
  2110. warn "Could not retrieve Slurm node states"
  2111. return
  2112. fi
  2113. local down drained
  2114. down=$(echo "$sinfo_output" | grep -E "\bdown\b|\bdraining\b" | awk '{print $1}' || echo "")
  2115. drained=$(echo "$sinfo_output" | grep -E "\bdrained\b" | awk '{print $1}' || echo "")
  2116. if [[ -z "$down" && -z "$drained" ]]; then
  2117. success "All Slurm nodes: UP/IDLE/ALLOC — no NCCL-impacting states"
  2118. else
  2119. if [[ -n "$down" ]]; then
  2120. error "DOWN/DRAINING nodes: $down"
  2121. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  2122. add_issue_detail "Slurm nodes DOWN/DRAINING: $down → references/operations.md § 7 Slurm — NCCL-specific operations" "P1"
  2123. while IFS= read -r node; do
  2124. [[ -z "$node" ]] && continue
  2125. done <<< "$(echo "$down" | tr ',' '\n')"
  2126. fi
  2127. [[ -n "$drained" ]] && warn "DRAINED nodes (not available): $drained"
  2128. fi
  2129. section "Slurm Job Queue"
  2130. local q=""
  2131. if command -v squeue &>/dev/null; then
  2132. q=$(squeue -o "%i %j %T %R %N" --noheader 2>/dev/null || echo "")
  2133. fi
  2134. if [[ -z "$q" ]]; then
  2135. q=$(run_slurm_cmd_via_ssm "squeue -o '%i %j %T %R %N' --noheader" 2>/dev/null || echo "")
  2136. fi
  2137. # Same SSM-error detection as above — without this, the error string is
  2138. # parsed as a job list and produces false "stuck" rows.
  2139. if echo "$q" | grep -qiE "^(Cannot perform start session|SessionManagerPlugin is not found)|EOF$|TargetNotConnected|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable|AccessDenied|UnauthorizedOperation|not authorized to perform"; then
  2140. warn "Could not retrieve Slurm job queue — SSM transient error after retries"
  2141. q=""
  2142. fi
  2143. if [[ -z "$q" ]]; then
  2144. info "No jobs in queue"
  2145. else
  2146. local stuck
  2147. stuck=$(echo "$q" | grep -E "PENDING|COMPLETING" | head -10 || echo "")
  2148. if [[ -n "$stuck" ]]; then
  2149. warn "Stuck PENDING/COMPLETING jobs:"
  2150. echo "$stuck"
  2151. ISSUES_FOUND=$((ISSUES_FOUND+1))
  2152. add_issue_detail "Stuck PENDING/COMPLETING Slurm jobs → references/operations.md § 7 Slurm — NCCL-specific operations" "P1"
  2153. else
  2154. success "No stuck jobs in queue"
  2155. fi
  2156. info "Queue (top 10):"; echo "$q" | head -10
  2157. fi
  2158. }
  2159. check_slurm_nccl_logs() {
  2160. header "Check 6 [Slurm]: NCCL Log Pattern Analysis"
  2161. check_cloudwatch_nccl_logs
  2162. }
  2163. check_slurm_nccl_env() {
  2164. header "Check 7 [Slurm]: NCCL Environment Variable Audit (via SSM)"
  2165. local env_check
  2166. env_check=$(run_slurm_cmd_via_ssm \
  2167. "{ cat /etc/profile.d/nccl.sh /opt/ml/config/nccl.conf /etc/slurm/prolog.d/*.sh 2>/dev/null; env; } \
  2168. | grep -E '^(NCCL_|FI_|MASTER_)' | sort -u | head -30 || echo '(none)'" \
  2169. 2>/dev/null || echo "")
  2170. # If SSM returned a transport error, don't interpret it as the controller's
  2171. # env output — that produces false "FI_PROVIDER=efa not set" warnings.
  2172. if echo "$env_check" | grep -qiE "^(Cannot perform start session|SessionManagerPlugin is not found)|EOF$|TargetNotConnected|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable|AccessDenied|UnauthorizedOperation|not authorized to perform"; then
  2173. warn "Could not retrieve NCCL env vars from controller — SSM transient error after retries"
  2174. info " Rerun the diagnostic; if persistent, delegate to hyperpod-ssm skill."
  2175. return
  2176. fi
  2177. if [[ -n "$env_check" && "$env_check" != "(none)" ]]; then
  2178. info "NCCL/EFA env vars on head node:"
  2179. echo "$env_check" | while IFS= read -r line; do info " $line"; done
  2180. local warn_count=0
  2181. if echo "$env_check" | grep -q "NCCL_DEBUG=INFO"; then
  2182. warn "NCCL_DEBUG=INFO detected — verbose logging adds runtime overhead. Set NCCL_DEBUG=WARN for production."
  2183. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  2184. add_issue_detail "NCCL_DEBUG=INFO in Slurm env (set NCCL_DEBUG=WARN in production) → references/operations.md § 5 NCCL environment variable reference" "P1"
  2185. warn_count=$((warn_count + 1))
  2186. fi
  2187. if echo "$env_check" | grep -q "NCCL_DEBUG=TRACE"; then
  2188. warn "NCCL_DEBUG=TRACE detected — TRACE prints replayable trace info on every NCCL call (large overhead and verbose logs). Set NCCL_DEBUG=WARN immediately."
  2189. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  2190. add_issue_detail "NCCL_DEBUG=TRACE in Slurm env (set NCCL_DEBUG=WARN immediately) → references/operations.md § 5 NCCL environment variable reference" "P0"
  2191. warn_count=$((warn_count + 1))
  2192. fi
  2193. if ! echo "$env_check" | grep -q "FI_PROVIDER=efa"; then
  2194. warn "FI_PROVIDER=efa not set — EFA may not be used for NCCL transport"
  2195. warn_count=$((warn_count + 1))
  2196. fi
  2197. if ! echo "$env_check" | grep -q "NCCL_SOCKET_IFNAME"; then
  2198. warn "NCCL_SOCKET_IFNAME not set — NCCL may pick wrong interface. Recommend: ^lo,docker,efa,veth"
  2199. warn_count=$((warn_count + 1))
  2200. fi
  2201. if [[ "$warn_count" -eq 0 ]]; then
  2202. success "System-level NCCL env vars look correct"
  2203. fi
  2204. else
  2205. info "No NCCL env vars found in system config on head node"
  2206. info " (Expected — NCCL vars are typically set in job scripts, not system-wide)"
  2207. fi
  2208. }
  2209. check_slurm_controller_health() {
  2210. # Slurm controller health — retry up to 3× before declaring it down, because
  2211. # SSM cold-start / session-service EOF errors are common on the first call.
  2212. header "Check 0 [Slurm]: Controller Health"
  2213. local ping_result=""
  2214. for _ in 1 2 3; do
  2215. ping_result=$(run_slurm_cmd_via_ssm "scontrol ping 2>/dev/null" || echo "")
  2216. [[ -n "$ping_result" ]] && echo "$ping_result" | grep -qi "is UP\|slurmctld.*UP" && break
  2217. sleep 3
  2218. done
  2219. if echo "$ping_result" | grep -qi "is UP\|slurmctld.*UP"; then
  2220. success "slurmctld is responsive"
  2221. elif echo "$ping_result" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized to perform"; then
  2222. # IAM denial ≠ Slurm failure. Reporting "slurmctld down" would be wrong
  2223. # and would send the customer down a Slurm-rescue path for an IAM issue.
  2224. warn "Could not check slurmctld — caller lacks ssm:StartSession on this cluster"
  2225. info " Grant ssm:StartSession on the HyperPod cluster ARN and rerun."
  2226. elif echo "$ping_result" | grep -qiE "Cannot perform start session|SessionManager|EOF$|TargetNotConnected|ConnectTimeout|ServiceError|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable"; then
  2227. # Transport-level SSM errors — not a Slurm failure. Downgrade to WARN.
  2228. warn "Could not reach controller via SSM (transient): $(echo "$ping_result" | head -1)"
  2229. info " Rerun the diagnostic; if the error persists, delegate to hyperpod-ssm skill."
  2230. elif [[ -n "$ping_result" ]]; then
  2231. error "slurmctld not responding — all Slurm operations blocked"
  2232. local _diag_line
  2233. _diag_line="$(echo "$ping_result" | head -1)"
  2234. info " Controller response: $_diag_line"
  2235. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  2236. add_issue_detail "slurmctld down on controller → references/operations.md § 7 Slurm — NCCL-specific operations" "P0"
  2237. else
  2238. info "Could not reach controller via SSM — slurmctld status unknown"
  2239. fi
  2240. local munge_result
  2241. munge_result=$(run_slurm_cmd_via_ssm "systemctl is-active munge 2>/dev/null || echo munge_inactive" || echo "")
  2242. if echo "$munge_result" | grep -q "^active"; then
  2243. success "munge authentication service active"
  2244. elif echo "$munge_result" | grep -q "munge_inactive"; then
  2245. error "munge service inactive — Slurm auth will fail"
  2246. ISSUES_FOUND=$((ISSUES_FOUND + 1))
  2247. add_issue_detail "munge service inactive → references/operations.md § 7 Slurm — NCCL-specific operations" "P0"
  2248. fi
  2249. }
  2250. run_slurm_checks() {
  2251. check_slurm_controller_health
  2252. check_cluster_health
  2253. check_slurm_nodes
  2254. check_cluster_events
  2255. check_security_groups
  2256. check_slurm_nccl_logs
  2257. check_slurm_nccl_env
  2258. check_node_hardware_via_ssm
  2259. }
  2260. print_summary() {
  2261. header "NCCL Diagnostic Summary"
  2262. echo ""
  2263. echo -e " Cluster: ${BOLD}$CLUSTER_NAME${RESET}"
  2264. echo -e " Region: ${BOLD}$REGION${RESET}"
  2265. echo -e " Orchestrator: ${BOLD}$(printf '%s' "$ORCHESTRATOR" | tr '[:lower:]' '[:upper:]')${RESET}"
  2266. [[ "$ORCHESTRATOR" == "eks" ]] && \
  2267. echo -e " Namespace: ${BOLD}${NAMESPACE:-all}${RESET}"
  2268. [[ -n "$JOB_NAME" ]] && echo -e " Job: ${BOLD}$JOB_NAME${RESET}"
  2269. [[ -n "$NODE_ID" ]] && echo -e " Node: ${BOLD}$NODE_ID${RESET}"
  2270. echo -e " Mode: ${BOLD}READ-ONLY${RESET} (no changes applied)"
  2271. echo ""
  2272. echo -e " ┌──────────────────────────────────┐"
  2273. echo -e " │ Issues Found: ${RED}${BOLD}$ISSUES_FOUND${RESET} │"
  2274. echo -e " └──────────────────────────────────┘"
  2275. if [[ ${#ISSUE_DETAILS[@]} -gt 0 ]]; then
  2276. echo ""
  2277. echo " Issue Details (prioritized):"
  2278. for priority in P0 P1 P2; do
  2279. local has_items=false
  2280. for detail in "${ISSUE_DETAILS[@]}"; do
  2281. if [[ "$detail" == "${priority}|"* ]]; then
  2282. if ! $has_items; then
  2283. case "$priority" in
  2284. P0) echo -e " ${RED}${BOLD}[$priority — Fix Immediately]${RESET}" ;;
  2285. P1) echo -e " ${YELLOW}${BOLD}[$priority — Fix Soon]${RESET}" ;;
  2286. P2) echo -e " ${BOLD}[$priority — Advisory]${RESET}" ;;
  2287. esac
  2288. has_items=true
  2289. fi
  2290. echo " → ${detail#*|}"
  2291. fi
  2292. done
  2293. done
  2294. fi
  2295. echo ""
  2296. if [[ "$ISSUES_FOUND" -eq 0 ]]; then
  2297. success "No actionable NCCL issues detected — cluster looks healthy"
  2298. echo ""
  2299. info "If training is still hanging, check:"
  2300. echo " 1. CloudWatch: aws logs filter-log-events --log-group-name /aws/sagemaker/Clusters/$CLUSTER_NAME/..."
  2301. echo " 2. Version check: hyperpod-version-checker skill"
  2302. echo " 3. Full diagnostics: hyperpod-issue-report skill"
  2303. else
  2304. warn "$ISSUES_FOUND issue(s) found — see the Issue Details list above."
  2305. warn "Each issue line includes a reference pointer (→ references/<file>.md § <section>)."
  2306. warn "The hyperpod-nccl skill will read these findings, look up the matching section,"
  2307. warn "and guide you through remediation. This script does not modify cluster state."
  2308. fi
  2309. echo ""
  2310. echo -e "${BOLD}References:${RESET}"
  2311. echo " Debugging guide: references/debugging-guide.md"
  2312. echo " Operations: references/operations.md"
  2313. echo " Performance test: references/performance-testing.md"
  2314. echo ""
  2315. }
  2316. main() {
  2317. header "NCCL Diagnostic — SageMaker HyperPod (read-only)"
  2318. detect_orchestrator
  2319. echo -e " Cluster: ${BOLD}$CLUSTER_NAME${RESET}"
  2320. echo -e " Region: ${BOLD}$REGION${RESET}"
  2321. echo -e " Orchestrator: ${BOLD}$(printf '%s' "$ORCHESTRATOR" | tr '[:lower:]' '[:upper:]')${RESET}"
  2322. [[ "$ORCHESTRATOR" == "eks" ]] && echo -e " Namespace: ${BOLD}${NAMESPACE:-all}${RESET}"
  2323. info "READ-ONLY DIAGNOSTIC — no cluster state will be modified."
  2324. info "This script collects signals only. The hyperpod-nccl skill interprets findings"
  2325. info "and looks up remediation in references/*.md."
  2326. echo ""
  2327. check_prerequisites
  2328. if [[ "$ORCHESTRATOR" == "slurm" ]]; then
  2329. info "Running Slurm NCCL diagnostics..."
  2330. run_slurm_checks
  2331. else
  2332. info "Running EKS NCCL diagnostics..."
  2333. check_cluster_health
  2334. check_cluster_events
  2335. check_security_groups
  2336. if $K8S_CONNECTED; then
  2337. check_k8s_nodes
  2338. check_efa_k8s
  2339. check_pod_status
  2340. check_nccl_infra_prereqs
  2341. analyze_nccl_logs
  2342. check_nccl_env_vars
  2343. check_network_policies
  2344. else
  2345. warn "K8s checks skipped (2, 2b, 5, 5b, 6, 7, 9) — kubectl not authenticated"
  2346. # CloudWatch analysis doesn't need kubectl.
  2347. check_cloudwatch_nccl_logs
  2348. fi
  2349. check_node_hardware_via_ssm
  2350. fi
  2351. print_summary
  2352. # Exit 1 only on P0/P1 findings; P2 are informational.
  2353. local _critical=0
  2354. for _issue in "${ISSUE_DETAILS[@]:-}"; do
  2355. [[ -z "$_issue" ]] && continue
  2356. case "${_issue%%|*}" in P0|P1) _critical=$((_critical+1)) ;; esac
  2357. done
  2358. [[ "$_critical" -eq 0 ]] && exit 0 || exit 1
  2359. }
  2360. main "$@"
plugins/sagemaker-ai/skills/hyperpod-node-debugger/scripts/check-node-reachability.sh · Bundled script · 389 lines · 1 finding marked
  1. #!/usr/bin/env bash
  2. # check-node-reachability.sh
  3. #
  4. # Diagnose EFA reachability and inter-node communication health on a single
  5. # HyperPod node. Run this on each node via the hyperpod-ssm skill.
  6. #
  7. # Usage (via ssm-exec.sh):
  8. # ssm-exec.sh --target <TARGET> --upload scripts/check-node-reachability.sh /tmp/check-node-reachability.sh
  9. # ssm-exec.sh --target <TARGET> 'bash /tmp/check-node-reachability.sh'
  10. #
  11. # Usage (direct on node):
  12. # bash check-node-reachability.sh [--json] [--no-color]
  13. #
  14. # Exit codes:
  15. # 0 — all critical checks passed
  16. # 1 — one or more critical checks failed
  17. set -euo pipefail
  18. # Note: this script runs ON the node (via SSM), so aws CLI may not be present.
  19. # Only python3 is checked here; other tools are checked individually per section.
  20. JSON_MODE=false
  21. USE_COLOR=true
  22. usage() {
  23. cat <<EOF
  24. Usage: bash check-node-reachability.sh [--json] [--no-color]
  25. Read-only on-node diagnostic for EFA reachability and inter-node communication
  26. health. Must be executed on a HyperPod compute node (typically via the
  27. hyperpod-ssm skill). Checks EFA interfaces, /dev/infiniband devices, GPU
  28. count and Neuron device count against the expected counts for the node's
  29. instance type.
  30. Options:
  31. --json Emit findings as JSON instead of human-readable output.
  32. --no-color Disable ANSI colors.
  33. -h, --help Show this message.
  34. Exit codes:
  35. 0 All critical checks passed.
  36. 1 One or more critical checks failed.
  37. EOF
  38. }
  39. while [[ $# -gt 0 ]]; do
  40. case "$1" in
  41. --json) JSON_MODE=true; shift ;;
  42. --no-color) USE_COLOR=false; shift ;;
  43. -h|--help) usage; exit 0 ;;
  44. *) echo "Unknown argument: $1" >&2; usage >&2; exit 1 ;;
  45. esac
  46. done
  47. # Colors — auto-disable when stdout isn't a TTY.
  48. if ! [ -t 1 ] || [ "${TERM:-}" = "dumb" ]; then
  49. USE_COLOR=false
  50. fi
  51. if "$USE_COLOR"; then
  52. RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
  53. BOLD='\033[1m'; NC='\033[0m'
  54. else
  55. RED=''; GREEN=''; YELLOW=''; BOLD=''; NC=''
  56. fi
  57. HOSTNAME=$(hostname 2>/dev/null || echo "unknown")
  58. TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
  59. CRITICAL_FAILURES=0
  60. declare -A RESULTS # associative array: check_name → pass|fail|warn|skip
  61. pass() { RESULTS["$1"]="pass"; [[ "$JSON_MODE" == false ]] && echo -e " ${GREEN}[PASS]${NC} $1${2:+ — $2}"; }
  62. fail() { RESULTS["$1"]="fail"; CRITICAL_FAILURES=$((CRITICAL_FAILURES+1)); \
  63. [[ "$JSON_MODE" == false ]] && echo -e " ${RED}[FAIL]${NC} $1${2:+ — $2}"; }
  64. warn() { RESULTS["$1"]="warn"; [[ "$JSON_MODE" == false ]] && echo -e " ${YELLOW}[WARN]${NC} $1${2:+ — $2}"; }
  65. skip() { RESULTS["$1"]="skip"; [[ "$JSON_MODE" == false ]] && echo -e " [SKIP] $1${2:+ — $2}"; }
  66. info() { [[ "$JSON_MODE" == false ]] && echo -e " $1"; }
  67. if [[ "$JSON_MODE" == false ]]; then
  68. echo ""
  69. echo -e "${BOLD}=== HyperPod Node EFA Reachability Check ===${NC}"
  70. echo -e "Host: ${BOLD}${HOSTNAME}${NC}"
  71. echo -e "Timestamp: ${TIMESTAMP}"
  72. echo ""
  73. fi
  74. if [[ "$JSON_MODE" == false ]]; then echo -e "${BOLD}--- EFA Kernel Module ---${NC}"; fi
  75. EFA_MODULE=$(lsmod 2>/dev/null | grep -E '^efa\b' | awk '{print $1}' || true)
  76. if [[ -n "$EFA_MODULE" ]]; then
  77. EFA_MODULE_VER=$(modinfo efa 2>/dev/null | grep -E '^version:' | awk '{print $2}' || echo "unknown")
  78. pass "efa_kernel_module" "loaded (version: ${EFA_MODULE_VER})"
  79. else
  80. # Read-only invariant: detect only, never `sudo modprobe efa` — loading kernel
  81. # modules mutates node state, which the hyperpod-ssm skill's approval flow owns.
  82. fail "efa_kernel_module" "not loaded — see references/node-diagnostics-detail.md § A (EFA / Security Group)"
  83. fi
  84. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- EFA Devices ---${NC}"; fi
  85. # shellcheck disable=SC2010 # /dev/ entries are kernel-named, safe to ls|grep
  86. EFA_DEVICES=$(ls /dev/infiniband/ 2>/dev/null | grep -E 'rdma_cm|uverbs|efa' || true)
  87. if [[ -n "$EFA_DEVICES" ]]; then
  88. pass "efa_devices_present" "found in /dev/infiniband/: $(echo "$EFA_DEVICES" | tr '\n' ' ')"
  89. else
  90. fail "efa_devices_present" "/dev/infiniband/ is empty or missing — EFA hardware not detected"
  91. fi
  92. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- libfabric EFA Provider ---${NC}"; fi
  93. if command -v fi_info &>/dev/null; then
  94. # If the previous section found no EFA hardware, fi_info failing is expected —
  95. # don't emit [FAIL] on top of the hardware [FAIL], which would double-count and
  96. # conflate "libfabric can't see EFA" with "node has no EFA at all".
  97. if [[ -z "$EFA_DEVICES" ]]; then
  98. skip "fi_info_efa_provider" "no EFA devices detected upstream — see efa_devices_present"
  99. else
  100. FI_EXIT=0
  101. FI_OUTPUT=$(fi_info -p efa 2>&1) || FI_EXIT=$?
  102. if echo "$FI_OUTPUT" | grep -q "provider: efa"; then
  103. EFA_PROVIDER_COUNT=$(echo "$FI_OUTPUT" | { grep -c "provider: efa" 2>/dev/null; true; })
  104. pass "fi_info_efa_provider" "EFA provider found (${EFA_PROVIDER_COUNT} endpoint(s))"
  105. info "$(echo "$FI_OUTPUT" | grep -E 'provider:|fabric:|domain:|version:' | head -8 | sed 's/^/ /')"
  106. else
  107. fail "fi_info_efa_provider" "fi_info -p efa returned no EFA provider (exit code ${FI_EXIT}) — libfabric cannot enumerate EFA devices. See references/node-diagnostics-detail.md § A (EFA / Security Group)"
  108. info "fi_info output: ${FI_OUTPUT:0:200}"
  109. fi
  110. fi
  111. else
  112. warn "fi_info_efa_provider" "fi_info not found — install libfabric to run this check (fi_info comes with EFA installer)"
  113. fi
  114. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- EFA Network Interfaces ---${NC}"; fi
  115. # EFA interfaces typically appear as eth0/ens* for primary + rdmaX or efa* for EFA devices
  116. # EFA ifaces on p5/p5en use regular kernel names (ens*) — filter by driver via ethtool
  117. # rather than by name pattern (the old 'rdma|efa' name grep misses ens* on p5).
  118. EFA_IFACES=""
  119. if command -v ethtool &>/dev/null; then
  120. while IFS= read -r iface; do
  121. [[ -z "$iface" ]] && continue
  122. DRIVER=$(ethtool -i "$iface" 2>/dev/null | awk -F': ' '/^driver:/{print $2}')
  123. if [[ "$DRIVER" == "efa" ]]; then
  124. EFA_IFACES+="${iface}"$'\n'
  125. fi
  126. done < <(ip -o link show 2>/dev/null | awk -F': ' '{print $2}' | awk -F'@' '{print $1}' | grep -v '^lo$')
  127. fi
  128. # Fallback to name-based detection for older kernels / containers without ethtool
  129. if [[ -z "$EFA_IFACES" ]]; then
  130. EFA_IFACES=$(ip link show 2>/dev/null | grep -E 'rdma|efa' | awk -F': ' '{print $2}' | tr -d '@' || true)
  131. fi
  132. REGULAR_IFACES=$(ip link show 2>/dev/null | grep -E 'state UP' | awk -F': ' '{print $2}' | tr -d '@' || true)
  133. if [[ -n "$EFA_IFACES" ]]; then
  134. pass "efa_interfaces_up" "EFA interfaces found: $(echo "$EFA_IFACES" | tr '\n' ' ')"
  135. while IFS= read -r iface; do
  136. [[ -z "$iface" ]] && continue
  137. IP=$(ip addr show "$iface" 2>/dev/null | grep 'inet ' | awk '{print $2}' || true)
  138. if [[ -n "$IP" ]]; then
  139. info " $iface → $IP"
  140. else
  141. warn "efa_interface_ip_${iface}" "interface $iface has no IP address — check DHCP/subnet config"
  142. fi
  143. done <<< "$EFA_IFACES"
  144. else
  145. info "No EFA interfaces detected (by driver or name)"
  146. if [[ -n "$REGULAR_IFACES" ]]; then
  147. skip "efa_interfaces_up" "no separate EFA interface — primary interfaces: $(echo "$REGULAR_IFACES" | tr '\n' ' ' | head -c 80)"
  148. else
  149. warn "efa_interfaces_up" "no UP network interfaces found"
  150. fi
  151. fi
  152. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- EFA Installation ---${NC}"; fi
  153. EFA_VER_FILE="/opt/amazon/efa_installed_packages"
  154. if [[ -f "$EFA_VER_FILE" ]]; then
  155. # Format is "EFA installer version: 1.30.0" — grab only the version token.
  156. EFA_VER=$(grep -iE '^EFA installer version' "$EFA_VER_FILE" 2>/dev/null \
  157. | head -1 \
  158. | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?' \
  159. | head -1 || echo "")
  160. if [[ -z "$EFA_VER" ]]; then
  161. warn "efa_installer_present" "EFA installer file present but version line not parsed"
  162. else
  163. pass "efa_installer_present" "EFA installer version: ${EFA_VER}"
  164. fi
  165. else
  166. warn "efa_installer_present" "EFA installer marker not found at ${EFA_VER_FILE} — EFA may not be installed via standard method"
  167. fi
  168. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- NCCL / OFI Configuration ---${NC}"; fi
  169. NCCL_VARS=("FI_PROVIDER" "FI_EFA_USE_DEVICE_RDMA" "NCCL_SOCKET_IFNAME" "NCCL_ALGO" "LD_LIBRARY_PATH")
  170. ANY_NCCL_SET=false
  171. for var in "${NCCL_VARS[@]}"; do
  172. val="${!var:-}"
  173. if [[ -n "$val" ]]; then
  174. info " ${var}=${val}"
  175. ANY_NCCL_SET=true
  176. fi
  177. done
  178. if "$ANY_NCCL_SET"; then
  179. FI_PROVIDER_VAL="${FI_PROVIDER:-}"
  180. if [[ -n "$FI_PROVIDER_VAL" && "$FI_PROVIDER_VAL" != "efa" ]]; then
  181. warn "nccl_fi_provider" "FI_PROVIDER=${FI_PROVIDER_VAL} — for EFA workloads this should be 'efa'"
  182. elif [[ "$FI_PROVIDER_VAL" == "efa" ]]; then
  183. pass "nccl_fi_provider" "FI_PROVIDER=efa"
  184. fi
  185. else
  186. skip "nccl_env_vars" "no NCCL/OFI env vars set in current shell — may be set in job launcher environment"
  187. fi
  188. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- AWS OFI NCCL Plugin ---${NC}"; fi
  189. OFI_LIB=$(find /opt/amazon/efa /opt/aws-ofi-nccl /usr/local/lib /usr/lib \
  190. -name "libnccl-net.so*" -o -name "aws-ofi-nccl.so*" 2>/dev/null | head -1 || true)
  191. if [[ -n "$OFI_LIB" ]]; then
  192. pass "aws_ofi_nccl_plugin" "found: ${OFI_LIB}"
  193. else
  194. if [[ -f "$EFA_VER_FILE" ]] && grep -q "ofi\|OFI" "$EFA_VER_FILE" 2>/dev/null; then
  195. pass "aws_ofi_nccl_plugin" "referenced in ${EFA_VER_FILE}"
  196. else
  197. warn "aws_ofi_nccl_plugin" "libnccl-net.so not found — required for EFA-accelerated NCCL (distributed training)"
  198. fi
  199. fi
  200. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- Instance Metadata Reachability ---${NC}"; fi
  201. IMDS_TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
    • Note: Quoted as data in code: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
    • Line 233 on GitHub
  202. -H "X-aws-ec2-metadata-token-ttl-seconds: 60" --connect-timeout 3 -m 5 2>/dev/null || true)
  203. if [[ -n "$IMDS_TOKEN" ]]; then
  204. INSTANCE_TYPE=$(curl -s -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  205. http://169.254.169.254/latest/meta-data/instance-type --connect-timeout 3 -m 5 2>/dev/null || echo "unknown")
  206. LOCAL_IP=$(curl -s -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  207. http://169.254.169.254/latest/meta-data/local-ipv4 --connect-timeout 3 -m 5 2>/dev/null || echo "unknown")
  208. pass "imds_reachable" "instance-type=${INSTANCE_TYPE}, local-ipv4=${LOCAL_IP}"
  209. # Static list of EFA-capable families; unknown types fall through to the
  210. # EC2 API check. aws CLI may not be present on-node, so the static path
  211. # covers the common case.
  212. case "$INSTANCE_TYPE" in
  213. p4de*|p4d*|p5en*|p5e*|p5*|p6*|trn1*|trn2*|inf2*|g5.48xlarge|g6e.48xlarge|g6.48xlarge|hpc6a*|hpc6id*|hpc7a*|hpc7g*|dl1*|dl2q*)
  214. pass "efa_capable_instance" "${INSTANCE_TYPE} supports EFA" ;;
  215. *)
  216. if command -v aws &>/dev/null; then
  217. EFA_CHECK=$(aws ec2 describe-instance-types \
  218. --instance-types "${INSTANCE_TYPE}" \
  219. --query 'InstanceTypes[0].NetworkInfo.EfaSupported' \
  220. --output text 2>/dev/null || echo "unknown")
  221. if [[ "$EFA_CHECK" == "True" ]]; then
  222. pass "efa_capable_instance" "${INSTANCE_TYPE} supports EFA (verified via API)"
  223. elif [[ "$EFA_CHECK" == "False" ]]; then
  224. warn "efa_capable_instance" "${INSTANCE_TYPE} does NOT support EFA"
  225. else
  226. warn "efa_capable_instance" "${INSTANCE_TYPE} — could not verify EFA support"
  227. fi
  228. else
  229. warn "efa_capable_instance" "${INSTANCE_TYPE} — not in known EFA list; verify with: aws ec2 describe-instance-types --instance-types ${INSTANCE_TYPE} --query 'InstanceTypes[0].NetworkInfo.EfaSupported'"
  230. fi
  231. ;;
  232. esac
  233. # Multi-EFA validation — counts per EC2 instance-type documentation.
  234. # NOTE: EFA counts vary between instance families (p5en has fewer than p5/p5e).
  235. EXPECTED_EFA=0
  236. case "$INSTANCE_TYPE" in
  237. p5.48xlarge|p5e.48xlarge) EXPECTED_EFA=32 ;;
  238. p5en.48xlarge) EXPECTED_EFA=16 ;;
  239. p4d.24xlarge|p4de.24xlarge) EXPECTED_EFA=4 ;;
  240. trn1.32xlarge) EXPECTED_EFA=8 ;;
  241. trn2.48xlarge) EXPECTED_EFA=16 ;;
  242. # p6 family and newer: don't hardcode counts; discover via ethtool to avoid false FAILs.
  243. esac
  244. if [[ "$EXPECTED_EFA" -gt 0 ]]; then
  245. # Count actual EFA devices — avoid grep -c pattern that returns "0\n0" fallthrough.
  246. ACTUAL_EFA=$(find /dev/infiniband -maxdepth 1 -name 'uverbs*' 2>/dev/null | wc -l)
  247. [[ -z "$ACTUAL_EFA" ]] && ACTUAL_EFA=0
  248. if [[ "$ACTUAL_EFA" -ge "$EXPECTED_EFA" ]]; then
  249. pass "multi_efa_interfaces" "${ACTUAL_EFA}/${EXPECTED_EFA} EFA interfaces present for ${INSTANCE_TYPE}"
  250. elif [[ "$ACTUAL_EFA" -gt 0 ]]; then
  251. warn "multi_efa_interfaces" "only ${ACTUAL_EFA}/${EXPECTED_EFA} EFA interfaces — some may not be attached or driver issue"
  252. else
  253. fail "multi_efa_interfaces" "0/${EXPECTED_EFA} EFA interfaces on ${INSTANCE_TYPE} — EFA driver or attachment issue"
  254. fi
  255. fi
  256. else
  257. warn "imds_reachable" "IMDS not reachable. If running inside a container: check IMDSv2 HttpPutResponseHopLimit on the instance (default 1 is often too low for container networking — set to 2 or higher). Otherwise: verify the instance metadata service is enabled (HttpEndpoint != disabled) and that no local iptables / nftables rules block 169.254.169.254. Note: SGs do not filter link-local addresses."
  258. fi
  259. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- Network Interface Statistics ---${NC}"; fi
  260. if command -v ip &>/dev/null; then
  261. IFACE_ERRORS=$(ip -s link show 2>/dev/null | awk '
  262. BEGIN { rx_err=0; tx_err=0; iface="" }
  263. /^[0-9]+:/ {
  264. if (iface != "" && (rx_err > 0 || tx_err > 0))
  265. print " " iface ": RX errors=" rx_err " TX errors=" tx_err
  266. iface=$2; gsub(/:$/, "", iface)
  267. rx_err=0; tx_err=0
  268. }
  269. /RX:/ { getline; rx_err=$3+0 }
  270. /TX:/ { getline; tx_err=$3+0 }
  271. END {
  272. if (iface != "" && (rx_err > 0 || tx_err > 0))
  273. print " " iface ": RX errors=" rx_err " TX errors=" tx_err
  274. }
  275. ' || true)
  276. if [[ -n "$IFACE_ERRORS" ]]; then
  277. warn "network_interface_errors" "interfaces with errors detected:"
  278. info "$IFACE_ERRORS"
  279. else
  280. pass "network_interface_errors" "no RX/TX errors on active interfaces"
  281. fi
  282. else
  283. skip "network_interface_errors" "ip command not available"
  284. fi
  285. if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- Neuron Devices (Trainium/Inferentia) ---${NC}"; fi
  286. if command -v neuron-ls &>/dev/null; then
  287. NEURON_OUTPUT=$(neuron-ls 2>&1 || true)
  288. NEURON_DEVICE_COUNT=$(echo "$NEURON_OUTPUT" | { grep -c "neuron_device" 2>/dev/null; true; })
  289. if [[ "$NEURON_DEVICE_COUNT" -gt 0 ]]; then
  290. pass "neuron_devices" "${NEURON_DEVICE_COUNT} Neuron device(s) detected"
  291. info "$(echo "$NEURON_OUTPUT" | head -10 | sed 's/^/ /')"
  292. else
  293. NEURON_MOD=$(lsmod 2>/dev/null | grep -E '^neuron' || true)
  294. if [[ -n "$NEURON_MOD" ]]; then
  295. warn "neuron_devices" "Neuron driver loaded but neuron-ls shows 0 devices → references/node-diagnostics-detail.md § G.2 (Trainium/Inferentia)"
  296. else
  297. fail "neuron_devices" "Neuron driver not loaded → references/node-diagnostics-detail.md § G.2 (Trainium/Inferentia)"
  298. fi
  299. fi
  300. elif ls /dev/neuron* &>/dev/null 2>&1; then
  301. NEURON_DEV_COUNT=$(find /dev -maxdepth 1 -name 'neuron*' 2>/dev/null | wc -l)
  302. NEURON_DEV_COUNT=${NEURON_DEV_COUNT:-0}
  303. warn "neuron_devices" "${NEURON_DEV_COUNT} /dev/neuron* device(s) found but neuron-ls not installed → references/node-diagnostics-detail.md § G.2 (Trainium/Inferentia)"
  304. else
  305. skip "neuron_devices" "not a Trainium/Inferentia instance (no Neuron devices)"
  306. fi
  307. if [[ "$JSON_MODE" == false ]]; then
  308. echo ""
  309. echo -e "${BOLD}--- Summary ---${NC}"
  310. TOTAL=${#RESULTS[@]}
  311. PASSED=$(printf '%s\n' "${RESULTS[@]}" | { grep -c "^pass$" 2>/dev/null; true; })
  312. WARNED=$(printf '%s\n' "${RESULTS[@]}" | { grep -c "^warn$" 2>/dev/null; true; })
  313. FAILED=$(printf '%s\n' "${RESULTS[@]}" | { grep -c "^fail$" 2>/dev/null; true; })
  314. SKIPPED=$(printf '%s\n' "${RESULTS[@]}" | { grep -c "^skip$" 2>/dev/null; true; })
  315. echo -e " Host: ${HOSTNAME}"
  316. echo -e " Checks: ${TOTAL} total | ${GREEN}${PASSED} passed${NC} | ${YELLOW}${WARNED} warnings${NC} | ${RED}${FAILED} failed${NC} | ${SKIPPED} skipped"
  317. if [[ $CRITICAL_FAILURES -eq 0 ]]; then
  318. echo -e "\n ${GREEN}${BOLD}Node EFA reachability checks PASSED.${NC}"
  319. echo " If inter-node communication still fails, verify security group rules with check-efa-sg.sh"
  320. echo " and compare EFA versions across nodes with the hyperpod-version-checker skill."
  321. else
  322. echo -e "\n ${RED}${BOLD}Node EFA reachability checks FAILED (${CRITICAL_FAILURES} critical issue(s)).${NC}"
  323. echo " See [FAIL] items above. Each finding ends with a pointer of the form"
  324. echo " '→ references/node-diagnostics-detail.md § <section>' — open that section"
  325. echo " for root cause and remediation. Remediation lives in references, not in scripts."
  326. fi
  327. echo ""
  328. else
  329. CHECKS_JSON=""
  330. for key in "${!RESULTS[@]}"; do
  331. val="${RESULTS[$key]}"
  332. CHECKS_JSON+="\"${key}\": \"${val}\","
  333. done
  334. CHECKS_JSON="${CHECKS_JSON%,}" # remove trailing comma
  335. cat <<EOF
  336. {
  337. "hostname": "${HOSTNAME}",
  338. "timestamp": "${TIMESTAMP}",
  339. "critical_failures": ${CRITICAL_FAILURES},
  340. "overall_pass": $([ $CRITICAL_FAILURES -eq 0 ] && echo true || echo false),
  341. "checks": { ${CHECKS_JSON} }
  342. }
  343. EOF
  344. fi
  345. exit "$([[ $CRITICAL_FAILURES -eq 0 ]] && echo 0 || echo 1)"
plugins/sagemaker-ai/skills/hyperpod-node-debugger/scripts/triage-cluster.sh · Bundled script · 1,258 lines · 1 finding marked
  1. #!/usr/bin/env bash
  2. # triage-cluster.sh — read-only HyperPod node triage.
  3. #
  4. # Collects signals to route node issues to the right reference section:
  5. # - Cluster status, orchestrator, NodeRecovery
  6. # - Cluster events (root-cause signal for provisioning failures)
  7. # - Per-node health (HyperPod + EKS labels, Slurm state)
  8. # - VPC / SG config
  9. # - SSM reachability to compute nodes (hardware checks)
  10. #
  11. # Read-only: never modifies cluster state, never prints remediation commands.
  12. # Each [FAIL] / added issue carries a pointer of the form
  13. # "... → references/node-diagnostics-detail.md § <section>"
  14. # which the hyperpod-node-debugger skill uses to look up remediation.
  15. #
  16. # Usage:
  17. # bash triage-cluster.sh --cluster <name-or-arn> --region <region>
  18. # bash triage-cluster.sh --cluster <name-or-arn> --region <region> --node <instance-id>
  19. #
  20. # Exit codes:
  21. # 0 No critical (P0/P1) issues; P2 informational findings are allowed.
  22. # 1 One or more critical issues, or a fatal prerequisite / cluster-not-found.
  23. # 2 Invalid argument.
  24. set -euo pipefail
  25. for cmd in aws python3; do
  26. command -v "$cmd" &>/dev/null || {
  27. echo "ERROR: '$cmd' is required but not found." >&2
  28. exit 1
  29. }
  30. done
  31. HAS_UNBUFFER=true
  32. if ! command -v unbuffer &>/dev/null; then
  33. HAS_UNBUFFER=false
  34. fi
  35. CLUSTER=""
  36. REGION="${AWS_DEFAULT_REGION:-}"
  37. TARGET_NODE=""
  38. USE_COLOR=true
  39. usage() {
  40. cat <<EOF
  41. Usage: $0 --cluster <name-or-arn> --region <region> [options]
  42. Options:
  43. --cluster <name-or-arn> HyperPod cluster name or ARN (required)
  44. --region <region> AWS region (required unless \$AWS_DEFAULT_REGION is set)
  45. --node <instance-id> Focus on a single instance ID
  46. --no-color Disable ANSI colors
  47. -h, --help This message
  48. Read-only diagnostic. Every [FAIL] line carries a pointer like
  49. "→ references/node-diagnostics-detail.md § <section>" which the
  50. hyperpod-node-debugger skill uses to look up remediation.
  51. EOF
  52. }
  53. while [[ $# -gt 0 ]]; do
  54. case "$1" in
  55. --cluster) [[ $# -lt 2 ]] && { echo "ERROR: --cluster needs a value"; exit 2; }
  56. [[ ! "$2" =~ ^(arn:aws[a-z-]*:sagemaker:[a-z0-9-]+:[0-9]{12}:cluster/[a-z0-9]{12}|[a-zA-Z0-9]([-a-zA-Z0-9]{0,62}))$ ]] && { echo "ERROR: --cluster must be a valid HyperPod cluster name or ARN (got '$2')"; exit 2; }
  57. CLUSTER="$2"; shift 2 ;;
  58. --region) [[ $# -lt 2 ]] && { echo "ERROR: --region needs a value"; exit 2; }
  59. [[ ! "$2" =~ ^[a-z]{2}(-[a-z]+){1,2}-[0-9]+$ ]] && { echo "ERROR: --region must be a valid AWS region (got '$2')"; exit 2; }
  60. REGION="$2"; shift 2 ;;
  61. --node) [[ $# -lt 2 ]] && { echo "ERROR: --node needs a value"; exit 2; }
  62. [[ ! "$2" =~ ^i-[0-9a-f]{8,17}$ ]] && { echo "ERROR: --node must be an EC2 instance ID (i-xxxxxxxx...)"; exit 2; }
  63. TARGET_NODE="$2"; shift 2 ;;
  64. --no-color) USE_COLOR=false; shift ;;
  65. -h|--help) usage; exit 0 ;;
  66. *) echo "Unknown argument: $1"; usage; exit 2 ;;
  67. esac
  68. done
  69. [[ -z "$CLUSTER" ]] && {
  70. echo "Usage: $0 --cluster <name-or-arn> --region <region> [--node <instance-id>]"
  71. exit 1
  72. }
  73. if [[ -z "$REGION" ]]; then
  74. echo "ERROR: --region is required (or set AWS_DEFAULT_REGION before running)." >&2
  75. exit 2
  76. fi
  77. _CREDS=$(aws sts get-caller-identity --output json 2>&1) || {
  78. echo "ERROR: AWS credentials not configured or expired."
  79. echo "$_CREDS"
  80. echo ""
  81. echo "→ references/node-diagnostics-detail.md § K (Node Access via SSM) for credential setup"
  82. exit 1
  83. }
  84. # Auto-disable colors when stdout is not a TTY (agent-piped / redirected).
  85. if ! [ -t 1 ] || [ "${TERM:-}" = "dumb" ]; then
  86. USE_COLOR=false
  87. fi
  88. if "$USE_COLOR"; then
  89. RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
  90. CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
  91. else
  92. RED=''; GREEN=''; YELLOW=''; CYAN=''; BOLD=''; NC=''
  93. fi
  94. section() { echo ""; echo -e "${BOLD}${CYAN}════════════════════════════════════════════════════════${NC}"; echo -e "${BOLD}${CYAN} $1${NC}"; echo -e "${BOLD}${CYAN}════════════════════════════════════════════════════════${NC}"; }
  95. ok() { echo -e " ${GREEN}[PASS]${NC} $1"; }
  96. warn() { echo -e " ${YELLOW}[WARN]${NC} $1"; }
  97. bad() { echo -e " ${RED}[FAIL]${NC} $1"; }
  98. info() { echo -e " ${BOLD}[INFO]${NC} $1"; }
  99. ISSUES_FOUND=()
  100. add_issue() {
  101. local priority="${2:-P1}"
  102. ISSUES_FOUND+=("${priority}|$1")
  103. }
  104. aws_check_perms() {
  105. local result="$1" api_name="$2"
  106. if echo "$result" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized|AuthorizationError"; then
  107. warn "Permission denied: $api_name — results may be incomplete"
  108. add_issue "Missing IAM permission for $api_name → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P1"
  109. return 0
  110. fi
  111. return 1
  112. }
  113. _TEMP_FILES=()
  114. cleanup_temp() {
  115. [[ ${#_TEMP_FILES[@]} -gt 0 ]] && rm -f "${_TEMP_FILES[@]}" 2>/dev/null || true
  116. }
  117. trap cleanup_temp EXIT
  118. # Run a shell command on a HyperPod node via SSM.
  119. #
  120. # HyperPod uses a SageMaker-managed instance fleet, so `aws ssm send-command`
  121. # with a bare instance-id is not supported. The supported path is
  122. # `aws ssm start-session` with target `sagemaker-cluster:<cluster-id>_<group>-<iid>`
  123. # and document `AWS-StartNonInteractiveCommand`.
  124. #
  125. # Usage: ssm_run_on_node <instance-id> <instance-group-name> "<shell command>"
  126. # Returns remote stdout. start-session does not propagate the remote exit code.
  127. ssm_run_on_node() {
  128. local iid="$1" grp="$2" cmd="$3"
  129. [[ -z "$iid" || -z "$grp" || -z "$cmd" ]] && return 1
  130. [[ ! "$iid" =~ ^i-[0-9a-f]{8,17}$ ]] && return 1
  131. [[ -z "${CLUSTER_ID:-}" ]] && return 1
  132. [[ ! "$grp" =~ ^[A-Za-z0-9._-]+$ ]] && return 1
  133. if [[ "${HAS_UNBUFFER:-true}" != "true" ]]; then
  134. echo " [SKIP] on-node SSM probe skipped — install 'unbuffer' (expect package) to enable" >&2
  135. return 1
  136. fi
  137. local target="sagemaker-cluster:${CLUSTER_ID}_${grp}-${iid}"
  138. local tmp; tmp=$(mktemp 2>/dev/null) || return 1
  139. chmod 600 "$tmp" 2>/dev/null || true
  140. _TEMP_FILES+=("$tmp")
  141. # Embed the command as base64 because AWS-StartNonInteractiveCommand
  142. # collapses newlines in a single command element.
  143. local cmd_b64
  144. cmd_b64=$(printf '%s' "$cmd" | base64 | tr -d '\n') || return 1
  145. local remote="bash -c \"echo $cmd_b64 | base64 -d | bash\""
    • Note: Quoted as data in code: Decodes base64 and runs the result in a shell
    • Line 164 on GitHub
  146. python3 -c "import json,sys; print(json.dumps({'command':[sys.argv[1]]}))" "$remote" > "$tmp" || return 1
  147. local attempt=0 out rc
  148. while (( attempt < 5 )); do
  149. out=$(unbuffer timeout 180 aws ssm start-session \
  150. --target "$target" \
  151. --document-name AWS-StartNonInteractiveCommand \
  152. --parameters "file://$tmp" \
  153. --region "$REGION" 2>&1)
  154. rc=$?
  155. # SSM sometimes returns rc=0 with a transport error baked into stdout —
  156. # retry those (EOF, SessionManagerPlugin not found, i/o timeout).
  157. if (( rc == 0 )) && ! echo "$out" | grep -qiE "Cannot perform start session|EOF$|SessionManagerPlugin is not found|ERROR: Unable to|i/o timeout"; then
  158. # Strip SSM session banners and the echoed base64 command line.
  159. echo "$out" | grep -vE '^(Starting session with SessionId:|Exiting session with sessionId:|\s*$)' \
  160. | grep -vE "^(bash -c \"echo [A-Za-z0-9+/=]+ \| base64 -d \| bash\"|echo '[A-Za-z0-9+/=]+'|[A-Za-z0-9+/=]{40,}={0,2})[[:space:]]*\|?[[:space:]]*base64?[[:space:]]*-?d?[[:space:]]*\|?[[:space:]]*bash\"?\$" || true
  161. return 0
  162. fi
  163. if echo "$out" | grep -qiE "ThrottlingException|RequestLimitExceeded|InternalFailure|InternalError|ServiceUnavailable|TooManyUpdates|Cannot perform start session|EOF$|SessionManagerPlugin is not found|i/o timeout"; then
  164. attempt=$((attempt + 1))
  165. sleep $((attempt * 3))
  166. continue
  167. fi
  168. # Non-transient error; surface stderr so callers can diagnose.
  169. echo "$out" >&2
  170. return 1
  171. done
  172. return 1
  173. }
  174. echo ""
  175. echo -e "${CYAN}${BOLD}HyperPod Node Triage — READ-ONLY${NC}"
  176. echo -e "${CYAN} No cluster state will be modified. Each issue line below includes a${NC}"
  177. echo -e "${CYAN} pointer to references/node-diagnostics-detail.md for remediation.${NC}"
  178. section "1. Cluster Identity"
  179. CLUSTER_JSON=$(aws sagemaker describe-cluster \
  180. --cluster-name "$CLUSTER" \
  181. --region "$REGION" \
  182. --cli-read-timeout 30 \
  183. --output json 2>&1) || {
  184. echo -e "${RED}ERROR: Cannot find cluster '$CLUSTER' in region '$REGION'${NC}"
  185. echo ""
  186. echo "Available clusters in $REGION:"
  187. aws sagemaker list-clusters --region "$REGION" \
  188. --query 'ClusterSummaries[*].{Name:ClusterName,Status:ClusterStatus,ARN:ClusterArn}' \
  189. --output table 2>/dev/null || echo " (unable to list)"
  190. exit 1
  191. }
  192. CLUSTER_ARN=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ClusterArn',''))" 2>/dev/null || echo "")
  193. CLUSTER_STATUS=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ClusterStatus',''))" 2>/dev/null || echo "")
  194. ORCHESTRATOR=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print('EKS' if 'Eks' in d.get('Orchestrator',{}) else 'Slurm')" 2>/dev/null || echo "Unknown")
  195. NODE_RECOVERY=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('NodeRecovery','Unknown'))" 2>/dev/null || echo "Unknown")
  196. CLUSTER_ID=$(echo "$CLUSTER_ARN" | cut -d/ -f2 2>/dev/null || echo "")
  197. echo -e " ARN: ${CLUSTER_ARN}"
  198. echo -e " Status: ${BOLD}${CLUSTER_STATUS}${NC}"
  199. echo -e " Orchestrator: ${ORCHESTRATOR}"
  200. echo -e " NodeRecovery: ${NODE_RECOVERY}"
  201. echo -e " ClusterId: ${CLUSTER_ID}"
  202. [[ "$NODE_RECOVERY" == "None" || "$NODE_RECOVERY" == "Unknown" ]] && \
  203. warn "NodeRecovery is '$NODE_RECOVERY' — auto-replacement disabled. Manual intervention required for hardware failures."
  204. section "2. Cluster Events (Root Cause Signals)"
  205. # Fetch multiple pages and merge into a single JSON blob. Cap at 500 events to
  206. # bound memory and runtime on long-lived clusters (each page is up to 100).
  207. fetch_cluster_events() {
  208. local merged='[]' token='' page_json i=0
  209. while (( i < 5 )); do
  210. # Only pass --next-token if the token parses as a non-empty, strictly
  211. # base64/URL-safe string. Sending garbage (e.g. an error message that
  212. # leaked into $token) would cause ValidationException / BadRequest.
  213. if [[ -n "$token" && "$token" =~ ^[a-zA-Z0-9/+]*={0,2}$ ]]; then
  214. page_json=$(aws sagemaker list-cluster-events \
  215. --cluster-name "$CLUSTER" --region "$REGION" \
  216. --max-results 100 --next-token "$token" \
  217. --cli-read-timeout 30 --output json 2>&1) || break
  218. else
  219. page_json=$(aws sagemaker list-cluster-events \
  220. --cluster-name "$CLUSTER" --region "$REGION" \
  221. --max-results 100 \
  222. --cli-read-timeout 30 --output json 2>&1) || break
  223. fi
  224. local combined
  225. combined=$(printf '%s\0%s' "$merged" "$page_json" | python3 -c "
  226. import sys, json
  227. blob = sys.stdin.buffer.read()
  228. try:
  229. a, b = blob.split(b'\0', 1)
  230. merged = json.loads(a)
  231. page = json.loads(b)
  232. except (json.JSONDecodeError, ValueError):
  233. sys.exit(2)
  234. merged.extend(page.get('Events', []))
  235. print(json.dumps(merged))
  236. print(page.get('NextToken','') or '')
  237. " 2>/dev/null) || break
  238. merged=$(printf '%s\n' "$combined" | sed -n '1p')
  239. token=$(printf '%s\n' "$combined" | sed -n '2p')
  240. i=$((i+1))
  241. [[ -z "$token" ]] && break
  242. done
  243. printf '%s' "$merged" | python3 -c "
  244. import sys, json
  245. try:
  246. print(json.dumps({'Events': json.loads(sys.stdin.read())}))
  247. except json.JSONDecodeError:
  248. print('{\"Events\":[]}')
  249. " 2>/dev/null || echo '{"Events":[]}'
  250. }
  251. EVENTS=$(fetch_cluster_events)
  252. if [[ -z "$EVENTS" ]] || echo "$EVENTS" | grep -qE "AccessDenied|not authorized"; then
  253. aws_check_perms "$EVENTS" "sagemaker:ListClusterEvents"
  254. EVENTS='{"Events":[]}'
  255. fi
  256. EVENT_COUNT=$(echo "$EVENTS" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('Events',[])))" 2>/dev/null || echo "0")
  257. if [[ "$EVENT_COUNT" -gt 0 ]]; then
  258. echo -e " Found ${BOLD}${EVENT_COUNT}${NC} cluster events. Recent events:"
  259. echo ""
  260. echo "$EVENTS" | python3 -c "
  261. import sys, json
  262. events = json.load(sys.stdin).get('Events', [])
  263. for e in events[:20]:
  264. ts = e.get('EventTime','?')
  265. msg = e.get('Description','') or ''
  266. grp = e.get('InstanceGroupName','') or ''
  267. rt = e.get('ResourceType','') or ''
  268. tag = ''
  269. low = msg.lower()
  270. if 'EFA health checks did not run' in msg:
  271. tag = ' ← [GO TO SECTION A: EFA/SG FIX]'
  272. elif 'bootstrap failed' in low and 'network' in low:
  273. tag = ' ← [GO TO SECTION A+B: VPC/EKS FIX]'
  274. elif 'Lifecycle scripts' in msg or 'lifecycle script' in low:
  275. tag = ' ← [GO TO SECTION D: LIFECYCLE FIX]'
  276. elif 'hardware failure' in low:
  277. tag = ' ← [GO TO SECTION F: HARDWARE]'
  278. elif 'insufficient capacity' in low or 'sufficient capacity' in low:
  279. tag = ' ← [GO TO SECTION C: CAPACITY]'
  280. elif 'failed to provision' in low:
  281. tag = ' ← [CHECK SECTION C or F]'
  282. elif 'successfully' in low and 'failed' not in low:
  283. tag = ' [OK]'
  284. label = (grp or rt or '?')
  285. print(f' [{label}] {ts}')
  286. print(f' {msg[:120]}{\"...\" if len(msg) > 120 else \"\"}{tag}')
  287. print()
  288. " 2>/dev/null
  289. FAILURE_EVENTS=$(echo "$EVENTS" | python3 -c "
  290. import sys,json
  291. events=json.load(sys.stdin).get('Events',[])
  292. fails=[(e.get('Description','') or '') for e in events if any(k in (e.get('Description','') or '').lower() for k in ['failed','error','timeout','fault','unhealthy'])]
  293. for f in fails[:5]:
  294. print(f)
  295. " 2>/dev/null || echo "")
  296. if echo "$FAILURE_EVENTS" | grep -qi "efa health"; then
  297. add_issue "EFA health check failure → references/node-diagnostics-detail.md § A (EFA / Security Group)" "P0"
  298. fi
  299. if echo "$FAILURE_EVENTS" | grep -qi "network misconfiguration\|bootstrap failed"; then
  300. add_issue "K8s bootstrap network error → references/node-diagnostics-detail.md § A (EFA / Security Group) + § B (VPC / Routing)" "P1"
  301. fi
  302. if echo "$FAILURE_EVENTS" | grep -qi "lifecycle script"; then
  303. add_issue "Lifecycle script failure → references/node-diagnostics-detail.md § D (Lifecycle Scripts)" "P1"
  304. fi
  305. if echo "$FAILURE_EVENTS" | grep -qi "hardware failure"; then
  306. add_issue "Hardware failure detected → references/node-diagnostics-detail.md § F (Hardware / Auto-Repair)" "P1"
  307. fi
  308. if echo "$FAILURE_EVENTS" | grep -qi "insufficient capacity"; then
  309. add_issue "Insufficient capacity → references/node-diagnostics-detail.md § C (Capacity / AZ)" "P1"
  310. fi
  311. else
  312. warn "No cluster events available (may be Slurm cluster or no events yet)"
  313. fi
  314. section "3. Node Health Status"
  315. # Paginate list-cluster-nodes — default page is only 10 nodes, so large clusters
  316. # would otherwise be diagnosed on a tiny sample.
  317. fetch_all_cluster_nodes() {
  318. local merged='[]' token='' page_json combined i=0
  319. local max_pages=200 # 200 × 100 = 20 000 nodes, supports 7k+ clusters
  320. while (( i < max_pages )); do
  321. # Validate token format before sending — avoid BadRequest on garbage.
  322. if [[ -n "$token" && "$token" =~ ^[a-zA-Z0-9/+]*={0,2}$ ]]; then
  323. page_json=$(aws sagemaker list-cluster-nodes \
  324. --cluster-name "$CLUSTER" --region "$REGION" \
  325. --max-results 100 --next-token "$token" \
  326. --cli-read-timeout 30 --output json 2>&1) || break
  327. else
  328. page_json=$(aws sagemaker list-cluster-nodes \
  329. --cluster-name "$CLUSTER" --region "$REGION" \
  330. --max-results 100 \
  331. --cli-read-timeout 30 --output json 2>&1) || break
  332. fi
  333. # Merge via stdin (NUL-delimited) to avoid ARG_MAX truncation at ~500 nodes.
  334. combined=$(printf '%s\0%s' "$merged" "$page_json" | python3 -c "
  335. import sys, json
  336. blob = sys.stdin.buffer.read()
  337. try:
  338. a, b = blob.split(b'\0', 1)
  339. merged = json.loads(a)
  340. page = json.loads(b)
  341. except (json.JSONDecodeError, ValueError):
  342. sys.exit(2)
  343. merged.extend(page.get('ClusterNodeSummaries', []))
  344. print(json.dumps(merged))
  345. print(page.get('NextToken','') or '')
  346. " 2>/dev/null) || break
  347. merged=$(printf '%s\n' "$combined" | sed -n '1p')
  348. token=$(printf '%s\n' "$combined" | sed -n '2p')
  349. i=$((i+1))
  350. [[ -z "$token" ]] && break
  351. done
  352. if (( i == max_pages )) && [[ -n "$token" ]]; then
  353. echo "WARN: list-cluster-nodes truncated at ${max_pages} pages (~$((max_pages*100)) nodes). Diagnostic sample is incomplete for very large clusters." >&2
  354. fi
  355. printf '%s' "$merged" | python3 -c "
  356. import sys, json
  357. try:
  358. print(json.dumps({'ClusterNodeSummaries': json.loads(sys.stdin.read())}))
  359. except json.JSONDecodeError:
  360. print('{\"ClusterNodeSummaries\":[]}')
  361. " 2>/dev/null || echo '{"ClusterNodeSummaries":[]}'
  362. }
  363. NODES_JSON=$(fetch_all_cluster_nodes)
  364. if [[ -z "$NODES_JSON" ]] || echo "$NODES_JSON" | grep -qE "AccessDenied|not authorized"; then
  365. aws_check_perms "$NODES_JSON" "sagemaker:ListClusterNodes"
  366. NODES_JSON='{"ClusterNodeSummaries":[]}'
  367. fi
  368. TOTAL_NODES=$(echo "$NODES_JSON" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('ClusterNodeSummaries',[])))" 2>/dev/null || echo "0")
  369. RUNNING_NODES=$(echo "$NODES_JSON" | python3 -c "import sys,json; print(sum(1 for n in json.load(sys.stdin).get('ClusterNodeSummaries',[]) if n.get('InstanceStatus',{}).get('Status')=='Running'))" 2>/dev/null || echo "0")
  370. BAD_NODES=$(echo "$NODES_JSON" | python3 -c "import sys,json; print(sum(1 for n in json.load(sys.stdin).get('ClusterNodeSummaries',[]) if n.get('InstanceStatus',{}).get('Status') not in ('Running','')))" 2>/dev/null || echo "0")
  371. echo -e " Total: ${TOTAL_NODES} Running: ${GREEN}${RUNNING_NODES}${NC} Problems: ${RED}${BAD_NODES}${NC}"
  372. if [[ "$BAD_NODES" -gt 0 ]]; then
  373. echo ""
  374. echo -e " ${RED}Non-Running nodes:${NC}"
  375. echo "$NODES_JSON" | python3 -c "
  376. import sys,json
  377. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  378. for n in nodes:
  379. status=n.get('InstanceStatus',{})
  380. st=status.get('Status','?')
  381. if st not in ('Running',''):
  382. iid=n.get('InstanceId','?')
  383. grp=n.get('InstanceGroupName','?')
  384. itype=n.get('InstanceType','?')
  385. msg=status.get('Message','')
  386. print(f' [FAIL] {iid} ({grp} / {itype})')
  387. print(f' Status: {st}')
  388. if msg:
  389. print(f' Message: {msg[:100]}')
  390. print()
  391. " 2>/dev/null
  392. add_issue "$BAD_NODES node(s) not Running → references/node-diagnostics-detail.md § F (Hardware / Auto-Repair)" "P1"
  393. else
  394. ok "All $TOTAL_NODES nodes are Running"
  395. fi
  396. if [[ -n "$TARGET_NODE" ]]; then
  397. echo ""
  398. echo -e " ${BOLD}Targeted node: ${TARGET_NODE}${NC}"
  399. NODE_DETAIL=$(aws sagemaker describe-cluster-node \
  400. --cluster-name "$CLUSTER" \
  401. --node-id "$TARGET_NODE" \
  402. --region "$REGION" \
  403. --cli-read-timeout 30 \
  404. --output json 2>&1 || true)
  405. if echo "$NODE_DETAIL" | grep -qiE "ResourceNotFound|not found|ValidationException"; then
  406. bad "Node '$TARGET_NODE' not found in cluster '$CLUSTER'"
  407. info "Verify the instance ID belongs to this cluster:"
  408. info " aws sagemaker list-cluster-nodes --cluster-name $CLUSTER --region $REGION --query 'ClusterNodeSummaries[*].InstanceId' --output text"
  409. add_issue "Node $TARGET_NODE not found in cluster $CLUSTER → verify --cluster and --node arguments" "P0"
  410. TARGET_NODE="" # clear so downstream SSM probe doesn't retry on nonexistent node
  411. elif echo "$NODE_DETAIL" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
  412. warn "Permission denied: sagemaker:DescribeClusterNode — check IAM policy"
  413. else
  414. echo "$NODE_DETAIL" | python3 -c "
  415. import sys,json
  416. try:
  417. d=json.load(sys.stdin).get('NodeDetails',{})
  418. st=d.get('InstanceStatus',{})
  419. print(f' Status: {st.get(\"Status\",\"?\")}')
  420. print(f' Launch: {d.get(\"LaunchTime\",\"?\")}')
  421. print(f' Message: {st.get(\"Message\",\"\")}')
  422. print(f' Type: {d.get(\"InstanceType\",\"?\")}')
  423. print(f' Group: {d.get(\"InstanceGroupName\",\"?\")}')
  424. except Exception:
  425. pass
  426. " 2>/dev/null
  427. fi
  428. fi
  429. if [[ "$ORCHESTRATOR" == "EKS" ]]; then
  430. section "4. EKS Node Health Labels"
  431. if command -v kubectl &>/dev/null; then
  432. UNHEALTHY_LABELS=$(kubectl get nodes \
  433. -l 'sagemaker.amazonaws.com/node-health-status notin (Schedulable)' \
  434. -o custom-columns='NODE:.metadata.name,HEALTH:.metadata.labels.sagemaker\.amazonaws\.com/node-health-status,FAULT:.metadata.labels.sagemaker\.amazonaws\.com/fault-types,DHC:.metadata.labels.sagemaker\.amazonaws\.com/deep-health-check-status' \
  435. --no-headers 2>/dev/null || echo "")
  436. if [[ -n "$UNHEALTHY_LABELS" ]]; then
  437. bad "Nodes with health issues:"
  438. while IFS= read -r line; do
  439. echo " $line"
  440. if echo "$line" | grep -q "PendingReplacement"; then
  441. add_issue "Node pending replacement (UnschedulablePendingReplacement) → references/node-diagnostics-detail.md § F (Hardware / Auto-Repair)" "P1"
  442. elif echo "$line" | grep -q "PendingReboot"; then
  443. add_issue "Node pending reboot (UnschedulablePendingReboot) → references/node-diagnostics-detail.md § F (Hardware / Auto-Repair)" "P1"
  444. fi
  445. done <<< "$UNHEALTHY_LABELS"
  446. else
  447. ok "All EKS nodes have healthy labels (Schedulable)"
  448. fi
  449. # Check deep health check status. Under `set -o pipefail`, a failed kubectl
  450. # with `| wc -l || echo 0` yields "0\n0". Count safely via a tmp var.
  451. DHC_FAILED_OUT=$(kubectl get nodes \
  452. -l 'sagemaker.amazonaws.com/deep-health-check-status=Failed' \
  453. -o name 2>/dev/null || true)
  454. DHC_FAILED=$(echo -n "$DHC_FAILED_OUT" | grep -c . || true)
  455. [[ -z "$DHC_FAILED" ]] && DHC_FAILED=0
  456. [[ "$DHC_FAILED" -gt 0 ]] && bad "$DHC_FAILED node(s) have deep-health-check-status=Failed → references/node-diagnostics-detail.md § G (GPU/Accelerator) + § F (Hardware / Auto-Repair)"
  457. else
  458. warn "kubectl not available — cannot check EKS node labels (install kubectl to enable this check)"
  459. fi
  460. fi
  461. if [[ "$ORCHESTRATOR" == "EKS" ]] && command -v kubectl &>/dev/null; then
  462. section "4a. EKS CNI & System Pod Health"
  463. CNI_ISSUES=0
  464. # aws-node (VPC CNI plugin) — if this crashes, no pods can get IPs
  465. AWS_NODE_DS=$(kubectl get ds -n kube-system aws-node -o json 2>/dev/null || echo "")
  466. if [[ -n "$AWS_NODE_DS" && "$AWS_NODE_DS" != "" ]]; then
  467. AWS_NODE_STATUS=$(echo "$AWS_NODE_DS" | python3 -c "
  468. import sys, json
  469. ds = json.load(sys.stdin)
  470. desired = ds.get('status',{}).get('desiredNumberScheduled', 0)
  471. ready = ds.get('status',{}).get('numberReady', 0)
  472. unavail = ds.get('status',{}).get('numberUnavailable', 0)
  473. if unavail > 0:
  474. print(f'FAIL:{unavail} of {desired} aws-node pods not ready — pod networking broken on those nodes')
  475. elif ready == desired and desired > 0:
  476. print(f'PASS:aws-node DaemonSet healthy ({ready}/{desired} ready)')
  477. elif desired == 0:
  478. print('WARN:aws-node DaemonSet has 0 desired pods')
  479. else:
  480. print(f'WARN:aws-node DaemonSet {ready}/{desired} ready')
  481. " 2>/dev/null || echo "")
  482. if [[ -n "$AWS_NODE_STATUS" ]]; then
  483. _level="${AWS_NODE_STATUS%%:*}"
  484. _msg="${AWS_NODE_STATUS#*:}"
  485. case "$_level" in
  486. PASS) ok "$_msg" ;;
  487. FAIL) bad "$_msg"
  488. add_issue "aws-node (VPC CNI) pods failing → references/node-diagnostics-detail.md § O (CNI / Pod Networking)" "P0"
  489. CNI_ISSUES=$((CNI_ISSUES + 1))
  490. ;;
  491. WARN) warn "$_msg" ;;
  492. esac
  493. fi
  494. CNI_CRASHES=$(kubectl get pods -n kube-system -l k8s-app=aws-node --no-headers 2>/dev/null \
  495. | grep -iE "CrashLoopBackOff|Error|ImagePullBackOff" || true)
  496. if [[ -n "$CNI_CRASHES" ]]; then
  497. bad "aws-node pods in crash state:"
  498. echo "$CNI_CRASHES" | while IFS= read -r line; do echo " $line"; done
  499. add_issue "aws-node CrashLoopBackOff — pod networking broken → references/node-diagnostics-detail.md § O (CNI / Pod Networking)" "P0"
  500. CNI_ISSUES=$((CNI_ISSUES + 1))
  501. CNI_LOGS=$(kubectl logs -n kube-system -l k8s-app=aws-node --tail=20 2>/dev/null | \
  502. grep -iE "error|failed|refused|timeout|fatal|gRPC|ipamd|eni" | tail -5 || true)
  503. if [[ -n "$CNI_LOGS" ]]; then
  504. info "Recent aws-node error logs:"
  505. echo "$CNI_LOGS" | while IFS= read -r line; do info " $line"; done
  506. fi
  507. fi
  508. else
  509. info "aws-node DaemonSet not found in kube-system (may use alternate CNI)"
  510. fi
  511. # kube-proxy — if down, service networking breaks
  512. KP_CRASHES=$(kubectl get pods -n kube-system -l k8s-app=kube-proxy --no-headers 2>/dev/null \
  513. | grep -iE "CrashLoopBackOff|Error|ImagePullBackOff" || true)
  514. if [[ -n "$KP_CRASHES" ]]; then
  515. bad "kube-proxy pods in crash state:"
  516. echo "$KP_CRASHES" | while IFS= read -r line; do echo " $line"; done
  517. add_issue "kube-proxy crash — service networking broken → references/node-diagnostics-detail.md § O (CNI / Pod Networking)" "P0"
  518. CNI_ISSUES=$((CNI_ISSUES + 1))
  519. fi
  520. # CoreDNS — if down, DNS resolution fails (NCCL MASTER_ADDR, service discovery)
  521. COREDNS_CRASHES=$(kubectl get pods -n kube-system -l k8s-app=kube-dns --no-headers 2>/dev/null \
  522. | grep -iE "CrashLoopBackOff|Error|ImagePullBackOff" || true)
  523. if [[ -n "$COREDNS_CRASHES" ]]; then
  524. bad "CoreDNS pods in crash state — DNS resolution will fail:"
  525. echo "$COREDNS_CRASHES" | while IFS= read -r line; do echo " $line"; done
  526. add_issue "CoreDNS crash — DNS broken → references/node-diagnostics-detail.md § O (CNI / Pod Networking)" "P0"
  527. CNI_ISSUES=$((CNI_ISSUES + 1))
  528. fi
  529. [[ "$CNI_ISSUES" -eq 0 ]] && ok "kube-system networking pods healthy (aws-node, kube-proxy, CoreDNS)"
  530. fi
  531. if [[ "$ORCHESTRATOR" == "Slurm" ]]; then
  532. section "4b. Slurm Node States"
  533. if command -v sinfo &>/dev/null; then
  534. SLURM_DOWN=$(sinfo -o "%N %T %30E" --noheader 2>/dev/null | grep -iE "down|drain|fail" || true)
  535. if [[ -n "$SLURM_DOWN" ]]; then
  536. bad "Slurm nodes with issues:"
  537. echo "$SLURM_DOWN" | while IFS= read -r line; do
  538. echo " $line"
  539. done
  540. DOWN_COUNT=$(echo "$SLURM_DOWN" | grep -c .)
  541. add_issue "$DOWN_COUNT Slurm node(s) down/drained → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P1"
  542. else
  543. ok "All Slurm nodes show idle/alloc/mixed state"
  544. fi
  545. STUCK_JOBS=$(squeue -o "%i %j %T %R %N" --noheader 2>/dev/null | grep -iE "COMPLETING|CONFIGURING" || true)
  546. if [[ -n "$STUCK_JOBS" ]]; then
  547. warn "Stuck jobs detected (COMPLETING/CONFIGURING):"
  548. echo "$STUCK_JOBS" | head -5 | while IFS= read -r line; do echo " $line"; done
  549. add_issue "Stuck Slurm jobs → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P1"
  550. fi
  551. else
  552. info "Slurm CLI not available locally — to check Slurm node states, SSM into the head node:"
  553. info " sinfo -o '%N %T %30E'"
  554. info " squeue -o '%i %j %T %R %N'"
  555. info ""
  556. info "Or use SSM to run remotely:"
  557. if [[ -n "$CLUSTER_ID" ]]; then
  558. HEAD_NODE=$(echo "$NODES_JSON" | python3 -c "
  559. import sys,json
  560. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  561. for n in nodes:
  562. g=n.get('InstanceGroupName','').lower()
  563. if any(x in g for x in ['controller','head','master']):
  564. print(n.get('InstanceId','') + ' ' + n.get('InstanceGroupName',''))
  565. break
  566. else:
  567. for n in nodes:
  568. if n.get('InstanceStatus',{}).get('Status')=='Running':
  569. print(n.get('InstanceId','') + ' ' + n.get('InstanceGroupName',''))
  570. break
  571. " 2>/dev/null || echo "")
  572. if [[ -n "$HEAD_NODE" ]]; then
  573. H_IID=$(echo "$HEAD_NODE" | awk '{print $1}')
  574. H_GRP=$(echo "$HEAD_NODE" | awk '{print $2}')
  575. info " aws ssm start-session --target sagemaker-cluster:${CLUSTER_ID}_${H_GRP}-${H_IID} --region $REGION"
  576. fi
  577. fi
  578. if command -v session-manager-plugin &>/dev/null && [[ -n "$HEAD_NODE" ]]; then
  579. H_IID=$(echo "$HEAD_NODE" | awk '{print $1}')
  580. # Validate instance ID format — defense-in-depth against unexpected input.
  581. if [[ "$H_IID" =~ ^i-[0-9a-f]{8,17}$ ]]; then
  582. info ""
  583. info "Running Slurm checks via SSM on controller ${H_IID}..."
  584. # Unique delimiter prevents false matches if check output happens to contain marker text.
  585. local_nonce=$(date +%s%N 2>/dev/null || echo "$RANDOM")
  586. SLURM_CHECK_SH=$(cat <<EOF
  587. echo SLURM_CHECK_START_${local_nonce}
  588. scontrol show config >/dev/null 2>&1 || echo SLURMCTLD_DOWN_${local_nonce}
  589. echo DOWN_NODES_${local_nonce}
  590. sinfo -o '%20N %10T %30E' --noheader 2>/dev/null | grep -iE 'down|drain|fail' | head -10
  591. echo END_DOWN_${local_nonce}
  592. echo STUCK_COUNT_${local_nonce}
  593. squeue -o '%i %T' --noheader 2>/dev/null | grep -cE 'COMPLETING|CONFIGURING' || echo 0
  594. echo MUNGE_${local_nonce}
  595. systemctl is-active munge 2>/dev/null || echo inactive
  596. echo SLURM_CHECK_END_${local_nonce}
  597. EOF
  598. )
  599. SSM_STDOUT=$(ssm_run_on_node "$H_IID" "$H_GRP" "$SLURM_CHECK_SH" || echo "")
  600. if [[ -z "$SSM_STDOUT" ]] || ! echo "$SSM_STDOUT" | grep -q "SLURM_CHECK_START_${local_nonce}"; then
  601. warn "Slurm SSM probe returned no usable output — controller may be unreachable or SSM agent not responding"
  602. add_issue "Slurm controller SSM probe failed → references/node-diagnostics-detail.md § K (Node Access via SSM) + § H (Slurm Node Management)" "P1"
  603. fi
  604. if echo "$SSM_STDOUT" | grep -q "SLURM_CHECK_START_${local_nonce}"; then
  605. if echo "$SSM_STDOUT" | grep -q "SLURMCTLD_DOWN_${local_nonce}"; then
  606. bad "slurmctld not responding on controller — all Slurm operations blocked"
  607. add_issue "slurmctld down → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P0"
  608. else
  609. ok "slurmctld responding"
  610. fi
  611. SSM_DOWN_LINES=$(echo "$SSM_STDOUT" | sed -n "/^DOWN_NODES_${local_nonce}\$/,/^END_DOWN_${local_nonce}\$/p" | grep -v "^DOWN_NODES_\|^END_DOWN_" | grep -v "^$" || true)
  612. if [[ -n "$SSM_DOWN_LINES" ]]; then
  613. bad "Slurm nodes with issues (via SSM):"
  614. echo "$SSM_DOWN_LINES" | while IFS= read -r line; do info " $line"; done
  615. SSM_DOWN_COUNT=$(echo "$SSM_DOWN_LINES" | grep -c .)
  616. add_issue "$SSM_DOWN_COUNT Slurm node(s) down/drained → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P1"
  617. else
  618. ok "All Slurm nodes healthy (via SSM)"
  619. fi
  620. STUCK_COUNT=$(echo "$SSM_STDOUT" | sed -n "/^STUCK_COUNT_${local_nonce}\$/{n;p;}" | tr -d '[:space:]')
  621. [[ "${STUCK_COUNT:-0}" =~ ^[0-9]+$ ]] && [[ "${STUCK_COUNT:-0}" -gt 0 ]] && \
  622. add_issue "$STUCK_COUNT stuck Slurm jobs → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P1"
  623. if echo "$SSM_STDOUT" | sed -n "/^MUNGE_${local_nonce}\$/{n;p;}" | grep -q inactive; then
  624. bad "munge authentication service inactive on controller"
  625. add_issue "munge service inactive → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P0"
  626. fi
  627. fi
  628. fi
  629. fi
  630. fi
  631. fi
  632. section "5. Cluster VPC Resources"
  633. RESOURCES=$(echo "$CLUSTER_JSON" | python3 -c "
  634. import sys,json
  635. d=json.load(sys.stdin)
  636. vpc=d.get('VpcConfig',{})
  637. sgs=vpc.get('SecurityGroupIds',[])
  638. subnets=vpc.get('Subnets',[])
  639. print('SGs=' + ','.join(sgs))
  640. print('Subnets=' + ','.join(subnets))
  641. " 2>/dev/null || echo "")
  642. CLUSTER_SGS=$(echo "$RESOURCES" | grep "^SGs=" | cut -d= -f2)
  643. CLUSTER_SUBNETS=$(echo "$RESOURCES" | grep "^Subnets=" | cut -d= -f2)
  644. if [[ -n "$CLUSTER_SGS" ]]; then
  645. echo -e " Security Groups: ${BOLD}${CLUSTER_SGS}${NC}"
  646. echo -e " Subnets: ${BOLD}${CLUSTER_SUBNETS}${NC}"
  647. for SG in $(echo "$CLUSTER_SGS" | tr ',' ' '); do
  648. # Nested JMESPath filter `UserIdGroupPairs[?GroupId=='...']` inside an
  649. # already-filtered projection returns empty under AWS CLI even when the
  650. # rule is present — false-flags healthy SGs as a P0. Flatten the array
  651. # and match in bash instead.
  652. _SG_RESULT=$(aws ec2 describe-security-groups \
  653. --group-ids "$SG" --region "$REGION" \
  654. --cli-read-timeout 15 \
  655. --query "SecurityGroups[0].IpPermissionsEgress[?IpProtocol=='-1'].UserIdGroupPairs[].GroupId" \
  656. --output text 2>&1)
  657. if aws_check_perms "$_SG_RESULT" "ec2:DescribeSecurityGroups"; then
  658. info "SG check skipped for $SG (permission denied)"
  659. continue
  660. fi
  661. if echo "$_SG_RESULT" | tr '\t' '\n' | grep -qxF "$SG"; then
  662. ok "SG ${SG} has outbound self-referencing rule (EFA ready)"
  663. else
  664. bad "SG ${SG} missing outbound self-referencing rule → EFA will fail"
  665. add_issue "Missing SG outbound self-ref rule on ${SG} → references/node-diagnostics-detail.md § A (EFA / Security Group)" "P0"
  666. fi
  667. done
  668. if [[ -n "$CLUSTER_SUBNETS" ]]; then
  669. echo ""
  670. # shellcheck disable=SC2046 # intentional word splitting for multiple subnet IDs
  671. IFS=',' read -ra _subnet_arr <<< "$CLUSTER_SUBNETS"
  672. _SUB_RESULT=$(aws ec2 describe-subnets \
  673. --subnet-ids "${_subnet_arr[@]}" \
  674. --region "$REGION" \
  675. --cli-read-timeout 15 \
  676. --query 'Subnets[*].{SubnetId:SubnetId,AZ:AvailabilityZone,FreeIPs:AvailableIpAddressCount}' \
  677. --output table 2>&1)
  678. if ! aws_check_perms "$_SUB_RESULT" "ec2:DescribeSubnets"; then
  679. echo "$_SUB_RESULT"
  680. fi
  681. fi
  682. else
  683. warn "No VpcConfig found in cluster — cluster may not have customer VPC"
  684. fi
  685. section "6. CloudWatch Logs"
  686. if [[ -n "$CLUSTER_ID" ]]; then
  687. CLUSTER_NAME_ONLY=$(echo "$CLUSTER" | awk -F/ '{print $NF}')
  688. LOG_GROUP="/aws/sagemaker/Clusters/${CLUSTER_NAME_ONLY}/${CLUSTER_ID}"
  689. echo -e " Log group: ${LOG_GROUP}"
  690. _LOG_RESULT=$(aws logs describe-log-groups \
  691. --log-group-name-prefix "$LOG_GROUP" \
  692. --region "$REGION" \
  693. --cli-read-timeout 15 \
  694. --query 'logGroups[0].logGroupName' \
  695. --output text 2>&1)
  696. if aws_check_perms "$_LOG_RESULT" "logs:DescribeLogGroups"; then
  697. LOG_EXISTS="None"
  698. else
  699. LOG_EXISTS="$_LOG_RESULT"
  700. fi
  701. if [[ "$LOG_EXISTS" == "None" || -z "$LOG_EXISTS" ]]; then
  702. warn "No CloudWatch log group found — logs may not be configured or cluster is new"
  703. info "Expected: $LOG_GROUP"
  704. else
  705. ok "Log group exists: $LOG_EXISTS"
  706. # Count recent log streams — paginate so the count reflects all streams,
  707. # not just the first 50 (default CloudWatch page size).
  708. STREAM_COUNT=0
  709. _LS_TOKEN=""; _LS_I=0
  710. while (( _LS_I < 20 )); do
  711. # Validate token format before sending — avoid BadRequest on garbage.
  712. if [[ -n "$_LS_TOKEN" && "$_LS_TOKEN" =~ ^[a-zA-Z0-9/+]*={0,2}$ ]]; then
  713. _LS_PAGE=$(aws logs describe-log-streams --log-group-name "$LOG_GROUP" \
  714. --region "$REGION" --cli-read-timeout 15 --limit 50 --next-token "$_LS_TOKEN" \
  715. --output json 2>/dev/null) || break
  716. else
  717. _LS_PAGE=$(aws logs describe-log-streams --log-group-name "$LOG_GROUP" \
  718. --region "$REGION" --cli-read-timeout 15 --limit 50 \
  719. --output json 2>/dev/null) || break
  720. fi
  721. _LS_INC=$(echo "$_LS_PAGE" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('logStreams',[])))" 2>/dev/null || echo 0)
  722. STREAM_COUNT=$((STREAM_COUNT + _LS_INC))
  723. _LS_TOKEN=$(echo "$_LS_PAGE" | python3 -c "import sys,json; print(json.load(sys.stdin).get('nextToken',''))" 2>/dev/null || echo "")
  724. _LS_I=$((_LS_I + 1))
  725. [[ -z "$_LS_TOKEN" ]] && break
  726. done
  727. info "$STREAM_COUNT log stream(s) available"
  728. info "To view: aws logs describe-log-streams --log-group-name \"$LOG_GROUP\" --region $REGION --output table"
  729. fi
  730. fi
  731. section "7. SSM Connectivity"
  732. if command -v session-manager-plugin &>/dev/null; then
  733. # `command -v` only verifies the binary exists — run --version to confirm it
  734. # actually works (permissions, broken install, etc.).
  735. if SSM_VER=$(session-manager-plugin --version 2>/dev/null); then
  736. ok "SSM Session Manager plugin installed (${SSM_VER})"
  737. else
  738. warn "SSM Session Manager plugin installed but --version failed — plugin may be corrupt or missing libs"
  739. add_issue "SSM plugin installed but broken → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P1"
  740. fi
  741. else
  742. warn "SSM Session Manager plugin NOT found"
  743. info "Install session-manager-plugin (see AWS Systems Manager documentation)"
  744. add_issue "SSM plugin missing → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P2"
  745. fi
  746. RUNNING_IDS=$(echo "$NODES_JSON" | python3 -c "
  747. import sys,json
  748. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  749. ids=[n.get('InstanceId') for n in nodes if n.get('InstanceStatus',{}).get('Status')=='Running']
  750. print(','.join(ids[:3]))
  751. " 2>/dev/null || echo "")
  752. if [[ -n "$RUNNING_IDS" ]]; then
  753. ok "Running nodes available for SSM (examples: ${RUNNING_IDS})"
  754. info "Use hyperpod-ssm skill with cluster ID: ${CLUSTER_ID}"
  755. else
  756. warn "No Running nodes found — SSM access not possible until nodes are healthy"
  757. fi
  758. # 8: On-Node Resource Checks (Memory / Storage / Utilities)
  759. # Runs via SSM on the target node (or first running node) to detect resource
  760. # exhaustion issues that only show up on-node: disk full, /dev/shm too small,
  761. # huge pages misconfigured, OOM signals.
  762. NODE_TO_PROBE="${TARGET_NODE}"
  763. NODE_TO_PROBE_GROUP=""
  764. if [[ -z "$NODE_TO_PROBE" ]]; then
  765. # Prefer GPU / accelerator nodes: a node probe on a CPU-only utility node
  766. # produces empty GPU / EFA sections and the user can't tell whether the
  767. # result is "no hardware" or "hardware is broken." Three-tier fallback
  768. # so the script still returns something on a CPU-only cluster.
  769. NODE_TO_PROBE=$(echo "$NODES_JSON" | python3 -c "
  770. import sys, json
  771. nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
  772. GPU_PREFIXES = ('ml.p3', 'ml.p3dn', 'ml.p4d', 'ml.p4de', 'ml.p5', 'ml.p5e',
  773. 'ml.p5en', 'ml.p6', 'ml.g4dn', 'ml.g5', 'ml.g6', 'ml.g6e', 'ml.g7e')
  774. NEURON_PREFIXES = ('ml.trn1', 'ml.trn2', 'ml.inf2')
  775. ACCEL_PREFIXES = GPU_PREFIXES + NEURON_PREFIXES
  776. def is_utility(n):
  777. g = (n.get('InstanceGroupName','') or '').lower()
  778. return any(x in g for x in ('controller', 'head', 'master'))
  779. running = [n for n in nodes if n.get('InstanceStatus', {}).get('Status','') == 'Running']
  780. tier1 = [n for n in running if (n.get('InstanceType','') or '').startswith(ACCEL_PREFIXES) and not is_utility(n)]
  781. tier2 = [n for n in running if n not in tier1 and not is_utility(n)]
  782. tier3 = [n for n in running if n not in tier1 and n not in tier2]
  783. for n in tier1 + tier2 + tier3:
  784. print(n.get('InstanceId', ''))
  785. break
  786. " 2>/dev/null || echo "")
  787. fi
  788. if [[ -n "$NODE_TO_PROBE" ]]; then
  789. NODE_TO_PROBE_GROUP=$(echo "$NODES_JSON" | NODE_ID_ENV="$NODE_TO_PROBE" python3 -c "
  790. import sys,json,os
  791. target=os.environ['NODE_ID_ENV']
  792. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  793. for n in nodes:
  794. if n.get('InstanceId','')==target:
  795. print(n.get('InstanceGroupName',''))
  796. break
  797. " 2>/dev/null || echo "")
  798. fi
  799. if [[ -n "$NODE_TO_PROBE" ]] \
  800. && [[ "$NODE_TO_PROBE" =~ ^i-[0-9a-f]{8,17}$ ]] \
  801. && [[ -n "$NODE_TO_PROBE_GROUP" ]] \
  802. && command -v session-manager-plugin &>/dev/null; then
  803. section "8. On-Node Resource Checks (via SSM)"
  804. info "Probing node: $NODE_TO_PROBE (group: ${NODE_TO_PROBE_GROUP})"
  805. resource_nonce=$(date +%s%N 2>/dev/null || echo "$RANDOM")
  806. RESOURCE_SH=$(cat <<EOF
  807. echo RESOURCE_CHECK_START_${resource_nonce}
  808. echo DISK_ROOT_${resource_nonce}
  809. df -h / 2>/dev/null | tail -1
  810. echo DISK_OPT_${resource_nonce}
  811. df -h /opt/sagemaker 2>/dev/null | tail -1 || echo NOT_MOUNTED
  812. echo DISK_NVME_${resource_nonce}
  813. df -h /opt/dlami/nvme 2>/dev/null | tail -1 || echo NOT_MOUNTED
  814. echo SHM_SIZE_${resource_nonce}
  815. df -h /dev/shm 2>/dev/null | tail -1
  816. echo MEMORY_INFO_${resource_nonce}
  817. free -h | grep Mem
  818. echo HUGEPAGES_${resource_nonce}
  819. cat /proc/meminfo 2>/dev/null | grep -i huge | head -5
  820. echo EFA_HUGE_PAGE_${resource_nonce}
  821. env 2>/dev/null | grep FI_EFA_USE_HUGE_PAGE || echo NOT_SET
  822. echo OOM_RECENT_${resource_nonce}
  823. dmesg 2>/dev/null | grep -iE 'oom|out of memory|cannot allocate' | tail -5 || echo NONE
  824. echo INODE_CHECK_${resource_nonce}
  825. df -i / 2>/dev/null | tail -1
  826. echo TIME_SYNC_${resource_nonce}
  827. chronyc tracking 2>/dev/null | grep -E 'System time|Leap status' || timedatectl status 2>/dev/null | grep -E 'synchronized|NTP service' || echo UNKNOWN
  828. echo SSM_AGENT_${resource_nonce}
  829. systemctl is-active amazon-ssm-agent 2>/dev/null || echo inactive
  830. echo NVME_MOUNTS_${resource_nonce}
  831. lsblk -nr -o NAME,MOUNTPOINT 2>/dev/null | grep -E 'nvme[0-9]+n[0-9]+\$' | head -10 || echo NONE
  832. echo GPU_XID_${resource_nonce}
  833. if command -v nvidia-smi >/dev/null 2>&1; then
  834. _gpu_xid_out=\$(
  835. dmesg 2>/dev/null | grep -E 'NVRM: Xid' | tail -10
  836. nvidia-smi -q 2>/dev/null | awk '
  837. /Uncorrectable/ { if (\$NF ~ /^[0-9]+\$/ && \$NF+0 > 0) print; next }
  838. /Pending Page (Blacklist|Blocklist|Retirement)/ { if (\$NF ~ /^[0-9]+\$/ && \$NF+0 > 0) print; next }
  839. '
  840. )
  841. if [[ -z "\$_gpu_xid_out" ]]; then echo NONE; else echo "\$_gpu_xid_out" | head -20; fi
  842. else
  843. echo NO_NVIDIA_SMI
  844. fi
  845. echo GPU_REMAP_${resource_nonce}
  846. # Row-remap state: 'Pending' rows indicate marginal GPU memory that needs a reset
  847. # to finalize the remap. If remap is reported Failed, the GPU is bad.
  848. # A stuck 'Pending' state across reboots is a known firmware edge case that can
  849. # silently degrade training without NCCL/DCGM flagging it — capture explicitly.
  850. if command -v nvidia-smi >/dev/null 2>&1; then
  851. nvidia-smi --query-remapped-rows=gpu_bus_id,remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure \
  852. --format=csv,noheader 2>/dev/null | head -16 || echo UNSUPPORTED
  853. else
  854. echo NO_NVIDIA_SMI
  855. fi
  856. echo GPU_DCGM_${resource_nonce}
  857. # DCGM health summary. Presence of 'Health Monitor Report' + 'PASS'/'Warn'/'Fail'
  858. # tells us DCGM has run recently. Absence is informational, not an error.
  859. # Row-remap errors surface here on drivers where nvidia-smi lags the firmware.
  860. if command -v dcgmi >/dev/null 2>&1; then
  861. dcgmi health --check -j 2>/dev/null | head -40 || dcgmi health --check 2>/dev/null | head -20 || echo DCGM_UNAVAILABLE
  862. else
  863. echo NO_DCGMI
  864. fi
  865. echo GPU_DCGM_LOGS_${resource_nonce}
  866. # DCGM nvvs log presence — SageMaker HyperPod runs DCGM medium/memtest as part
  867. # of deep-health-check. If this log is present the node has been health-checked
  868. # recently; tail captures last run result.
  869. if [ -d /var/log/nvidia-dcgm ] 2>/dev/null; then
  870. find /var/log/nvidia-dcgm -maxdepth 1 -type f -printf '%f\n' 2>/dev/null | head -5
  871. # \$ escapes are required: this heredoc is <<EOF (not <<'EOF'), so unescaped
  872. # shell variables would expand locally. Keep \$ to defer to the remote shell.
  873. NVVS_LATEST=\$(find /var/log/nvidia-dcgm -maxdepth 1 -name 'nvvs*.log' -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -1 | awk '{print \$2}')
  874. if [ -n "\$NVVS_LATEST" ]; then
  875. echo "--- tail of \$NVVS_LATEST ---"
  876. tail -n 5 "\$NVVS_LATEST" 2>/dev/null || true
  877. fi
  878. else
  879. echo NO_DCGM_LOG_DIR
  880. fi
  881. echo KERNEL_PANIC_${resource_nonce}
  882. dmesg 2>/dev/null | grep -iE 'Kernel panic - not syncing|watchdog: BUG|soft lockup|hard lockup|hung_task: blocked|BUG: unable to handle|BUG: kernel NULL|NMI watchdog' | tail -10 || echo NONE
  883. echo CONTAINERD_${resource_nonce}
  884. if command -v systemctl >/dev/null 2>&1; then
  885. systemctl is-active containerd 2>/dev/null || echo inactive
  886. else
  887. echo UNKNOWN
  888. fi
  889. echo RESOURCE_CHECK_END_${resource_nonce}
  890. EOF
  891. )
  892. RES_STDOUT=$(ssm_run_on_node "$NODE_TO_PROBE" "$NODE_TO_PROBE_GROUP" "$RESOURCE_SH" || echo "")
  893. extract_section() {
  894. local start="$1" end="$2"
  895. # grep -v returns 1 when every line is filtered out; under pipefail this
  896. # kills the pipeline even though the EMPTY output is legitimate. Force 0.
  897. { echo "$RES_STDOUT" | sed -n "/^${start}_${resource_nonce}\$/,/^${end}_${resource_nonce}\$/p" \
  898. | grep -v "^${start}_${resource_nonce}\$\|^${end}_${resource_nonce}\$" || true; }
  899. }
  900. if echo "$RES_STDOUT" | grep -q "RESOURCE_CHECK_START_${resource_nonce}"; then
  901. echo ""
  902. echo -e " ${BOLD}Storage:${NC}"
  903. ROOT_LINE=$(extract_section DISK_ROOT DISK_OPT | head -1)
  904. if [[ -n "$ROOT_LINE" ]]; then
  905. ROOT_USE_PCT=$(echo "$ROOT_LINE" | awk '{print $5}' | tr -d '%')
  906. if [[ "$ROOT_USE_PCT" =~ ^[0-9]+$ ]] && [[ "$ROOT_USE_PCT" -gt 90 ]]; then
  907. bad "Root volume: ${ROOT_USE_PCT}% used — CRITICALLY FULL (100GB fixed, cannot expand)"
  908. add_issue "Root volume ${ROOT_USE_PCT}% full → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P0"
  909. elif [[ "$ROOT_USE_PCT" =~ ^[0-9]+$ ]] && [[ "$ROOT_USE_PCT" -gt 80 ]]; then
  910. warn "Root volume: ${ROOT_USE_PCT}% used — approaching full"
  911. add_issue "Root volume ${ROOT_USE_PCT}% used → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
  912. else
  913. ok "Root volume: ${ROOT_USE_PCT:-?}% used"
  914. fi
  915. fi
  916. OPT_LINE=$(extract_section DISK_OPT DISK_NVME | head -1)
  917. if [[ "$OPT_LINE" != "NOT_MOUNTED" && -n "$OPT_LINE" ]]; then
  918. OPT_USE=$(echo "$OPT_LINE" | awk '{print $5}' | tr -d '%')
  919. if [[ "$OPT_USE" =~ ^[0-9]+$ ]] && [[ "$OPT_USE" -gt 90 ]]; then
  920. warn "/opt/sagemaker: ${OPT_USE}% used — secondary EBS nearing full"
  921. add_issue "/opt/sagemaker ${OPT_USE}% full → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
  922. else
  923. ok "/opt/sagemaker: ${OPT_USE:-?}% used"
  924. fi
  925. fi
  926. NVME_LINE=$(extract_section DISK_NVME SHM_SIZE | head -1)
  927. if [[ "$NVME_LINE" != "NOT_MOUNTED" && -n "$NVME_LINE" ]]; then
  928. ok "NVMe instance store: mounted at /opt/dlami/nvme"
  929. else
  930. # On GPU training instances NVMe is expected — flag if not mounted
  931. INSTANCE_TYPE_LOC=$(echo "$NODES_JSON" | NODE_ID_ENV="$NODE_TO_PROBE" python3 -c "
  932. import sys,json,os
  933. target=os.environ['NODE_ID_ENV']
  934. for n in json.load(sys.stdin).get('ClusterNodeSummaries',[]):
  935. if n.get('InstanceId','')==target:
  936. print(n.get('InstanceType',''))
  937. break
  938. " 2>/dev/null || echo "")
  939. if [[ "$INSTANCE_TYPE_LOC" =~ ^ml\.(p5|p5e|p5en|p4d|p4de|p6|trn1|trn2)\. ]]; then
  940. warn "/opt/dlami/nvme not mounted on $INSTANCE_TYPE_LOC — instance store expected"
  941. add_issue "NVMe instance store not mounted on $NODE_TO_PROBE ($INSTANCE_TYPE_LOC) → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
  942. fi
  943. fi
  944. INODE_LINE=$(extract_section INODE_CHECK TIME_SYNC | head -1)
  945. if [[ -n "$INODE_LINE" ]]; then
  946. INODE_PCT=$(echo "$INODE_LINE" | awk '{print $5}' | tr -d '%')
  947. if [[ "$INODE_PCT" =~ ^[0-9]+$ ]] && [[ "$INODE_PCT" -gt 90 ]]; then
  948. bad "Inode usage: ${INODE_PCT}% — filesystem running out of inodes"
  949. add_issue "Inode exhaustion ${INODE_PCT}% → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
  950. fi
  951. fi
  952. echo ""
  953. echo -e " ${BOLD}Memory:${NC}"
  954. MEM_LINE=$(extract_section MEMORY_INFO HUGEPAGES | head -1)
  955. [[ -n "$MEM_LINE" ]] && info "RAM: $MEM_LINE"
  956. SHM_LINE=$(extract_section SHM_SIZE MEMORY_INFO | head -1)
  957. if [[ -n "$SHM_LINE" ]]; then
  958. SHM_SIZE=$(echo "$SHM_LINE" | awk '{print $2}')
  959. SHM_USE_PCT=$(echo "$SHM_LINE" | awk '{print $5}' | tr -d '%')
  960. if [[ "$SHM_USE_PCT" =~ ^[0-9]+$ ]] && [[ "$SHM_USE_PCT" -gt 80 ]]; then
  961. warn "/dev/shm: ${SHM_USE_PCT}% used (size: $SHM_SIZE) — NCCL may fail with 'Bus error'"
  962. add_issue "/dev/shm ${SHM_USE_PCT}% full → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
  963. else
  964. ok "/dev/shm: ${SHM_USE_PCT:-?}% used (size: ${SHM_SIZE:-?})"
  965. fi
  966. fi
  967. EFA_HP=$(extract_section EFA_HUGE_PAGE OOM_RECENT | head -1)
  968. if [[ "$EFA_HP" == "NOT_SET" ]]; then
  969. HUGEPAGES_TOTAL=$(extract_section HUGEPAGES EFA_HUGE_PAGE | { grep "HugePages_Total" || true; } | awk '{print $2}')
  970. if [[ "${HUGEPAGES_TOTAL:-0}" == "0" ]]; then
  971. warn "FI_EFA_USE_HUGE_PAGE not set and HugePages_Total=0"
  972. add_issue "FI_EFA_USE_HUGE_PAGE not configured → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P2"
  973. fi
  974. elif echo "$EFA_HP" | grep -q "=0"; then
  975. ok "FI_EFA_USE_HUGE_PAGE=0 (huge pages disabled for EFA — os.fork() safe)"
  976. fi
  977. OOM_LINES=$(extract_section OOM_RECENT INODE_CHECK | { grep -v "^NONE$" || true; } | head -3)
  978. if [[ -n "$OOM_LINES" ]]; then
  979. echo ""
  980. bad "Recent OOM events detected on node:"
  981. echo "$OOM_LINES" | while IFS= read -r line; do info " $line"; done
  982. add_issue "OOM events on node $NODE_TO_PROBE → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
  983. else
  984. echo ""
  985. ok "No recent OOM events"
  986. fi
  987. # Time sync health — clock drift breaks TLS/SigV4 and Slurm accounting.
  988. TIME_STATUS=$(extract_section TIME_SYNC SSM_AGENT | head -3)
  989. if echo "$TIME_STATUS" | grep -qiE "synchronized: no|Not synchronised|UNKNOWN"; then
  990. warn "Time sync unhealthy — chronyc/timedatectl reports not synchronised"
  991. info "Clock drift breaks TLS/IAM (SigV4) and Slurm accounting"
  992. add_issue "Node $NODE_TO_PROBE time sync not healthy → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
  993. elif [[ -n "$TIME_STATUS" ]]; then
  994. ok "Time sync healthy"
  995. fi
  996. # SSM agent health — if we got here it's mostly working, but flag if systemd says otherwise.
  997. SSM_AGENT_STATUS=$(extract_section SSM_AGENT NVME_MOUNTS | head -1)
  998. if [[ "$SSM_AGENT_STATUS" == "inactive" ]]; then
  999. warn "amazon-ssm-agent reported inactive — may be restarting or broken"
  1000. add_issue "amazon-ssm-agent inactive on $NODE_TO_PROBE → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P1"
  1001. fi
  1002. # GPU XID / ECC / page-retirement — hardware faults visible via nvidia-smi query.
  1003. GPU_XID_LINES=$(extract_section GPU_XID GPU_REMAP | { grep -v "^NONE$" || true; } | { grep -v "^NO_NVIDIA_SMI$" || true; } | head -5)
  1004. if [[ -n "$GPU_XID_LINES" ]]; then
  1005. echo ""
  1006. bad "GPU XID / ECC / page-retirement signals on node $NODE_TO_PROBE:"
  1007. echo "$GPU_XID_LINES" | while IFS= read -r line; do info " $line"; done
  1008. add_issue "GPU XID / ECC / page-retirement on $NODE_TO_PROBE → references/node-diagnostics-detail.md § G (GPU/Accelerator) + § F (Hardware / Auto-Repair)" "P0"
  1009. fi
  1010. # GPU row-remapping — marginal GPU memory. Pending rows that never clear
  1011. # indicate a firmware edge case where the remap is stuck; Failed rows mean
  1012. # the GPU is bad and must be replaced. Silent degrader — NCCL and DCGM's
  1013. # default checks can miss this.
  1014. GPU_REMAP_LINES=$(extract_section GPU_REMAP GPU_DCGM | { grep -v "^NO_NVIDIA_SMI$" || true; } | { grep -v "^UNSUPPORTED$" || true; })
  1015. if [[ -n "$GPU_REMAP_LINES" ]]; then
  1016. # Columns (csv,noheader): gpu_bus_id, correctable, uncorrectable, pending, failure
  1017. REMAP_PENDING_TOTAL=0
  1018. REMAP_FAILED_TOTAL=0
  1019. REMAP_UNCORRECT_TOTAL=0
  1020. while IFS= read -r line; do
  1021. [[ -z "$line" ]] && continue
  1022. _p=$(echo "$line" | awk -F, '{gsub(/ /,""); print $4}')
  1023. _f=$(echo "$line" | awk -F, '{gsub(/ /,""); print $5}')
  1024. _u=$(echo "$line" | awk -F, '{gsub(/ /,""); print $3}')
  1025. [[ "$_p" =~ ^[0-9]+$ ]] && REMAP_PENDING_TOTAL=$((REMAP_PENDING_TOTAL + _p))
  1026. [[ "$_u" =~ ^[0-9]+$ ]] && REMAP_UNCORRECT_TOTAL=$((REMAP_UNCORRECT_TOTAL + _u))
  1027. [[ "$_f" == "Yes" || "$_f" == "1" ]] && REMAP_FAILED_TOTAL=$((REMAP_FAILED_TOTAL + 1))
  1028. done <<< "$GPU_REMAP_LINES"
  1029. if [[ "$REMAP_FAILED_TOTAL" -gt 0 ]]; then
  1030. bad "GPU row-remap FAILED on $REMAP_FAILED_TOTAL device(s) — GPU has exceeded remap capacity"
  1031. add_issue "GPU row-remap failure on $NODE_TO_PROBE (bad memory, replace GPU) → references/node-diagnostics-detail.md § G (GPU/Accelerator) + § F (Hardware / Auto-Repair)" "P0"
  1032. elif [[ "$REMAP_PENDING_TOTAL" -gt 0 ]]; then
  1033. bad "GPU row-remap PENDING — $REMAP_PENDING_TOTAL row(s) awaiting reset"
  1034. info " Pending remaps indicate marginal memory that a GPU reset/reboot should finalize."
  1035. info " If pending persists across reboots, the firmware may be stuck (known edge case) — escalate."
  1036. add_issue "GPU row-remap pending on $NODE_TO_PROBE (reset/reboot to finalize; if stuck, marginal memory) → references/node-diagnostics-detail.md § G (GPU/Accelerator) + § F (Hardware / Auto-Repair)" "P1"
  1037. elif [[ "$REMAP_UNCORRECT_TOTAL" -gt 0 ]]; then
  1038. warn "GPU has $REMAP_UNCORRECT_TOTAL uncorrectable remapped rows (healthy now, but history of faults)"
  1039. fi
  1040. fi
  1041. GPU_DCGM_LINES=$(extract_section GPU_DCGM GPU_DCGM_LOGS | { grep -v "^NO_DCGMI$" || true; } | { grep -v "^DCGM_UNAVAILABLE$" || true; })
  1042. if [[ -n "$GPU_DCGM_LINES" ]]; then
  1043. if echo "$GPU_DCGM_LINES" | grep -qiE '"overall_health"\s*:\s*"(Fail|Warn)"|HEALTH_RESULT_FAIL|HEALTH_RESULT_WARN|Health Monitor Report.*(Fail|Warn)'; then
  1044. bad "DCGM health check reported Fail/Warn on $NODE_TO_PROBE"
  1045. add_issue "DCGM health Fail/Warn on $NODE_TO_PROBE → references/node-diagnostics-detail.md § G (GPU/Accelerator)" "P0"
  1046. fi
  1047. fi
  1048. # DCGM log presence — informational. Confirms deep-health-check history.
  1049. GPU_DCGM_LOG_LINES=$(extract_section GPU_DCGM_LOGS KERNEL_PANIC)
  1050. if echo "$GPU_DCGM_LOG_LINES" | grep -qi "nvvs"; then
  1051. ok "DCGM nvvs logs present on $NODE_TO_PROBE (/var/log/nvidia-dcgm/)"
  1052. if echo "$GPU_DCGM_LOG_LINES" | grep -qE "^--- tail"; then
  1053. DCGM_TAIL=$(echo "$GPU_DCGM_LOG_LINES" | sed -n '/^--- tail/,$p' | head -20)
  1054. if echo "$DCGM_TAIL" | grep -qiE 'FAIL|Error:|row ?remap.*(pending|fail)'; then
  1055. warn "DCGM nvvs log tail contains failure/row-remap signals — inspect on node:"
  1056. echo "$DCGM_TAIL" | while IFS= read -r line; do info " $line"; done
  1057. add_issue "DCGM nvvs log shows failure/row-remap signals on $NODE_TO_PROBE → references/node-diagnostics-detail.md § G (GPU/Accelerator)" "P0"
  1058. fi
  1059. fi
  1060. fi
  1061. # Kernel panic / watchdog / hung task signals — indicate node-level instability.
  1062. KERNEL_PANIC_LINES=$(extract_section KERNEL_PANIC CONTAINERD | { grep -v "^NONE$" || true; } | head -5)
  1063. if [[ -n "$KERNEL_PANIC_LINES" ]]; then
  1064. echo ""
  1065. bad "Kernel panic / watchdog / hung_task signals on node $NODE_TO_PROBE:"
  1066. echo "$KERNEL_PANIC_LINES" | while IFS= read -r line; do info " $line"; done
  1067. add_issue "Kernel panic / watchdog on $NODE_TO_PROBE → references/node-diagnostics-detail.md § N (Kernel & System)" "P0"
  1068. fi
  1069. # containerd health — if the runtime is inactive, every pod on this node fails.
  1070. CONTAINERD_STATUS=$(extract_section CONTAINERD RESOURCE_CHECK_END | head -1)
  1071. if [[ "$CONTAINERD_STATUS" == "inactive" ]]; then
  1072. warn "containerd is inactive on $NODE_TO_PROBE — all pods on this node will fail"
  1073. add_issue "containerd inactive on $NODE_TO_PROBE → references/node-diagnostics-detail.md § M (Container Runtime)" "P0"
  1074. fi
  1075. else
  1076. warn "SSM command returned no output — node may not be reachable"
  1077. add_issue "Cannot reach node $NODE_TO_PROBE via SSM → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P1"
  1078. fi
  1079. else
  1080. if [[ -z "$NODE_TO_PROBE" ]]; then
  1081. info "No running nodes to probe for resource checks"
  1082. else
  1083. info "SSM plugin not installed — skipping on-node resource checks → references/node-diagnostics-detail.md § K (Node Access via SSM)"
  1084. fi
  1085. fi
  1086. if [[ "$ORCHESTRATOR" == "Slurm" && "$TOTAL_NODES" -gt 0 ]]; then
  1087. section "8b. Slurm Node Mapping"
  1088. info "Slurm node name → HyperPod instance ID mapping:"
  1089. info "(PrivateDnsHostname is not in list-cluster-nodes; use 'describe-cluster-node --node-id <i-...>' to retrieve it for a specific instance.)"
  1090. echo ""
  1091. echo "$NODES_JSON" | python3 -c "
  1092. import sys,json
  1093. nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
  1094. print(f' {\"Instance ID\":<22} {\"Group\":<20} {\"Type\":<22} {\"Status\"}')
  1095. print(f' {\"─\"*22} {\"─\"*20} {\"─\"*22} {\"─\"*10}')
  1096. for n in nodes[:20]:
  1097. iid=n.get('InstanceId','?')
  1098. grp=n.get('InstanceGroupName','?')
  1099. itype=n.get('InstanceType','?')
  1100. st=n.get('InstanceStatus',{}).get('Status','?')
  1101. print(f' {iid:<22} {grp:<20} {itype:<22} {st}')
  1102. if len(nodes) > 20:
  1103. print(f' ... and {len(nodes)-20} more nodes')
  1104. " 2>/dev/null
  1105. echo ""
  1106. info "To get PrivateDnsHostname for a specific instance: aws sagemaker describe-cluster-node --cluster-name $CLUSTER --region $REGION --node-id <i-...> --query 'NodeDetails.PrivateDnsHostname' --output text"
  1107. fi
  1108. section "9. Triage Summary"
  1109. echo ""
  1110. if [[ ${#ISSUES_FOUND[@]} -eq 0 ]]; then
  1111. echo -e " ${GREEN}${BOLD}No critical issues detected from available signals.${NC}"
  1112. echo ""
  1113. echo " Next steps:"
  1114. echo " • If cluster is still failing: check cluster events above for error details"
  1115. echo " • For node-level issues: use hyperpod-ssm skill to inspect nodes directly"
  1116. echo " • For EFA issues: bash scripts/check-efa-sg.sh --cluster ${CLUSTER} --region ${REGION}"
  1117. else
  1118. echo -e " ${RED}${BOLD}Issues found (${#ISSUES_FOUND[@]}):${NC}"
  1119. echo ""
  1120. for priority in P0 P1 P2; do
  1121. has_items=false
  1122. for issue in "${ISSUES_FOUND[@]}"; do
  1123. if [[ "$issue" == "${priority}|"* ]]; then
  1124. if ! "$has_items"; then
  1125. case "$priority" in
  1126. P0) echo -e " ${RED}${BOLD}[$priority — Fix Immediately]${NC}" ;;
  1127. P1) echo -e " ${YELLOW}${BOLD}[$priority — Fix Soon]${NC}" ;;
  1128. P2) echo -e " ${BOLD}[$priority — Informational]${NC}" ;;
  1129. esac
  1130. has_items=true
  1131. fi
  1132. echo -e " → ${issue#*|}"
  1133. fi
  1134. done
  1135. done
  1136. echo ""
  1137. echo -e " ${BOLD}Recommended next steps:${NC}"
  1138. echo " 1. Address P0 issues first, then P1. Each issue above includes a"
  1139. echo " pointer of the form '→ references/node-diagnostics-detail.md § X'."
  1140. echo " 2. The hyperpod-node-debugger skill will open the referenced section"
  1141. echo " and guide you through the fix with explicit approval."
  1142. echo " 3. After fixing, re-run: bash scripts/triage-cluster.sh --cluster ${CLUSTER} --region ${REGION}"
  1143. echo " 4. For shell access on nodes, use the hyperpod-ssm skill."
  1144. fi
  1145. echo ""
  1146. echo -e "${BOLD}Cluster: ${CLUSTER} | Region: ${REGION} | Orchestrator: ${ORCHESTRATOR}${NC}"
  1147. echo ""
  1148. # Exit 1 only on critical (P0/P1) issues so CI / retry loops don't fail on
  1149. # P2 informational findings. Fatal prerequisite failures exit 1 earlier at
  1150. # argument-validation time.
  1151. _critical=0
  1152. for _issue in "${ISSUES_FOUND[@]}"; do
  1153. case "${_issue%%|*}" in P0|P1) _critical=$((_critical+1)) ;; esac
  1154. done
  1155. [[ "$_critical" -eq 0 ]] && exit 0 || exit 1
plugins/sagemaker-ai/skills/hyperpod-slurm-debugger/scripts/slurm-diagnose.sh · Bundled script · 802 lines · 2 findings marked
  1. #!/usr/bin/env bash
  2. # slurm-diagnose.sh
  3. #
  4. # Read-only diagnostic for Slurm node-management issues on Amazon SageMaker HyperPod
  5. # Slurm clusters. Covers the scenarios documented in the HyperPod troubleshooting guide:
  6. #
  7. # A. Node DOWN / not responding
  8. # B. Node DOWN with reason "Node unexpectedly rebooted"
  9. # C. Controller state — slurmctld desync, plus the two folded triggers:
  10. # C (slurmdbd): accounting daemon connectivity
  11. # C (config): pending slurm.conf reconfiguration
  12. # D. Auto-recovery reason-string mismatches (Action:Reboot / Action:Replace)
  13. # E. HyperPod --auto-resume support and recent missed-resume detection
  14. #
  15. # Security model:
  16. # - All CLI inputs are validated against strict regexes at parse time.
  17. # - All AWS-derived values (instance IDs, group names, node names) are validated before
  18. # they reach any shell context — invalid values cause an immediate exit.
  19. # - Remote SSM payloads are base64-encoded literals; server-derived values are
  20. # prepended to the remote script as `export VAR='<jq @sh-quoted VALUE>'` lines so
  21. # they are never string-interpolated into shell commands.
  22. # - Local printf calls use `%s` with the data as a separate argument; format-string
  23. # attacks via server values are not possible.
  24. #
  25. # The script never mutates cluster state.
  26. #
  27. # Usage:
  28. # bash slurm-diagnose.sh --cluster <NAME-or-ARN> --region <REGION>
  29. # bash slurm-diagnose.sh --cluster <N> --region <R> --node <SLURM_NODE>
  30. # bash slurm-diagnose.sh --cluster <N> --region <R> --controller-group <NAME>
  31. #
  32. # Optional flags:
  33. # --node <SLURM_NODE> Scope inspection to a single Slurm node.
  34. # --controller-group <N> Override controller-group discovery (for self-managed
  35. # Slurm clusters where SlurmConfig is not set).
  36. # --no-color Plain output (no ANSI colors).
  37. set -euo pipefail
  38. CLUSTER=""
  39. REGION="${AWS_DEFAULT_REGION:-us-east-1}"
  40. TARGET_NODE=""
  41. CONTROLLER_GROUP_OVERRIDE=""
  42. USE_COLOR=true
  43. # --- Input-validation helpers -------------------------------------------------
  44. # Each validator prints the value if valid, exits non-zero if not. All callsites
  45. # capture into a local variable; failure aborts the script via `set -e`.
  46. # AWS region: lowercase letters, digits, dashes only.
  47. validate_region() {
  48. local v="${1-}"
  49. [[ "$v" =~ ^[a-z]{2,3}-[a-z]+-[0-9]+$ ]] || { echo "Error: invalid region: $v" >&2; exit 2; }
  50. printf '%s' "$v"
  51. }
  52. # HyperPod cluster name OR ARN. Names are 1-63 chars of [a-zA-Z0-9_-]; ARNs match the
  53. # documented SageMaker cluster ARN shape.
  54. validate_cluster() {
  55. local v="${1-}"
  56. if [[ "$v" =~ ^arn:aws[a-zA-Z-]*:sagemaker:[a-z0-9-]+:[0-9]{12}:cluster/[a-zA-Z0-9-]+$ ]]; then
  57. printf '%s' "$v"
  58. elif [[ "$v" =~ ^[a-zA-Z0-9_-]{1,63}$ ]]; then
  59. printf '%s' "$v"
  60. else
  61. echo "Error: invalid cluster name/ARN: $v" >&2
  62. exit 2
  63. fi
  64. }
  65. # Slurm node names on HyperPod follow the `ip-x-x-x-x` form, but admins may rename.
  66. # Allow [a-zA-Z0-9._-]+ with length 1..253; reject anything that could escape a shell.
  67. validate_node_name() {
  68. local v="${1-}"
  69. [[ "$v" =~ ^[a-zA-Z0-9._-]{1,253}$ ]] || { echo "Error: invalid node name: $v" >&2; exit 2; }
  70. printf '%s' "$v"
  71. }
  72. # EC2 instance IDs: i- followed by 8 or 17 hex characters. Documented and stable.
  73. validate_instance_id() {
  74. local v="${1-}"
  75. [[ "$v" =~ ^i-[a-f0-9]{8}([a-f0-9]{9})?$ ]] || { echo "Error: invalid instance ID: $v" >&2; exit 2; }
  76. printf '%s' "$v"
  77. }
  78. # Cluster ID (from ARN): lowercase alphanumeric, currently 12 chars (e.g.
  79. # qrmv6xhralg4). Allow 4..32 to be future-tolerant.
  80. validate_cluster_id() {
  81. local v="${1-}"
  82. [[ "$v" =~ ^[a-z0-9]{4,32}$ ]] || { echo "Error: invalid cluster ID: $v" >&2; exit 2; }
  83. printf '%s' "$v"
  84. }
  85. # Instance group name: SageMaker allows 1..63 chars [a-zA-Z0-9_-] per the
  86. # CreateCluster API.
  87. validate_group_name() {
  88. local v="${1-}"
  89. [[ "$v" =~ ^[a-zA-Z0-9_-]{1,63}$ ]] || { echo "Error: invalid instance group name: $v" >&2; exit 2; }
  90. printf '%s' "$v"
  91. }
  92. # --- Argument parsing ---------------------------------------------------------
  93. while [[ $# -gt 0 ]]; do
  94. case "$1" in
  95. --cluster)
  96. [[ $# -lt 2 ]] && { echo "Error: --cluster requires a value" >&2; exit 2; }
  97. CLUSTER=$(validate_cluster "$2"); shift 2 ;;
  98. --region)
  99. [[ $# -lt 2 ]] && { echo "Error: --region requires a value" >&2; exit 2; }
  100. REGION=$(validate_region "$2"); shift 2 ;;
  101. --node)
  102. [[ $# -lt 2 ]] && { echo "Error: --node requires a value" >&2; exit 2; }
  103. TARGET_NODE=$(validate_node_name "$2"); shift 2 ;;
  104. --controller-group)
  105. [[ $# -lt 2 ]] && { echo "Error: --controller-group requires a value" >&2; exit 2; }
  106. CONTROLLER_GROUP_OVERRIDE=$(validate_group_name "$2"); shift 2 ;;
  107. --no-color) USE_COLOR=false; shift ;;
  108. -h|--help)
  109. # Print every leading-comment line at the top of this file (lines 2..N until the
  110. # first non-comment line). Robust against future header edits.
  111. awk 'NR==1{next} /^#/{sub(/^# ?/,""); print; next} {exit}' "$0"
  112. exit 0 ;;
  113. --*) echo "Error: unknown flag: $1" >&2; exit 2 ;;
  114. *) echo "Error: unexpected positional argument: $1" >&2; exit 2 ;;
  115. esac
  116. done
  117. [[ -z "$CLUSTER" ]] && { echo "Error: --cluster is required" >&2; exit 2; }
  118. REGION=$(validate_region "$REGION") # validate even when sourced from env default
  119. # --- Prerequisite checks ------------------------------------------------------
  120. command -v aws >/dev/null 2>&1 || { echo "Error: aws CLI is required (v2 recommended)." >&2; exit 1; }
  121. command -v jq >/dev/null 2>&1 || { echo "Error: jq is required. Install with your package manager." >&2; exit 1; }
  122. # `unbuffer` (from the `expect` package) attaches a PTY to aws ssm start-session, which
  123. # avoids a known race where session-manager-plugin closes stdout before flushing and the
  124. # caller sees "Cannot perform start session: EOF" with empty output. Without it, every
  125. # SSM command silently returns empty, causing every downstream check to misreport.
  126. command -v unbuffer >/dev/null 2>&1 || {
  127. echo "Error: unbuffer (from the 'expect' package) is required." >&2
  128. echo " Install: sudo yum install expect | sudo apt install expect | brew install expect" >&2
  129. exit 1
  130. }
  131. # --- Output formatting --------------------------------------------------------
  132. if "$USE_COLOR"; then
  133. RED=$'\033[0;31m'; GREEN=$'\033[0;32m'; YELLOW=$'\033[1;33m'
  134. CYAN=$'\033[0;36m'; BOLD=$'\033[1m'; NC=$'\033[0m'
  135. else
  136. RED=''; GREEN=''; YELLOW=''; CYAN=''; BOLD=''; NC=''
  137. fi
  138. # All status helpers use %s with the message as a separate arg — never embed message
  139. # text into the format string. Strip ANSI escape sequences from incoming server data
  140. # so a malicious or buggy upstream cannot rewrite the operator's terminal.
  141. _sanitize() {
  142. # Drop ANSI CSI sequences and bell, but leave printable UTF-8 alone.
  143. sed -e 's/\x1b\[[0-9;?]*[a-zA-Z]//g' -e 's/\x07//g' -e 's/\r$//' <<< "${1-}"
  144. }
  145. section() { printf '\n%s%s=== %s ===%s\n' "$BOLD" "$CYAN" "$(_sanitize "$1")" "$NC"; }
  146. ok() { printf ' %s[PASS]%s %s\n' "$GREEN" "$NC" "$(_sanitize "$1")"; }
  147. warn() { printf ' %s[WARN]%s %s\n' "$YELLOW" "$NC" "$(_sanitize "$1")"; }
  148. bad() { printf ' %s[FAIL]%s %s\n' "$RED" "$NC" "$(_sanitize "$1")"; }
  149. info() { printf ' %s\n' "$(_sanitize "$1")"; }
  150. hint() { printf ' %s[NEXT]%s %s\n' "$CYAN" "$NC" "$(_sanitize "$1")"; }
  151. ISSUES=()
  152. NEXT_STEPS=()
  153. # --- Verify cluster + orchestrator --------------------------------------------
  154. section "1. Cluster identity"
  155. DESC=$(aws sagemaker describe-cluster --cluster-name "$CLUSTER" --region "$REGION" \
  156. --output json 2>&1) || { bad "cannot describe cluster: $DESC"; exit 1; }
  157. ORCH=$(jq -r '.Orchestrator // {} | keys[0] // "Slurm"' <<< "$DESC")
  158. if [[ "$ORCH" == "Eks" ]]; then
  159. bad "cluster uses EKS orchestrator - this skill is for Slurm only"
  160. info "use hyperpod-node-debugger or hyperpod-nccl instead"
  161. exit 1
  162. fi
  163. # Managed Slurm vs self-managed Slurm:
  164. # - Managed: DescribeCluster.Orchestrator.Slurm is present AND the cluster was created
  165. # with the SlurmConfig API parameter — InstanceGroups[].SlurmConfig.NodeType identifies
  166. # controllers, login nodes, workers. AWS docs treat this as the authoritative source.
  167. # - Self-managed: anything else. The customer brought their own Slurm setup via the
  168. # lifecycle scripts and InstanceGroups[].SlurmConfig is empty. The controller-group
  169. # name lives in /opt/ml/config/provisioning_parameters.json on every node, or the
  170. # customer can pass --controller-group <NAME>.
  171. HAS_SLURM_CONFIG=$(jq -r '
  172. any(.InstanceGroups[]?; (.SlurmConfig // {}) != {})
  173. ' <<< "$DESC")
  174. CLUSTER_NAME=$(jq -r '.ClusterName // "unknown"' <<< "$DESC")
  175. CLUSTER_STATUS=$(jq -r '.ClusterStatus // "unknown"' <<< "$DESC")
  176. if [[ "$HAS_SLURM_CONFIG" == "true" ]]; then
  177. ok "Managed Slurm cluster: $CLUSTER_NAME status=$CLUSTER_STATUS"
  178. else
  179. ok "Self-managed Slurm cluster: $CLUSTER_NAME status=$CLUSTER_STATUS"
  180. fi
  181. # Cluster ID from ARN. Validate before it gets embedded into SSM target strings.
  182. CLUSTER_ID=$(jq -r '.ClusterArn // "" | split("/") | last' <<< "$DESC")
  183. [[ -n "$CLUSTER_ID" ]] || { bad "cannot extract cluster ID from ARN"; exit 1; }
  184. CLUSTER_ID=$(validate_cluster_id "$CLUSTER_ID")
  185. # --- SSM remote-execution helper ----------------------------------------------
  186. #
  187. # `ssm_run` runs a command on a HyperPod node via SSM (read-only).
  188. #
  189. # Design notes:
  190. # 1. The remote script is base64-encoded locally and decoded remotely. The agent's
  191. # command parameter is a fixed `sh -c "echo <BASE64> | base64 -d | bash"`; the
    • Note: Quoted as data in code: Mentions decoding base64 into a shell (no payload shown)
    • Line 212 on GitHub
  192. # base64 string contains only [A-Za-z0-9+/=] and is safe inside double quotes.
  193. # Nothing from the script's caller appears unescaped in the SSM-agent's argv.
  194. # 2. Server-derived values that need to be visible to the remote script are passed
  195. # as named environment variables (`VAR=VALUE` trailing args). Each value is run
  196. # through `jq @sh` (single-quoted shell-safe encoding with `'\''` escapes) and
  197. # prepended to the remote script as `export VAR='<safely-quoted>'; ...`. The remote
  198. # shell reads them as `$NODE`, `$NODELIST`, etc. — values never reach a remote
  199. # shell-eval context as raw interpolated text.
  200. # 3. `unbuffer` is required to defeat the SSM "Cannot perform start session: EOF"
  201. # race; the prerequisite check above guarantees it's present.
  202. # 4. Returns the underlying aws-cli exit code so callers can distinguish transport
  203. # failures from successful empty output.
  204. #
  205. # Usage:
  206. # ssm_run TARGET REMOTE_SCRIPT [VAR=VALUE ...]
  207. ssm_run() {
  208. local target="$1"; shift
  209. local script="$1"; shift
  210. local export_block="" raw_kv key val safe_val
  211. for raw_kv in "$@"; do
  212. [[ "$raw_kv" =~ ^([A-Za-z_][A-Za-z0-9_]*)=(.*)$ ]] || {
  213. echo "ssm_run: invalid VAR=VALUE: $raw_kv" >&2
  214. return 2
  215. }
  216. key="${BASH_REMATCH[1]}"
  217. val="${BASH_REMATCH[2]}"
  218. # jq's @sh produces single-quoted shell-safe text with embedded `'\''` escapes.
  219. safe_val=$(jq -nr --arg v "$val" '$v | @sh')
  220. export_block+="export ${key}=${safe_val}; "
  221. done
  222. local full_script="${export_block}${script}"
  223. local b64
  224. if base64 --help 2>&1 | grep -q '\-w'; then
  225. b64=$(printf '%s' "$full_script" | base64 -w0)
  226. else
  227. b64=$(printf '%s' "$full_script" | base64 -b0)
  228. fi
  229. local wrapper="sh -c \"echo $b64 | base64 -d | bash\""
    • Note: Quoted as data in code: Decodes base64 and runs the result in a shell
    • Line 250 on GitHub
  230. local params
  231. params=$(jq -nc --arg c "$wrapper" '{command: [$c]}')
  232. local out rc=0
  233. out=$(unbuffer aws ssm start-session --region "$REGION" --target "$target" \
  234. --document-name AWS-StartNonInteractiveCommand \
  235. --parameters "$params" 2>&1) || rc=$?
  236. # NOTE: do NOT strip 'Cannot perform start session' here — that line is the
  237. # SSM transport-failure signal that ssm_transport_failed() detects. Only filter
  238. # benign session chrome ('Starting session' / 'Exiting session') and ANSI escapes.
  239. printf '%s' "$out" \
  240. | sed -e 's/\x1b\[[0-9;?]*[a-zA-Z]//g' \
  241. -e '/^Starting session/d' \
  242. -e '/^Exiting session/d'
  243. return "$rc"
  244. }
  245. # Returns 0 if the SSM raw output indicates a transport-layer failure (no command
  246. # output, session refused, EOF before flush) — distinct from "command ran and returned
  247. # nothing." Used to bail out early rather than misreport every downstream check.
  248. ssm_transport_failed() {
  249. local raw="${1-}"
  250. grep -qiE 'Cannot perform start session|TargetNotConnected|InvalidTarget|AccessDeniedException|UnauthorizedOperation' <<< "$raw"
  251. }
  252. # --- Find controller node -----------------------------------------------------
  253. NODES_JSON=$(aws sagemaker list-cluster-nodes --cluster-name "$CLUSTER" --region "$REGION" \
  254. --output json 2>&1) || { bad "list-cluster-nodes failed: $NODES_JSON"; exit 1; }
  255. # Discovery priority:
  256. # 1. --controller-group <NAME> (operator override — always wins)
  257. # 2. InstanceGroups[].SlurmConfig.NodeType == "Controller" (managed-Slurm authoritative)
  258. # 3. /opt/ml/config/provisioning_parameters.json on a probe node (self-managed fallback)
  259. # 4. Refuse to guess — print available groups and exit.
  260. # We never guess based on instance-group naming — that's a lifecycle-script convention,
  261. # not a guarantee, and getting it wrong sends every command to a non-controller.
  262. CONTROLLER_GROUP=""
  263. CONTROLLER_DISCOVERY_METHOD=""
  264. # (1) Operator override — always wins.
  265. if [[ -n "$CONTROLLER_GROUP_OVERRIDE" ]]; then
  266. CONTROLLER_GROUP="$CONTROLLER_GROUP_OVERRIDE"
  267. CONTROLLER_DISCOVERY_METHOD="--controller-group flag"
  268. fi
  269. # (2) Managed-Slurm authoritative source.
  270. if [[ -z "$CONTROLLER_GROUP" && "$HAS_SLURM_CONFIG" == "true" ]]; then
  271. CONTROLLER_GROUP=$(jq -r '
  272. .InstanceGroups[]?
  273. | select((.SlurmConfig.NodeType // "") == "Controller")
  274. | .InstanceGroupName' <<< "$DESC" | head -1)
  275. if [[ -n "$CONTROLLER_GROUP" ]]; then
  276. CONTROLLER_DISCOVERY_METHOD="DescribeCluster.SlurmConfig"
  277. fi
  278. fi
  279. # (3) Self-managed: read provisioning_parameters.json from any node.
  280. # The lifecycle-script convention is that this file is dropped at the same path on every
  281. # node, so we pick any node arbitrarily, SSM in, and read the controller_group field.
  282. if [[ -z "$CONTROLLER_GROUP" ]]; then
  283. PROBE_ID=$(jq -r '.ClusterNodeSummaries[0].InstanceId // ""' <<< "$NODES_JSON")
  284. PROBE_GROUP=$(jq -r '.ClusterNodeSummaries[0].InstanceGroupName // ""' <<< "$NODES_JSON")
  285. if [[ -n "$PROBE_ID" && -n "$PROBE_GROUP" ]]; then
  286. PROBE_ID_V=$(validate_instance_id "$PROBE_ID")
  287. PROBE_GROUP_V=$(validate_group_name "$PROBE_GROUP")
  288. PROBE_TARGET="sagemaker-cluster:${CLUSTER_ID}_${PROBE_GROUP_V}-${PROBE_ID_V}"
  289. # Field name varies between lifecycle-script generations — try both.
  290. PROV_GROUP=$(ssm_run "$PROBE_TARGET" \
  291. 'jq -r ".controller_group // .ControllerGroup // empty" /opt/ml/config/provisioning_parameters.json 2>/dev/null' \
  292. 2>/dev/null | tr -d '\r\n' || true)
  293. if [[ -n "$PROV_GROUP" ]]; then
  294. MATCHED=$(jq -r --arg g "$PROV_GROUP" \
  295. '[.ClusterNodeSummaries[]? | select(.InstanceGroupName == $g)] | length' <<< "$NODES_JSON")
  296. if [[ "$MATCHED" -gt 0 ]]; then
  297. CONTROLLER_GROUP="$PROV_GROUP"
  298. CONTROLLER_DISCOVERY_METHOD="provisioning_parameters.json on $PROBE_ID_V"
  299. fi
  300. fi
  301. fi
  302. fi
  303. # (4) Out of options — refuse to guess. Tell the operator how to unblock.
  304. if [[ -z "$CONTROLLER_GROUP" ]]; then
  305. bad "cannot identify the Slurm controller instance group"
  306. if [[ "$HAS_SLURM_CONFIG" == "true" ]]; then
  307. info "no InstanceGroup has SlurmConfig.NodeType=Controller in DescribeCluster output"
  308. info "this is unexpected for a managed-Slurm cluster — verify the cluster was"
  309. info "created with the SlurmConfig parameter, or pass --controller-group <NAME>."
  310. else
  311. info "self-managed Slurm cluster — provisioning_parameters.json was not readable"
  312. info "from a probe node, and no --controller-group flag was provided."
  313. info ""
  314. info "Resolve by either:"
  315. info " 1. inspecting the head node manually:"
  316. info " aws ssm start-session --target $PROBE_TARGET --region $REGION"
  317. info " cat /opt/ml/config/provisioning_parameters.json | jq ."
  318. info " 2. re-running with the controller group's name:"
  319. info " --controller-group <INSTANCE_GROUP_NAME>"
  320. info ""
  321. info "Available instance groups in this cluster:"
  322. jq -r '.ClusterNodeSummaries[] | " - " + .InstanceGroupName + " (" + .InstanceId + ")"' \
  323. <<< "$NODES_JSON" | sort -u
  324. fi
  325. exit 1
  326. fi
  327. CONTROLLER_GROUP=$(validate_group_name "$CONTROLLER_GROUP")
  328. # Pick the first node from the controller group.
  329. CONTROLLER_ID=$(jq -r --arg g "$CONTROLLER_GROUP" \
  330. '.ClusterNodeSummaries[]? | select(.InstanceGroupName == $g) | .InstanceId' <<< "$NODES_JSON" | head -1)
  331. [[ -n "$CONTROLLER_ID" ]] || { bad "controller group $CONTROLLER_GROUP has no nodes"; exit 1; }
  332. CONTROLLER_ID=$(validate_instance_id "$CONTROLLER_ID")
  333. ok "controller node: $CONTROLLER_ID (group=$CONTROLLER_GROUP, source=$CONTROLLER_DISCOVERY_METHOD)"
  334. SSM_HEAD="sagemaker-cluster:${CLUSTER_ID}_${CONTROLLER_GROUP}-${CONTROLLER_ID}"
  335. # --- Collect Slurm state from head node ---------------------------------------
  336. section "2. Slurm cluster state (from head node)"
  337. SSM_PROBE=$(ssm_run "$SSM_HEAD" 'echo SSM_OK' || true)
  338. if ! grep -q '^SSM_OK$' <<< "$SSM_PROBE"; then
  339. bad "cannot reach head node via SSM — every downstream check would be unreliable"
  340. if ssm_transport_failed "$SSM_PROBE"; then
  341. info " transport error detected (TargetNotConnected, AccessDenied, or EOF race)"
  342. fi
  343. info " reproduce manually with the same target and region:"
  344. info " aws ssm start-session --target $SSM_HEAD --region $REGION"
  345. info " if that fails, route to the hyperpod-ssm skill before retrying."
  346. exit 1
  347. fi
  348. ok "SSM transport to head node working"
  349. SINFO_OUT=$(ssm_run "$SSM_HEAD" 'sinfo -h -o "%N|%T|%E" 2>&1 | head -200' || true)
  350. if [[ $(printf '%s\n' "$SINFO_OUT" | wc -l) -ge 200 ]]; then
  351. warn "sinfo output reached the 200-line cap — node-state results may be truncated on this large cluster"
  352. fi
  353. if grep -qi 'command not found' <<< "$SINFO_OUT"; then
  354. bad "sinfo not installed on head node — Slurm lifecycle script may not have run"
  355. info "verify on the node: systemctl status slurmctld; ls /opt/slurm*/etc /etc/slurm 2>/dev/null"
  356. exit 1
  357. fi
  358. if [[ -z "$SINFO_OUT" ]]; then
  359. warn "sinfo returned no rows — empty cluster, or controller not yet responding"
  360. fi
  361. # Parse sinfo lines. Node names from sinfo are server-controlled; validate before they
  362. # can be embedded into any later command. Values that fail validation are dropped, not
  363. # trusted; we report the count of skipped entries so the operator notices.
  364. DOWN_NODES=()
  365. REBOOT_NODES=()
  366. FAIL_NODES=()
  367. BAD_REASON_NODES=()
  368. SKIPPED_INVALID=0
  369. while IFS='|' read -r node state reason; do
  370. [[ -z "$node" ]] && continue
  371. if ! [[ "$node" =~ ^[a-zA-Z0-9._-]{1,253}$ ]]; then
  372. SKIPPED_INVALID=$((SKIPPED_INVALID+1))
  373. continue
  374. fi
  375. # Reasons can contain spaces and punctuation; allow them but strip ANSI/control chars.
  376. reason="$(_sanitize "$reason")"
  377. if grep -qi 'fail' <<< "$state"; then
  378. if [[ "$reason" =~ ^Action:(Reboot|Replace)$ ]]; then
  379. FAIL_NODES+=("$node|$reason")
  380. elif grep -qiE 'action[ :_-]*re(boot|place)|reboot|replace' <<< "$reason"; then
  381. BAD_REASON_NODES+=("$node|$reason")
  382. fi
  383. fi
  384. if grep -qiE 'down|drain' <<< "$state"; then
  385. if grep -qi 'unexpectedly rebooted' <<< "$reason"; then
  386. REBOOT_NODES+=("$node")
  387. else
  388. DOWN_NODES+=("$node|$reason")
  389. fi
  390. fi
  391. done <<< "$SINFO_OUT"
  392. [[ "$SKIPPED_INVALID" -gt 0 ]] && warn "$SKIPPED_INVALID sinfo row(s) had invalid node names and were ignored"
  393. if [[ ${#DOWN_NODES[@]} -eq 0 && ${#REBOOT_NODES[@]} -eq 0 && ${#FAIL_NODES[@]} -eq 0 && ${#BAD_REASON_NODES[@]} -eq 0 ]]; then
  394. ok "all nodes in healthy Slurm states"
  395. else
  396. [[ ${#DOWN_NODES[@]} -gt 0 ]] && bad "${#DOWN_NODES[@]} node(s) DOWN/DRAIN (Section A)"
  397. [[ ${#REBOOT_NODES[@]} -gt 0 ]] && bad "${#REBOOT_NODES[@]} node(s) with 'unexpectedly rebooted' (Section B)"
  398. [[ ${#FAIL_NODES[@]} -gt 0 ]] && warn "${#FAIL_NODES[@]} node(s) in fail state with valid Action:* reason (HyperPod recovery in progress)"
  399. [[ ${#BAD_REASON_NODES[@]} -gt 0 ]] && bad "${#BAD_REASON_NODES[@]} node(s) in fail state with non-matching reason (Section D)"
  400. fi
  401. # --- Section D: Action:* reason-string validation -----------------------------
  402. if [[ ${#BAD_REASON_NODES[@]} -gt 0 ]]; then
  403. section "D. Reason-string mismatch — HyperPod auto-recovery will NOT trigger"
  404. for entry in "${BAD_REASON_NODES[@]}"; do
  405. n="${entry%%|*}"; r="${entry#*|}"
  406. bad "$n: reason='$r'"
  407. done
  408. info "the reason field must match exactly: Action:Reboot or Action:Replace"
  409. info "(case-sensitive, no spaces, no trailing punctuation)"
  410. hint "for re-issue procedure, see:"
  411. info " https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-replace-faulty-instance.html"
  412. info " references/slurm-details.md#action-reason-string-validation"
  413. ISSUES+=("bad-action-reason")
  414. NEXT_STEPS+=("see AWS replace-faulty-instance docs (link above)")
  415. fi
  416. # --- Detect in-progress HyperPod replacements (informational) -----------------
  417. if [[ ${#FAIL_NODES[@]} -gt 0 ]]; then
  418. section " HyperPod recovery in progress (do not interfere)"
  419. for entry in "${FAIL_NODES[@]}"; do
  420. n="${entry%%|*}"; r="${entry#*|}"
  421. info "$n ($r)"
  422. done
  423. info "AWS docs: do NOT change node state or restart slurmctld until this completes."
  424. info "If a replacement seems stuck > 30 min, see:"
  425. info " https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-replace-faulty-instance.html"
  426. fi
  427. # --- Check controller health --------------------------------------------------
  428. section "3. slurmctld health"
  429. PING_OUT=$(ssm_run "$SSM_HEAD" 'scontrol ping 2>&1' || true)
  430. PING_FIRST_LINE=$(head -1 <<< "$PING_OUT" | tr -d '\r')
  431. if grep -qi 'UP' <<< "$PING_OUT"; then
  432. ok "slurmctld responding: $(tr '\n' ' ' <<< "$PING_OUT")"
  433. elif [[ -z "$PING_OUT" ]] || ssm_transport_failed "$PING_OUT"; then
  434. warn "could not get a response from scontrol ping — cannot determine controller health"
  435. info "this is most likely an SSM transport problem, not a hung controller"
  436. info "do NOT restart slurmctld based on this finding alone"
  437. elif grep -qi 'DOWN' <<< "$PING_OUT"; then
  438. bad "slurmctld reports DOWN: $PING_FIRST_LINE"
  439. ISSUES+=("controller-hung")
  440. NEXT_STEPS+=("controller restart — see references/slurm-details.md#-c-controller-state--diagnostic-context")
  441. else
  442. bad "slurmctld responded with an unrecognized status: $PING_FIRST_LINE"
  443. ISSUES+=("controller-hung")
  444. NEXT_STEPS+=("inspect logs first; controller restart only if logs confirm a hang")
  445. fi
  446. # --- Section C-1: slurmdbd connectivity (controller-state restart trigger) ---
  447. section "C (slurmdbd): accounting daemon connectivity"
  448. DBD_OUT=$(ssm_run "$SSM_HEAD" 'sacctmgr -i show stats 2>&1 | head -20' || true)
  449. if grep -qiE 'unable to contact|connection refused|cannot connect|no slurmdbd' <<< "$DBD_OUT"; then
  450. bad "slurmctld cannot reach slurmdbd"
  451. info "$(head -3 <<< "$DBD_OUT")"
  452. hint "diagnostic and recovery procedure:"
  453. info " https://slurm.schedmd.com/accounting.html"
  454. info " references/slurm-details.md#slurmdbd-connectivity"
  455. ISSUES+=("slurmdbd-disconnected")
  456. NEXT_STEPS+=("restore slurmdbd connectivity (see AWS / Slurm docs linked above)")
  457. elif grep -qiE 'rollup|rpc' <<< "$DBD_OUT"; then
  458. ok "slurmdbd reachable"
  459. else
  460. warn "could not determine slurmdbd state from sacctmgr output"
  461. info "if accounting is configured, run on the head node: sacctmgr show stats"
  462. fi
  463. # --- Section C-2: pending slurm.conf reconfiguration (controller-state restart trigger) ---
  464. # HyperPod's slurm.conf lives at /opt/slurm-<version>/etc/slurm.conf rather than the
  465. # upstream /etc/slurm/slurm.conf, so the remote script asks scontrol where the live
  466. # config is. The output is a `<conf-mtime>|<ctld-start>|<conf-path>` line that we
  467. # match strictly with a regex before parsing.
  468. section "C (config): slurm.conf freshness"
  469. read -r -d '' F_REMOTE <<'REMOTE_F' || true
  470. set -e
  471. # nosemgrep: bash.lang.correctness.unquoted-expansion.unquoted-variable-expansion-in-command
  472. _CONF="$(scontrol show config 2>/dev/null | awk -F= '/^SLURM_CONF/ {gsub(/ /,"",$2); print $2; exit}')"
  473. CONF_MTIME=0
  474. if [ -n "$_CONF" ] && [ -r "$_CONF" ]; then
  475. CONF_MTIME=$(stat -c %Y "$_CONF" 2>/dev/null || echo 0)
  476. fi
  477. CTLD_TS=$(systemctl show slurmctld -p ActiveEnterTimestamp --value 2>/dev/null || true)
  478. CTLD_START=0
  479. if [ -n "$CTLD_TS" ]; then
  480. CTLD_START=$(date -d "$CTLD_TS" +%s 2>/dev/null || echo 0)
  481. fi
  482. printf 'F_RESULT|%s|%s|%s\n' "${CONF_MTIME}" "${CTLD_START}" "${_CONF}"
  483. REMOTE_F
  484. F_LINE=$(ssm_run "$SSM_HEAD" "$F_REMOTE" 2>/dev/null | grep -E '^F_RESULT\|[0-9]+\|[0-9]+\|' | head -1 || true)
  485. if [[ "$F_LINE" =~ ^F_RESULT\|([0-9]+)\|([0-9]+)\|(.*)$ ]]; then
  486. CONF_MTIME="${BASH_REMATCH[1]}"
  487. CTLD_START="${BASH_REMATCH[2]}"
  488. CONF_PATH="${BASH_REMATCH[3]}"
  489. # CONF_PATH must be a real-looking absolute path before we put it into operator-
  490. # facing recommendations. Reject anything that has shell-active characters.
  491. if ! [[ "$CONF_PATH" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
  492. warn "slurm.conf path returned by remote did not validate; skipping freshness check"
  493. elif [[ "$CONF_MTIME" -gt "$CTLD_START" && "$CTLD_START" -gt 0 ]]; then
  494. DELTA=$((CONF_MTIME - CTLD_START))
  495. warn "$CONF_PATH modified ${DELTA}s after slurmctld last started — config may be stale in memory"
  496. hint "for the reload-vs-restart decision and procedure, see:"
  497. info " https://slurm.schedmd.com/scontrol.html"
  498. info " https://slurm.schedmd.com/slurm.conf.html"
  499. info " references/slurm-details.md#scontrol-reconfigure-vs-restart"
  500. ISSUES+=("stale-conf")
  501. NEXT_STEPS+=("review reload procedure in linked docs")
  502. else
  503. ok "slurm.conf older than slurmctld start time — no pending reconfigure"
  504. fi
  505. else
  506. warn "could not determine slurm.conf vs slurmctld timestamps"
  507. fi
  508. # --- Check for stuck jobs -----------------------------------------------------
  509. section "4. Job queue health"
  510. SQUEUE_OUT=$(ssm_run "$SSM_HEAD" 'squeue -h -o "%i|%T|%r" 2>&1 | head -200' || true)
  511. if [[ $(printf '%s\n' "$SQUEUE_OUT" | wc -l) -ge 200 ]]; then
  512. warn "squeue output reached the 200-line cap — stuck-job counts below may underreport on this large cluster"
  513. fi
  514. STUCK_PENDING=0
  515. STUCK_COMPLETING=0
  516. while IFS='|' read -r jobid state reason; do
  517. [[ -z "$jobid" ]] && continue
  518. [[ "$state" == "PENDING" && "$reason" == "Resources" ]] && STUCK_PENDING=$((STUCK_PENDING+1))
  519. [[ "$state" == "COMPLETING" ]] && STUCK_COMPLETING=$((STUCK_COMPLETING+1))
  520. done <<< "$SQUEUE_OUT"
  521. if [[ $STUCK_PENDING -gt 0 ]]; then
  522. warn "$STUCK_PENDING job(s) PENDING with Reason=Resources"
  523. if [[ ${#DOWN_NODES[@]} -eq 0 ]]; then
  524. ISSUES+=("stuck-pending-with-idle-nodes")
  525. NEXT_STEPS+=("controller restart — Section C")
  526. fi
  527. fi
  528. if [[ $STUCK_COMPLETING -gt 0 ]]; then
  529. bad "$STUCK_COMPLETING job(s) stuck in COMPLETING"
  530. ISSUES+=("stuck-completing")
  531. NEXT_STEPS+=("controller restart — Section C")
  532. fi
  533. [[ $STUCK_PENDING -eq 0 && $STUCK_COMPLETING -eq 0 ]] && ok "no stuck jobs"
  534. # --- Per-node inspection (read-only) ------------------------------------------
  535. inspect_node() {
  536. local slurm_node="$1"
  537. # Defense-in-depth: validate again at the boundary even though all upstream paths
  538. # validate. Cheap, and catches future refactors that miss a callsite.
  539. slurm_node=$(validate_node_name "$slurm_node")
  540. local instance_id group ssm_target
  541. # PrivateDnsName looks like `ip-10-1-2-3.us-west-2.compute.internal`. The strict
  542. # `<name>.` match handles the default `ip-x-x-x-x` form and rejects the false
  543. # positive where node `ip-10-1-2-3` would otherwise also match
  544. # `ip-10-1-2-30.<region>.compute.internal`.
  545. instance_id=$(jq -r --arg dns "$slurm_node" '
  546. .ClusterNodeSummaries[]?
  547. | select((.PrivateDnsName // "") | startswith($dns + "."))
  548. | .InstanceId' <<< "$NODES_JSON" | head -1)
  549. if [[ -z "$instance_id" ]]; then
  550. if [[ ! "$slurm_node" =~ ^ip-[0-9]+-[0-9]+-[0-9]+-[0-9]+$ ]]; then
  551. warn "$slurm_node: not in the default ip-X-X-X-X form — Slurm-node-name → instance-ID auto-mapping needs DNS lookup or scontrol show node, neither cheap from here. Pass --target-instance-id <i-xxx> if you have it, or look up via 'scontrol show node $slurm_node | grep NodeAddr' on the controller."
  552. else
  553. warn "$slurm_node: cannot map to instance ID (PrivateDnsName mismatch — verify node is in this cluster)"
  554. fi
  555. return
  556. fi
  557. instance_id=$(validate_instance_id "$instance_id")
  558. group=$(jq -r --arg id "$instance_id" \
  559. '.ClusterNodeSummaries[] | select(.InstanceId==$id) | .InstanceGroupName // ""' <<< "$NODES_JSON")
  560. group=$(validate_group_name "$group")
  561. ssm_target="sagemaker-cluster:${CLUSTER_ID}_${group}-${instance_id}"
  562. local slurmd_status disk mem rpc_check
  563. slurmd_status=$(ssm_run "$ssm_target" 'systemctl is-active slurmd 2>&1' | tr -d '\r\n' || true)
  564. disk=$(ssm_run "$ssm_target" 'df -h / | awk "NR==2 {print \$5}"' | tr -d '\r\n' || true)
  565. mem=$(ssm_run "$ssm_target" 'free -h | awk "/Mem:/ {print \$3\"/\"\$2}"' | tr -d '\r\n' || true)
  566. # Slurm-RPC reachability: srun -w "$NODE" hostname. The remote script reads $NODE
  567. # from the environment, so the slurm node name is never string-interpolated into
  568. # the remote shell — it lives in env-var space the whole way.
  569. rpc_check=$(ssm_run "$SSM_HEAD" 'timeout 10 srun --immediate=5 -w "$NODE" hostname 2>&1 | tail -1' \
  570. "NODE=$slurm_node" | tr -d '\r\n' || true)
  571. info "$slurm_node ($instance_id): slurmd=$slurmd_status disk=$disk mem=$mem"
  572. info " srun RPC: ${rpc_check:-<no output>}"
  573. local disk_num="${disk%\%}"
  574. if [[ "$disk_num" =~ ^[0-9]+$ && "$disk_num" -ge 95 ]]; then
  575. bad " $slurm_node: root volume ${disk} — clean up before any restart"
  576. info " HyperPod storage layout: https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
  577. ISSUES+=("disk-full-$slurm_node")
  578. NEXT_STEPS+=("clean disk on $slurm_node before recovery")
  579. fi
  580. if [[ "$slurmd_status" != "active" ]]; then
  581. bad " $slurm_node: slurmd is '$slurmd_status'"
  582. info " for recovery procedure, see:"
  583. info " https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
  584. fi
  585. if [[ -n "$rpc_check" ]] && grep -qiE 'auth|munge|invalid' <<< "$rpc_check"; then
  586. bad " $slurm_node: srun reports auth/munge error — slurmd-controller trust broken"
  587. info " for munge troubleshooting, see Slurm authentication docs:"
  588. info " https://slurm.schedmd.com/authentication.html"
  589. fi
  590. }
  591. if [[ -n "$TARGET_NODE" ]]; then
  592. section "5. Inspecting node: $TARGET_NODE"
  593. inspect_node "$TARGET_NODE"
  594. elif [[ ${#DOWN_NODES[@]} -gt 0 || ${#REBOOT_NODES[@]} -gt 0 ]]; then
  595. section "5. Inspecting affected nodes"
  596. for entry in "${DOWN_NODES[@]-}"; do
  597. [[ -z "$entry" ]] && continue
  598. inspect_node "${entry%%|*}"
  599. done
  600. for n in "${REBOOT_NODES[@]-}"; do
  601. [[ -z "$n" ]] && continue
  602. inspect_node "$n"
  603. done
  604. fi
  605. # --- Section E: HyperPod auto-resume support + recent missed-resume detection ---
  606. section "E. Auto-resume support"
  607. AR_HELP=$(ssm_run "$SSM_HEAD" 'srun --help 2>&1 | grep -i auto-resume | head -3' || true)
  608. if [[ -n "$AR_HELP" ]]; then
  609. ok "srun --auto-resume is available on this cluster"
  610. else
  611. warn "srun --auto-resume not found in srun --help output"
  612. info "this AMI / Slurm build may predate HyperPod auto-resume support"
  613. info "see: references/slurm-details.md#hyperpod-auto-resume"
  614. ISSUES+=("auto-resume-unsupported")
  615. NEXT_STEPS+=("upgrade the cluster AMI / Slurm package to enable --auto-resume")
  616. fi
  617. read -r -d '' G_FAILS <<'REMOTE_G' || true
  618. sacct -X -n --starttime=now-6hours \
  619. -o JobID,State,ExitCode,NodeList \
  620. --state=NODE_FAIL,FAILED 2>/dev/null \
  621. | awk 'NF>=4 && $4!~/None/ {print $1"|"$2"|"$4}' | head -50
  622. REMOTE_G
  623. RECENT_FAILS=$(ssm_run "$SSM_HEAD" "$G_FAILS" 2>/dev/null || true)
  624. MISSED_AR=()
  625. NOW_EPOCH=$(date +%s)
  626. while IFS='|' read -r jobid state nodelist; do
  627. [[ -z "$jobid" ]] && continue
  628. # Only single-node failures — multi-node lists need a real range expander.
  629. [[ "$nodelist" == *,* || "$nodelist" == *\[* ]] && continue
  630. # Validate before passing to remote.
  631. if ! [[ "$nodelist" =~ ^[a-zA-Z0-9._-]{1,253}$ ]]; then
  632. continue
  633. fi
  634. # A successful HyperPod replace clears the node's Reason field once the new instance
  635. # registers, so grepping for "Action:Replace" is unreliable. Detect a recent replace
  636. # by comparing scontrol show node's BootTime to wall-clock: a fresh BootTime within
  637. # the last 6h that's later than the failed-job's End time strongly suggests the node
  638. # was replaced (or rebooted) after the job died.
  639. BOOT_LINE=$(ssm_run "$SSM_HEAD" 'scontrol show node "$NODE" 2>/dev/null | tr " " "\n" | grep "^BootTime="' \
  640. "NODE=$nodelist" | head -1 | tr -d '\r\n' || true)
  641. BOOT_STR="${BOOT_LINE#BootTime=}"
  642. [[ -z "$BOOT_STR" || "$BOOT_STR" == "Unknown" ]] && continue
  643. BOOT_EPOCH=$(date -d "$BOOT_STR" +%s 2>/dev/null || echo 0)
  644. [[ "$BOOT_EPOCH" =~ ^[0-9]+$ && "$BOOT_EPOCH" -gt 0 ]] || continue
  645. AGE=$((NOW_EPOCH - BOOT_EPOCH))
  646. if [[ $AGE -ge 0 && $AGE -le 21600 ]]; then # 6h window
  647. MISSED_AR+=("$jobid|$state|$nodelist|$BOOT_STR")
  648. fi
  649. done <<< "$RECENT_FAILS"
  650. if [[ ${#MISSED_AR[@]} -gt 0 ]]; then
  651. bad "${#MISSED_AR[@]} recent job(s) failed on a node that was rebooted/replaced shortly after — possible missed auto-resume:"
  652. for entry in "${MISSED_AR[@]}"; do
  653. IFS='|' read -r jobid state nodelist boot <<< "$entry"
  654. info " job $jobid ($state) on $nodelist (node BootTime=$boot)"
  655. done
  656. info "(heuristic: node BootTime is within the last 6h, suggesting a replace or reboot)"
  657. hint "verify the launch command used srun --auto-resume=1 (NOT just sbatch):"
  658. info " sacct -j <JOBID> -o JobID,JobName,Submit,Start,End,State,ExitCode,NodeList -X"
  659. info " scontrol show job <JOBID> # only if still in the controller's recent history"
  660. info "see: references/slurm-details.md#hyperpod-auto-resume"
  661. ISSUES+=("missed-auto-resume")
  662. NEXT_STEPS+=("verify --auto-resume=1 is on the srun line, not just sbatch")
  663. elif [[ -n "$RECENT_FAILS" ]]; then
  664. ok "recent failed jobs do not match the missed-auto-resume pattern"
  665. else
  666. ok "no recent NODE_FAIL / FAILED jobs in the last 6h"
  667. fi
  668. # --- Findings → documentation links ------------------------------------------
  669. # This skill is diagnostic-only. It never prints a remediation command. For each
  670. # finding, point the user at the authoritative doc and let them act.
  671. section "Where to read next"
  672. if [[ ${#REBOOT_NODES[@]} -gt 0 ]]; then
  673. hint "Section B — nodes flagged 'unexpectedly rebooted':"
  674. for n in "${REBOOT_NODES[@]}"; do
  675. info " $n"
  676. done
  677. info " HyperPod Slurm troubleshooting:"
  678. info " https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
  679. info " diagnostic context: references/slurm-details.md#-b-unexpected-reboot--diagnostic-context"
  680. fi
  681. if [[ ${#DOWN_NODES[@]} -gt 0 ]]; then
  682. hint "Section A — nodes DOWN/DRAIN:"
  683. for entry in "${DOWN_NODES[@]}"; do
  684. n="${entry%%|*}"; r="${entry#*|}"
  685. info " $n (reason: $r)"
  686. done
  687. info " HyperPod Slurm troubleshooting:"
  688. info " https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
  689. info " if the node flaps after a manual recovery → route to hyperpod-node-debugger"
  690. fi
  691. CTRL_RESTART_REASON=""
  692. ISSUES_STR=" ${ISSUES[*]-} "
  693. [[ "$ISSUES_STR" == *" controller-hung "* ]] && CTRL_RESTART_REASON="scontrol ping failed"
  694. [[ "$ISSUES_STR" == *" stuck-completing "* ]] && CTRL_RESTART_REASON="${CTRL_RESTART_REASON:+$CTRL_RESTART_REASON, }jobs stuck COMPLETING"
  695. [[ "$ISSUES_STR" == *" stuck-pending-with-idle-nodes "* ]] && CTRL_RESTART_REASON="${CTRL_RESTART_REASON:+$CTRL_RESTART_REASON, }jobs PENDING with idle nodes"
  696. if [[ -n "$CTRL_RESTART_REASON" ]]; then
  697. hint "Section C — controller-state issue ($CTRL_RESTART_REASON):"
  698. info " Slurm slurmctld(8) — for what is preserved across a controller restart:"
  699. info " https://slurm.schedmd.com/slurmctld.html"
  700. info " HyperPod Slurm troubleshooting:"
  701. info " https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
  702. if [[ ${#FAIL_NODES[@]} -gt 0 ]]; then
  703. warn "HyperPod recovery is in progress on:"
  704. for entry in "${FAIL_NODES[@]}"; do
  705. n="${entry%%|*}"
  706. info " $n"
  707. done
  708. info "AWS docs warn against changing node state or restarting slurmctld during a"
  709. info "replacement; wait for it to complete, then re-run this script."
  710. fi
  711. info " diagnostic context: references/slurm-details.md#-c-controller-state--diagnostic-context"
  712. fi
  713. if [[ "$ISSUES_STR" == *" missed-auto-resume "* ]]; then
  714. hint "Section E — recent job failed on a node that was later replaced:"
  715. info " the most common cause is --auto-resume on sbatch instead of srun."
  716. info " Use SageMaker HyperPod auto-resume:"
  717. info " https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-auto-resume.html"
  718. info " diagnostic context: references/slurm-details.md#hyperpod-auto-resume"
  719. fi
  720. # --- Summary ------------------------------------------------------------------
  721. section "Summary"
  722. printf ' Issues detected: %d\n' "${#ISSUES[@]-0}"
  723. if [[ ${#ISSUES[@]-0} -eq 0 ]]; then
  724. ok "cluster Slurm state is healthy"
  725. else
  726. echo ""
  727. echo " Findings:"
  728. for i in "${ISSUES[@]}"; do
  729. info "- $i"
  730. done
  731. fi
  732. if [[ ${#NEXT_STEPS[@]-0} -gt 0 ]]; then
  733. echo ""
  734. echo " Where to read next:"
  735. for s in "${NEXT_STEPS[@]}"; do
  736. info "- $s"
  737. done
  738. fi
plugins/sagemaker-ai/skills/hyperpod-version-checker/scripts/hyperpod_check_versions.sh · Bundled script · 556 lines · 2 findings marked
  1. #!/usr/bin/env bash
  2. # HyperPod Version Checker - Detect software component versions on HyperPod cluster nodes
  3. #
  4. # Checks: NVIDIA driver, CUDA, cuDNN, NCCL, EFA, AWS OFI NCCL, GDRCopy, MPI,
  5. # Neuron SDK, Python, PyTorch, container runtime
  6. # Works on both EKS and Slurm HyperPod clusters.
  7. #
  8. # Usage: bash hyperpod_check_versions.sh [--json] [--no-color] [--output FILE]
  9. command -v jq >/dev/null 2>&1 || { echo "Error: jq is required but not installed" >&2; exit 1; }
  10. # --- Defaults ---
  11. JSON_OUTPUT=false
  12. USE_COLOR=true
  13. OUTPUT_FILE=""
  14. # --- Parse args ---
  15. while [[ $# -gt 0 ]]; do
  16. case "$1" in
  17. --json) JSON_OUTPUT=true; shift ;;
  18. --no-color) USE_COLOR=false; shift ;;
  19. --output|-o) OUTPUT_FILE="$2"; shift 2 ;;
  20. -h|--help)
  21. echo "Usage: bash hyperpod_check_versions.sh [--json] [--no-color] [--output FILE]"
  22. echo " --json Output ONLY JSON to stdout (text report still saved to file)"
  23. echo " --no-color Disable color output"
  24. echo " --output/-o Write report to FILE (default: component_versions_<host>_<time>.txt)"
  25. exit 0 ;;
  26. *) echo "Unknown option: $1"; exit 1 ;;
  27. esac
  28. done
  29. # --- Color setup ---
  30. if [[ "$USE_COLOR" == "true" ]] && [ -t 1 ] && [[ "$JSON_OUTPUT" != "true" ]]; then
  31. GREEN='\033[0;32m'; YELLOW='\033[1;33m'; BLUE='\033[0;34m'; NC='\033[0m'
  32. else
  33. GREEN=''; YELLOW=''; BLUE=''; NC=''
  34. fi
  35. # --- Output file ---
  36. if [ -z "$OUTPUT_FILE" ]; then
  37. OUTPUT_FILE="component_versions_$(hostname)_$(date +%Y%m%d_%H%M%S).txt"
  38. fi
  39. # --- Helpers ---
  40. # In JSON mode: text goes only to file. Otherwise: both console and file.
  41. log() {
  42. local stripped
  43. stripped=$(printf '%b\n' "$@" | sed 's/\x1b\[[0-9;]*m//g')
  44. echo "$stripped" >> "$OUTPUT_FILE"
  45. if [[ "$JSON_OUTPUT" != "true" ]]; then
  46. echo -e "$@"
  47. fi
  48. }
  49. section() {
  50. log "${BLUE}========================================${NC}"
  51. log "${BLUE}$1${NC}"
  52. log "${BLUE}========================================${NC}"
  53. }
  54. cmd_exists() { command -v "$1" >/dev/null 2>&1; }
  55. cmd_or_path() { command -v "$1" 2>/dev/null || echo "$2"; }
  56. # Detect instance type via IMDS
  57. IMDS_TOKEN=$(curl -s -m 2 -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60" 2>/dev/null) || true
    • Note: Quoted as data in code: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
    • Line 67 on GitHub
  58. if [[ -z "$IMDS_TOKEN" ]]; then
  59. echo "Error: Failed to retrieve IMDS token (IMDSv2 endpoint unreachable)" >&2
  60. INSTANCE_TYPE=""
  61. else
  62. INSTANCE_TYPE=$(curl -s -m 2 -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" http://169.254.169.254/latest/meta-data/instance-type 2>/dev/null) || true
    • Dismissed after review: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
    • Line 72 on GitHub
  63. if [[ -z "$INSTANCE_TYPE" ]]; then
  64. echo "Error: Failed to retrieve instance type from IMDS" >&2
  65. fi
  66. fi
  67. IS_NEURON=false
  68. [[ "$INSTANCE_TYPE" =~ (^|\.)(trn|inf) ]] && IS_NEURON=true
  69. # GPU detection is driven by `cmd_exists nvidia-smi` at each GPU section below —
  70. # no explicit IS_GPU flag needed. Keeps GPU checks working on instances where
  71. # the driver is present but the regex would miss (e.g. new p-family SKUs).
  72. # JSON-safe string escape via jq (handles all special/unicode characters correctly)
  73. json_escape() { jq -rn --arg v "$1" '$v | @json | .[1:-1]'; }
  74. declare -A VERSIONS
  75. # --- System Information ---
  76. : > "$OUTPUT_FILE"
  77. section "System Information"
  78. log "Host: $(hostname)"
  79. log "Date: $(date)"
  80. log "OS: $(grep PRETTY_NAME /etc/os-release 2>/dev/null | cut -d'"' -f2)"
  81. log "Kernel: $(uname -r)"
  82. log "Architecture: $(uname -m)"
  83. log "Instance Type: ${INSTANCE_TYPE:-unknown}"
  84. log ""
  85. # --- NVIDIA Driver & CUDA ---
  86. section "CUDA Information"
  87. if cmd_exists nvidia-smi; then
  88. DRIVER_VER=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | head -1)
  89. if [ $? -ne 0 ] || [ -z "$DRIVER_VER" ] || [[ "$DRIVER_VER" == *"failed"* ]] || [[ "$DRIVER_VER" == *"NVIDIA-SMI"* ]]; then
  90. DRIVER_VER=""
  91. if [[ "$IS_NEURON" == "true" ]]; then
  92. log "${YELLOW}NVIDIA driver: N/A (Trainium/Inferentia instance)${NC}"
  93. else
  94. log "${YELLOW}nvidia-smi found but driver not responding${NC}"
  95. fi
  96. else
  97. VERSIONS[NVIDIA_DRIVER]="$DRIVER_VER"
  98. log "NVIDIA Driver: $DRIVER_VER"
  99. fi
  100. MAX_CUDA=$(nvidia-smi 2>/dev/null | grep "CUDA Version" | sed -n 's/.*CUDA Version: \([0-9.]*\).*/\1/p' | head -1)
  101. if [ -n "$MAX_CUDA" ]; then
  102. VERSIONS[MAX_CUDA]="$MAX_CUDA"
  103. log "Max Supported CUDA: $MAX_CUDA (driver capability)"
  104. fi
  105. log ""
  106. log "GPUs:"
  107. nvidia-smi -L 2>/dev/null | while read -r line; do log " $line"; done
  108. log ""
  109. else
  110. log "${YELLOW}nvidia-smi not found - no NVIDIA GPU or driver not installed${NC}"
  111. log ""
  112. fi
  113. if cmd_exists nvcc; then
  114. CUDA_VER=$(nvcc --version 2>/dev/null | grep "release" | sed -n 's/.*release \([0-9.]*\).*/\1/p')
  115. VERSIONS[CUDA_TOOLKIT]="$CUDA_VER"
  116. log "CUDA Toolkit (nvcc): $CUDA_VER"
  117. elif [ -L /usr/local/cuda ]; then
  118. CUDA_LINK=$(readlink /usr/local/cuda)
  119. CUDA_VER=$(echo "$CUDA_LINK" | sed -n 's/.*cuda-\([0-9.]*\).*/\1/p')
  120. VERSIONS[CUDA_TOOLKIT]="${CUDA_VER} (symlink)"
  121. log "CUDA Toolkit (symlink): $CUDA_VER"
  122. fi
  123. CUDA_DIRS=$(ls -d /usr/local/cuda-* 2>/dev/null)
  124. if [ -n "$CUDA_DIRS" ]; then
  125. log "Installed CUDA dirs: $CUDA_DIRS"
  126. [ -L /usr/local/cuda ] && log "Active symlink: /usr/local/cuda -> $(readlink /usr/local/cuda)"
  127. fi
  128. log ""
  129. # --- cuDNN ---
  130. section "cuDNN Information"
  131. CUDNN_VER=""
  132. # Check header file
  133. CUDNN_HEADER=$(find /usr/local/cuda/include /usr/include -maxdepth 2 -name "cudnn_version.h" 2>/dev/null | head -1)
  134. if [ -z "$CUDNN_HEADER" ]; then
  135. CUDNN_HEADER=$(find /usr/local/cuda/include /usr/include -maxdepth 2 -name "cudnn.h" 2>/dev/null | head -1)
  136. fi
  137. if [ -n "$CUDNN_HEADER" ]; then
  138. MAJOR=$(grep "#define CUDNN_MAJOR" "$CUDNN_HEADER" 2>/dev/null | awk '{print $3}')
  139. MINOR=$(grep "#define CUDNN_MINOR" "$CUDNN_HEADER" 2>/dev/null | awk '{print $3}')
  140. PATCH=$(grep "#define CUDNN_PATCHLEVEL" "$CUDNN_HEADER" 2>/dev/null | awk '{print $3}')
  141. [ -n "$MAJOR" ] && [ -n "$MINOR" ] && CUDNN_VER="${MAJOR}.${MINOR}.${PATCH}"
  142. fi
  143. # Package fallback
  144. if [ -z "$CUDNN_VER" ]; then
  145. if cmd_exists dpkg; then
  146. CUDNN_VER=$(dpkg -l 2>/dev/null | grep -i "libcudnn[0-9]" | head -1 | awk '{print $3}' | sed 's/-.*//')
  147. fi
  148. if [ -z "$CUDNN_VER" ] && cmd_exists rpm; then
  149. CUDNN_VER=$(rpm -qa 2>/dev/null | grep -i "libcudnn" | head -1 | sed -n 's/.*-\([0-9][0-9.]*\)-.*/\1/p')
  150. fi
  151. fi
  152. if [ -n "$CUDNN_VER" ]; then
  153. VERSIONS[CUDNN]="$CUDNN_VER"
  154. log "cuDNN: v${CUDNN_VER}"
  155. else
  156. # Check if library exists at all
  157. CUDNN_LIB=$(find /usr/local/cuda/lib64 /usr/lib -maxdepth 2 -name "libcudnn.so*" 2>/dev/null | head -1)
  158. if [ -n "$CUDNN_LIB" ]; then
  159. log "cuDNN library found: $CUDNN_LIB (version unknown)"
  160. else
  161. log "${YELLOW}cuDNN not found${NC}"
  162. fi
  163. fi
  164. log ""
  165. # --- NCCL ---
  166. section "NCCL Information"
  167. NCCL_VER=""
  168. NCCL_LIBS=$(find /usr/local/cuda*/lib* /usr/lib* /usr/local/lib* /opt/nccl/lib -maxdepth 2 -name "libnccl.so*" 2>/dev/null | head -10)
  169. if [ -n "$NCCL_LIBS" ]; then
  170. log "Libraries found:"
  171. echo "$NCCL_LIBS" | while read -r lib; do log " $lib"; done
  172. while IFS= read -r lib; do
  173. if [[ $lib =~ libnccl\.so\.([0-9]+\.[0-9]+\.[0-9]+) ]]; then
  174. NCCL_VER="${BASH_REMATCH[1]}"
  175. break
  176. fi
  177. done <<< "$NCCL_LIBS"
  178. fi
  179. # Fallback to header
  180. if [ -z "$NCCL_VER" ]; then
  181. NCCL_HEADER=$(find /usr/local/cuda*/include /usr/include /usr/local/include /opt/nccl/include -maxdepth 2 -name "nccl.h" 2>/dev/null | head -1)
  182. if [ -n "$NCCL_HEADER" ]; then
  183. MAJOR=$(grep "NCCL_MAJOR" "$NCCL_HEADER" 2>/dev/null | head -1 | awk '{print $3}')
  184. MINOR=$(grep "NCCL_MINOR" "$NCCL_HEADER" 2>/dev/null | head -1 | awk '{print $3}')
  185. PATCH=$(grep "NCCL_PATCH" "$NCCL_HEADER" 2>/dev/null | head -1 | awk '{print $3}')
  186. [ -n "$MAJOR" ] && [ -n "$MINOR" ] && [ -n "$PATCH" ] && NCCL_VER="${MAJOR}.${MINOR}.${PATCH}"
  187. [ -n "$NCCL_VER" ] && log "Version from header ($NCCL_HEADER): $NCCL_VER"
  188. fi
  189. fi
  190. if [ -n "$NCCL_VER" ]; then
  191. VERSIONS[NCCL]="$NCCL_VER"
  192. log "NCCL version: v${NCCL_VER}"
  193. else
  194. log "${YELLOW}NCCL not found${NC}"
  195. fi
  196. # Package info
  197. if cmd_exists dpkg; then
  198. NCCL_PKGS=$(dpkg -l 2>/dev/null | grep -i nccl)
  199. [ -n "$NCCL_PKGS" ] && { log ""; log "Packages (dpkg):"; echo "$NCCL_PKGS" | while read -r p; do log " $p"; done; }
  200. fi
  201. if cmd_exists rpm; then
  202. NCCL_RPMS=$(rpm -qa 2>/dev/null | grep -i nccl)
  203. [ -n "$NCCL_RPMS" ] && { log ""; log "Packages (rpm):"; echo "$NCCL_RPMS" | while read -r p; do log " $p"; done; }
  204. fi
  205. # nccl-tests
  206. NCCL_TESTS=$(find /opt /usr/local -maxdepth 4 -name "all_reduce_perf" 2>/dev/null | head -1)
  207. [ -n "$NCCL_TESTS" ] && log "nccl-tests found: $(dirname "$NCCL_TESTS")"
  208. log ""
  209. # --- EFA ---
  210. section "EFA Information"
  211. EFA_VER=""
  212. LIBFABRIC_VER=""
  213. if [ -f /opt/amazon/efa_installed_packages ]; then
  214. EFA_VER=$(grep "# EFA installer version:" /opt/amazon/efa_installed_packages | sed -n 's/.*version: \([0-9.]*\).*/\1/p')
  215. LIBFABRIC_VER=$(grep "libfabric-aws-" /opt/amazon/efa_installed_packages | sed -n 's/.*libfabric-aws-\([0-9.]*\)amzn.*/\1/p' | head -1)
  216. log "EFA installed packages:"
  217. while read -r line; do log " $line"; done < /opt/amazon/efa_installed_packages
  218. log ""
  219. fi
  220. if [ -z "$LIBFABRIC_VER" ]; then
  221. FI_INFO=""
  222. cmd_exists fi_info && FI_INFO="fi_info"
  223. [ -z "$FI_INFO" ] && [ -f /opt/amazon/efa/bin/fi_info ] && FI_INFO="/opt/amazon/efa/bin/fi_info"
  224. if [ -n "$FI_INFO" ]; then
  225. LIBFABRIC_VER=$("$FI_INFO" --version 2>&1 | grep "libfabric" | sed -n 's/.*libfabric: \([0-9.]*\).*/\1/p' | head -1)
  226. log "Libfabric ($FI_INFO): $LIBFABRIC_VER"
  227. fi
  228. fi
  229. [ -n "$EFA_VER" ] && VERSIONS[EFA_INSTALLER]="$EFA_VER" && log "EFA Installer: $EFA_VER"
  230. [ -n "$LIBFABRIC_VER" ] && VERSIONS[LIBFABRIC]="$LIBFABRIC_VER" && log "Libfabric: $LIBFABRIC_VER"
  231. # EFA provider check
  232. FI_CMD=""
  233. cmd_exists fi_info && FI_CMD="fi_info"
  234. [ -z "$FI_CMD" ] && [ -f /opt/amazon/efa/bin/fi_info ] && FI_CMD="/opt/amazon/efa/bin/fi_info"
  235. if [ -n "$FI_CMD" ]; then
  236. if "$FI_CMD" -p efa 2>&1 | grep -q "provider: efa"; then
  237. log "${GREEN}EFA provider available${NC}"
  238. else
  239. log "${YELLOW}EFA provider not detected${NC}"
  240. fi
  241. fi
  242. [ -d /sys/class/infiniband ] && log "InfiniBand devices: $(ls /sys/class/infiniband/ 2>/dev/null | tr '\n' ' ')" || log "${YELLOW}No InfiniBand devices found${NC}"
  243. log ""
  244. # --- AWS OFI NCCL ---
  245. section "AWS OFI NCCL Plugin"
  246. OFI_NCCL_VER=""
  247. if [ -f /opt/amazon/efa_installed_packages ]; then
  248. OFI_NCCL_VER=$(grep "libnccl-ofi-" /opt/amazon/efa_installed_packages | sed -n 's/.*libnccl-ofi-\([0-9.]*\)-.*/\1/p' | head -1)
  249. fi
  250. if [ -n "$OFI_NCCL_VER" ]; then
  251. VERSIONS[AWS_OFI_NCCL]="$OFI_NCCL_VER"
  252. log "AWS OFI NCCL: v${OFI_NCCL_VER}"
  253. else
  254. OFI_LIB=$(find /opt/amazon/ofi-nccl /usr/lib* -maxdepth 3 -name "libnccl-net.so" 2>/dev/null | head -1)
  255. if [ -n "$OFI_LIB" ]; then
  256. log "AWS OFI NCCL library found: $OFI_LIB (version unknown)"
  257. else
  258. log "${YELLOW}AWS OFI NCCL not found${NC}"
  259. fi
  260. fi
  261. log ""
  262. # --- GDRCopy ---
  263. section "GDRCopy Information"
  264. GDRCOPY_VER=""
  265. if cmd_exists rpm; then
  266. GDRCOPY_VER=$(rpm -qa 2>/dev/null | grep "^gdrcopy-[0-9]" | head -1 | sed -n 's/gdrcopy-\([0-9.]*\)-.*/\1/p')
  267. fi
  268. if [ -z "$GDRCOPY_VER" ] && cmd_exists dpkg; then
  269. GDRCOPY_VER=$(dpkg -l 2>/dev/null | grep "^ii.*gdrcopy" | head -1 | awk '{print $3}' | sed -n 's/\([0-9.]*\)-.*/\1/p')
  270. fi
  271. if [ -n "$GDRCOPY_VER" ]; then
  272. VERSIONS[GDRCOPY]="$GDRCOPY_VER"
  273. log "GDRCopy: v${GDRCOPY_VER}"
  274. else
  275. GDRCOPY_LIB=$(find /usr /opt -maxdepth 4 -name "libgdrapi.so*" 2>/dev/null | head -1)
  276. [ -n "$GDRCOPY_LIB" ] && log "GDRCopy library found: $GDRCOPY_LIB (version unknown)" || log "${YELLOW}GDRCopy not found${NC}"
  277. fi
  278. if lsmod 2>/dev/null | grep -q gdrdrv; then
  279. log "Kernel module: ${GREEN}gdrdrv loaded${NC}"
  280. else
  281. log "Kernel module: ${YELLOW}gdrdrv not loaded${NC}"
  282. fi
  283. log ""
  284. # --- MPI ---
  285. section "MPI Information"
  286. MPI_VER=""
  287. if cmd_exists mpirun; then
  288. MPI_VER=$(mpirun --version 2>&1 | head -1)
  289. elif [ -f /opt/amazon/openmpi/bin/mpirun ]; then
  290. MPI_VER=$(/opt/amazon/openmpi/bin/mpirun --version 2>&1 | head -1)
  291. fi
  292. if [ -n "$MPI_VER" ]; then
  293. VERSIONS[MPI]="$MPI_VER"
  294. log "MPI: $MPI_VER"
  295. else
  296. log "${YELLOW}MPI not found${NC}"
  297. fi
  298. log ""
  299. # --- Neuron SDK (Trainium/Inferentia) ---
  300. section "Neuron SDK Information"
  301. NEURON_DETECTED=false
  302. NEURON_BIN="/opt/aws/neuron/bin"
  303. # Neuron driver (kernel module)
  304. NEURON_DRV_VER=$(modinfo neuron 2>/dev/null | grep "^version:" | awk '{print $2}')
  305. if [ -n "$NEURON_DRV_VER" ]; then
  306. VERSIONS[NEURON_DRIVER]="$NEURON_DRV_VER"
  307. log "Neuron Driver: $NEURON_DRV_VER"
  308. NEURON_DETECTED=true
  309. fi
  310. # Neuron devices
  311. NEURON_DEV_COUNT=$(ls /dev/neuron* 2>/dev/null | wc -l)
  312. if [ "$NEURON_DEV_COUNT" -gt 0 ]; then
  313. VERSIONS[NEURON_DEVICES]="$NEURON_DEV_COUNT"
  314. log "Neuron Devices: $NEURON_DEV_COUNT"
  315. NEURON_DETECTED=true
  316. fi
  317. # Neuron devices listing
  318. NEURON_LS=$(cmd_or_path neuron-ls "$NEURON_BIN/neuron-ls")
  319. if [ -x "$NEURON_LS" ]; then
  320. NEURON_DETECTED=true
  321. log "Neuron devices:"
  322. "$NEURON_LS" 2>/dev/null | while read -r line; do log " $line"; done
  323. log ""
  324. fi
  325. # Neuron compiler
  326. NEURON_CC=$(cmd_or_path neuronx-cc "$NEURON_BIN/neuronx-cc")
  327. if [ -x "$NEURON_CC" ]; then
  328. NEURON_CC_VER=$("$NEURON_CC" --version 2>&1 | head -1)
  329. VERSIONS[NEURON_COMPILER]="$NEURON_CC_VER"
  330. log "Neuron Compiler: $NEURON_CC_VER"
  331. NEURON_DETECTED=true
  332. fi
  333. # Neuron runtime
  334. NEURON_RT_VER=""
  335. if cmd_exists dpkg; then
  336. NEURON_RT_VER=$(dpkg -l 2>/dev/null | grep "aws-neuronx-runtime-lib" | head -1 | awk '{print $3}')
  337. fi
  338. if [ -z "$NEURON_RT_VER" ] && cmd_exists rpm; then
  339. NEURON_RT_VER=$(rpm -qa 2>/dev/null | grep "aws-neuronx-runtime" | head -1 | sed -n 's/.*-\([0-9][0-9.]*\)-.*/\1/p')
  340. fi
  341. if [ -n "$NEURON_RT_VER" ]; then
  342. VERSIONS[NEURON_RUNTIME]="$NEURON_RT_VER"
  343. log "Neuron Runtime: $NEURON_RT_VER"
  344. NEURON_DETECTED=true
  345. fi
  346. # torch-neuronx
  347. TORCH_NEURON_VER=$(python3 -c "import torch_neuronx; print(torch_neuronx.__version__)" 2>/dev/null)
  348. if [ -n "$TORCH_NEURON_VER" ]; then
  349. VERSIONS[TORCH_NEURONX]="$TORCH_NEURON_VER"
  350. log "torch-neuronx: $TORCH_NEURON_VER"
  351. NEURON_DETECTED=true
  352. fi
  353. # Neuron tools
  354. NEURON_TOP=$(cmd_or_path neuron-top "$NEURON_BIN/neuron-top")
  355. if [ -x "$NEURON_TOP" ]; then
  356. NEURON_TOOLS_VER=""
  357. if cmd_exists dpkg; then
  358. NEURON_TOOLS_VER=$(dpkg -l 2>/dev/null | grep "aws-neuronx-tools" | head -1 | awk '{print $3}')
  359. fi
  360. if [ -z "$NEURON_TOOLS_VER" ] && cmd_exists rpm; then
  361. NEURON_TOOLS_VER=$(rpm -qa 2>/dev/null | grep "aws-neuronx-tools" | head -1 | sed -n 's/.*-\([0-9][0-9.]*\)-.*/\1/p')
  362. fi
  363. [ -n "$NEURON_TOOLS_VER" ] && log "Neuron Tools: $NEURON_TOOLS_VER"
  364. NEURON_DETECTED=true
  365. fi
  366. if [[ "$NEURON_DETECTED" != "true" ]]; then
  367. log "${YELLOW}Neuron SDK not found (expected on non-Trainium/Inferentia instances)${NC}"
  368. fi
  369. log ""
  370. # --- Python & PyTorch ---
  371. section "Python / ML Frameworks"
  372. if cmd_exists python3; then
  373. PY_VER=$(python3 --version 2>&1 | awk '{print $2}')
  374. VERSIONS[PYTHON]="$PY_VER"
  375. log "Python: $PY_VER"
  376. PT_INFO=$(python3 -c "
  377. import torch
  378. print(f'{torch.__version__}')
  379. print(f'cuda_available={torch.cuda.is_available()}')
  380. print(f'cuda_version={torch.version.cuda or \"N/A\"}')
  381. if hasattr(torch, 'xpu') and hasattr(torch.xpu, 'is_available'):
  382. print(f'xpu_available={torch.xpu.is_available()}')
  383. " 2>/dev/null)
  384. if [ -n "$PT_INFO" ]; then
  385. PT_VER=$(echo "$PT_INFO" | head -1)
  386. VERSIONS[PYTORCH]="$PT_VER"
  387. log "PyTorch: $PT_VER"
  388. echo "$PT_INFO" | tail -n +2 | while read -r line; do log " $line"; done
  389. fi
  390. else
  391. log "${YELLOW}python3 not found${NC}"
  392. fi
  393. log ""
  394. # --- Container Runtime ---
  395. section "Container Runtime"
  396. cmd_exists docker && log "Docker: $(docker --version 2>&1)"
  397. cmd_exists containerd && log "Containerd: $(containerd --version 2>&1)"
  398. cmd_exists kubectl && log "kubectl: $(kubectl version --client 2>&1 | head -1)"
  399. # NVIDIA Container Toolkit
  400. if cmd_exists nvidia-ctk; then
  401. NCTK_VER=$(nvidia-ctk --version 2>&1 | head -1)
  402. VERSIONS[NVIDIA_CTK]="$NCTK_VER"
  403. log "NVIDIA Container Toolkit: $NCTK_VER"
  404. elif cmd_exists dpkg && dpkg -l 2>/dev/null | grep -q nvidia-container-toolkit; then
  405. NCTK_VER=$(dpkg -l 2>/dev/null | grep "nvidia-container-toolkit " | head -1 | awk '{print $3}')
  406. VERSIONS[NVIDIA_CTK]="$NCTK_VER"
  407. log "NVIDIA Container Toolkit: $NCTK_VER"
  408. fi
  409. log ""
  410. # --- Environment Variables ---
  411. section "Relevant Environment Variables"
  412. log "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH:-<not set>}"
  413. log "NCCL vars: $(env | grep -i "^NCCL" 2>/dev/null | tr '\n' ' ')"
  414. log "EFA vars: $(env | grep -i "^FI_\|^EFA_\|^RDMAV" 2>/dev/null | tr '\n' ' ')"
  415. log "NEURON vars: $(env | grep -i "^NEURON" 2>/dev/null | tr '\n' ' ')"
  416. log ""
  417. # --- CUDA/Driver Compatibility Analysis ---
  418. section "CUDA/Driver Compatibility Analysis"
  419. if [ -n "${VERSIONS[NVIDIA_DRIVER]}" ] && [ -n "${VERSIONS[MAX_CUDA]}" ]; then
  420. DRIVER_MAJOR=$(echo "${VERSIONS[NVIDIA_DRIVER]}" | cut -d'.' -f1)
  421. log "Driver ${VERSIONS[NVIDIA_DRIVER]} (series $DRIVER_MAJOR):"
  422. if [ "$DRIVER_MAJOR" -ge 580 ] 2>/dev/null; then
  423. log " ${GREEN}✓ Supports CUDA 13.x, 12.x, 11.x${NC}"
  424. elif [ "$DRIVER_MAJOR" -ge 570 ] 2>/dev/null; then
  425. log " ${GREEN}✓ Supports CUDA 12.8+ (Blackwell), 12.x, 11.x${NC}"
  426. elif [ "$DRIVER_MAJOR" -ge 545 ] 2>/dev/null; then
  427. log " ${GREEN}✓ Supports CUDA 12.3-12.7, 11.x${NC}"
  428. log " ${YELLOW}⚠ NOT compatible with CUDA 12.8+ (needs driver 570+)${NC}"
  429. elif [ "$DRIVER_MAJOR" -ge 525 ] 2>/dev/null; then
  430. log " ${GREEN}✓ Supports CUDA 12.0-12.2, 11.x${NC}"
  431. log " ${YELLOW}⚠ NOT compatible with CUDA 12.3+ (needs driver 545+)${NC}"
  432. elif [ "$DRIVER_MAJOR" -ge 450 ] 2>/dev/null; then
  433. log " ${GREEN}✓ Supports CUDA 11.x${NC}"
  434. log " ${YELLOW}⚠ NOT compatible with CUDA 12.x (needs driver 525+)${NC}"
  435. else
  436. log " ${YELLOW}⚠ Driver older than CUDA 11.x baseline${NC}"
  437. fi
  438. fi
  439. log ""
  440. # --- Version Summary ---
  441. section "Version Summary"
  442. log "NVIDIA_DRIVER: ${VERSIONS[NVIDIA_DRIVER]:-not found}"
  443. log "MAX_CUDA: ${VERSIONS[MAX_CUDA]:-not found}"
  444. log "CUDA_TOOLKIT: ${VERSIONS[CUDA_TOOLKIT]:-not found}"
  445. log "CUDNN: ${VERSIONS[CUDNN]:+v${VERSIONS[CUDNN]}}${VERSIONS[CUDNN]:-not found}"
  446. log "NCCL: ${VERSIONS[NCCL]:+v${VERSIONS[NCCL]}}${VERSIONS[NCCL]:-not found}"
  447. log "EFA_INSTALLER: ${VERSIONS[EFA_INSTALLER]:-not found}"
  448. log "LIBFABRIC: ${VERSIONS[LIBFABRIC]:-not found}"
  449. log "AWS_OFI_NCCL: ${VERSIONS[AWS_OFI_NCCL]:+v${VERSIONS[AWS_OFI_NCCL]}}${VERSIONS[AWS_OFI_NCCL]:-not found}"
  450. log "GDRCOPY: ${VERSIONS[GDRCOPY]:+v${VERSIONS[GDRCOPY]}}${VERSIONS[GDRCOPY]:-not found}"
  451. log "MPI: ${VERSIONS[MPI]:-not found}"
  452. log "NEURON_DRIVER: ${VERSIONS[NEURON_DRIVER]:-not found}"
  453. log "NEURON_DEVICES: ${VERSIONS[NEURON_DEVICES]:-0}"
  454. log "NEURON_COMPILER: ${VERSIONS[NEURON_COMPILER]:-not found}"
  455. log "NEURON_RUNTIME: ${VERSIONS[NEURON_RUNTIME]:-not found}"
  456. log "TORCH_NEURONX: ${VERSIONS[TORCH_NEURONX]:-not found}"
  457. log "PYTHON: ${VERSIONS[PYTHON]:-not found}"
  458. log "PYTORCH: ${VERSIONS[PYTORCH]:-not found}"
  459. log ""
  460. log "Report saved to: $OUTPUT_FILE"
  461. # --- JSON output (stdout only) ---
  462. if [[ "$JSON_OUTPUT" == "true" ]]; then
  463. cat <<EOF
  464. {
  465. "hostname": "$(hostname)",
  466. "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
  467. "instance_type": "$(json_escape "${INSTANCE_TYPE:-unknown}")",
  468. "versions": {
  469. "nvidia_driver": "$(json_escape "${VERSIONS[NVIDIA_DRIVER]:-}")",
  470. "max_cuda": "$(json_escape "${VERSIONS[MAX_CUDA]:-}")",
  471. "cuda_toolkit": "$(json_escape "${VERSIONS[CUDA_TOOLKIT]:-}")",
  472. "cudnn": "$(json_escape "${VERSIONS[CUDNN]:-}")",
  473. "nccl": "$(json_escape "${VERSIONS[NCCL]:-}")",
  474. "efa_installer": "$(json_escape "${VERSIONS[EFA_INSTALLER]:-}")",
  475. "libfabric": "$(json_escape "${VERSIONS[LIBFABRIC]:-}")",
  476. "aws_ofi_nccl": "$(json_escape "${VERSIONS[AWS_OFI_NCCL]:-}")",
  477. "gdrcopy": "$(json_escape "${VERSIONS[GDRCOPY]:-}")",
  478. "mpi": "$(json_escape "${VERSIONS[MPI]:-}")",
  479. "neuron_driver": "$(json_escape "${VERSIONS[NEURON_DRIVER]:-}")",
  480. "neuron_devices": "$(json_escape "${VERSIONS[NEURON_DEVICES]:-}")",
  481. "neuron_compiler": "$(json_escape "${VERSIONS[NEURON_COMPILER]:-}")",
  482. "neuron_runtime": "$(json_escape "${VERSIONS[NEURON_RUNTIME]:-}")",
  483. "torch_neuronx": "$(json_escape "${VERSIONS[TORCH_NEURONX]:-}")",
  484. "nvidia_container_toolkit": "$(json_escape "${VERSIONS[NVIDIA_CTK]:-}")",
  485. "python": "$(json_escape "${VERSIONS[PYTHON]:-}")",
  486. "pytorch": "$(json_escape "${VERSIONS[PYTORCH]:-}")"
  487. }
  488. }
  489. EOF
  490. fi

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.