agent-plugins-for-aws
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/awslabs--agent-plugins)HTML
<a href="/listings/awslabs--agent-plugins"><img src="/api/listings/awslabs--agent-plugins/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Claude Code
Add the marketplace
/plugin marketplace add awslabs/agent-plugins
Install a plugin
/plugin install amazon-location-service@agent-plugins-for-aws
or
/plugin install aws-amplify@agent-plugins-for-aws
or
/plugin install aws-serverless@agent-plugins-for-aws
or
/plugin install codebase-documentor-for-aws@agent-plugins-for-aws
or
/plugin install databases-on-aws@agent-plugins-for-aws
or
/plugin install deploy-on-aws@agent-plugins-for-aws
or
/plugin install sagemaker-ai@agent-plugins-for-aws
Codex
Codex supports repo-local marketplaces and plugin manifests through `.agents/plugins/marketplace.json` and per-plugin manifests under `plugins/*/.codex-plugin/plugin.json`; this repo includes both, following the official Codex plugin packaging guidance in [Build plugins](https://developers.openai.com/codex/plugins/build).
To try these plugins in Codex from this repo:
1. Clone this repository locally. 2. Open the repo in Codex so Codex can discover `.agents/plugins/marketplace.json`. 3. Restart Codex. 4. Open the plugin directory, choose the `Agent Plugins for AWS` marketplace, and install the plugin you want to use.
Known limitation:
- Claude-specific automatic hooks are not yet wired into the Codex manifests. - The `databases-on-aws` prompt hook is included because it does not depend on Claude-specific environment variables.
Cursor
You can install the **deploy-on-aws** plugin from the [Cursor Marketplace](https://cursor.com/marketplace/aws). For additional information, please refer to the [Cursor plugin documentation](https://cursor.com/docs/plugins). You can also install within the Cursor application:
- Open Cursor Settings - Navigate to `Plugins` - Search for `AWS` - Select the plugin you want to install and click `Add to Cursor` - Select the scope for the installed plugin - The plugin should appear under `Plugins -> Installed`
Kiro (experimental)
You can convert Claude Code plugins into Kiro format using the third-party [`@every-env/compound-plugin`](https://www.npmjs.com/package/@every-env/compound-plugin) CLI. This converts skills (SKILL.md workflow instructions and reference documents) and MCP server configs into Kiro's `.kiro/` format.
Requires [Bun](https://bun.sh) (latest version recommended).
> [!NOTE] > Hook conversion is a work in progress — hooks are currently dropped entirely during conversion because the Claude Code and Kiro hook models are not 1:1. Skills and MCP servers are fully supported.
Project scope
Writes to `.kiro/` in the current working directory:
COMPOUND_PLUGIN_GITHUB_SOURCE=https://github.com/awslabs/agent-plugins \ bunx @every-env/compound-plugin install deploy-on-aws --to kiro
Global scope
Writes to `~/.kiro/` so skills are available across all projects:
COMPOUND_PLUGIN_GITHUB_SOURCE=https://github.com/awslabs/agent-plugins \ bunx @every-env/compound-plugin install deploy-on-aws --to kiro --output ~/.kiro
Replace `deploy-on-aws` with any plugin name from the table above (e.g., `aws-serverless`, `amazon-location-service`, `migration-to-aws`, `aws-amplify`).
> [!TIP] > If a skill isn't activated automatically, you can explicitly invoke it by saying "Use the deploy skill to ..." in your prompt. Kiro's intent matching may not always auto-trigger skills. > > If `bunx` fails with a "Cannot find package" error, clear the bun cache and retry: > > ```bash > rm -rf ~/.bun/install/cache/@every-env > ```
For issues with the conversion tool, please report them at [EveryInc/compound-engineering-plugin](https://github.com/EveryInc/compound-engineering-plugin/issues).
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
.claude-plugin/marketplace.json · Manifest · 248 lines
- {
- "metadata": {
- "description": "Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS.",
- "version": "1.1.0"
- },
- "name": "agent-plugins-for-aws",
- "owner": {
- "email": "aws-agent-plugins@amazon.com",
- "name": "Amazon Web Services"
- },
- "plugins": [
- {
- "category": "location",
- "description": "Guide developers through adding maps, places search, geocoding, routing, and other geospatial features with Amazon Location Service, including authentication setup, SDK integration, and best practices.",
- "keywords": [
- "location",
- "maps",
- "geocoding",
- "routing",
- "places",
- "geofencing",
- "tracking",
- "amazon-location",
- "maplibre",
- "address",
- "coordinates"
- ],
- "name": "amazon-location-service",
- "source": "./plugins/amazon-location-service",
- "tags": [
- "aws",
- "location",
- "maps",
- "geospatial"
- ],
- "version": "1.1.0"
- },
- {
- "category": "fullstack",
- "description": "Build full-stack apps with AWS Amplify Gen 2 using guided workflows for authentication, data models, storage, GraphQL APIs, and Lambda functions.",
- "keywords": [
- "aws",
- "amplify",
- "fullstack",
- "authentication",
- "data",
- "storage",
- "graphql",
- "functions"
- ],
- "name": "aws-amplify",
- "source": "./plugins/aws-amplify",
- "tags": [
- "aws",
- "amplify",
- "fullstack"
- ],
- "version": "1.0.0"
- },
- {
- "category": "development",
- "description": "Design, build, deploy, test, and debug serverless applications with AWS Serverless services.",
- "keywords": [
- "aws",
- "aws agent skills",
- "amazon",
- "serverless",
- "lambda",
- "durable functions",
- "managed-instances",
- "lmi",
- "capacity-provider",
- "development",
- "sam",
- "api gateway",
- "eventbridge",
- "step functions",
- "microvms",
- "firecracker"
- ],
- "name": "aws-serverless",
- "source": "./plugins/aws-serverless",
- "tags": [
- "aws",
- "serverless",
- "lambda",
- "durable functions",
- "managed-instances",
- "lmi",
- "development",
- "sam",
- "api gateway",
- "eventbridge",
- "step functions",
- "microvms",
- "firecracker"
- ],
- "version": "1.3.0"
- },
- {
- "category": "migration",
- "description": "Migrate, modernize, and upgrade codebases to AWS. Transforms .NET Framework to .NET 8/10, mainframe COBOL to Java, VMware VMs to EC2, SQL Server to Aurora, and upgrades Java/Python/Node.js versions and AWS SDKs. AWS Transform - continuous modernization analyzes codebases for tech debt, security issues, and upgrade opportunities, then remediates them.",
- "keywords": [
- "aws",
- "aws-transform",
- "migration",
- "modernization",
- "dotnet",
- "mainframe",
- "cobol",
- "vmware",
- "ec2",
- "sql-server",
- "aurora",
- "java",
- "python",
- "sdk",
- "continuous modernization",
- "AWS Transform - continuous modernization",
- "tech debt",
- "security vulnerabilities",
- "analyze repos",
- "security analysis",
- "modernization analysis",
- "agentic readiness"
- ],
- "name": "aws-transform",
- "source": "./plugins/aws-transform",
- "tags": [
- "aws",
- "migration",
- "modernization",
- "transform",
- "continuous modernization"
- ],
- "version": "1.9.0"
- },
- {
- "category": "development",
- "description": "Analyze codebases to generate structured technical documentation with source-of-truth citations. Optimized for AWS workloads using CDK, CloudFormation, and Terraform.",
- "keywords": [
- "aws",
- "aws agent skills",
- "documentation",
- "architecture",
- "diagram",
- "draw.io",
- "codebase-analysis",
- "cdk",
- "cloudformation",
- "terraform",
- "onboarding"
- ],
- "name": "codebase-documentor-for-aws",
- "source": "./plugins/codebase-documentor-for-aws",
- "tags": [
- "aws",
- "documentation",
- "architecture",
- "diagram"
- ],
- "version": "0.1.0"
- },
- {
- "category": "database",
- "description": "Expert database guidance for the AWS database portfolio. Design schemas, execute queries, handle migrations, and choose the right database for your workload.",
- "keywords": [
- "aws",
- "aws agent skills",
- "amazon",
- "aurora",
- "database",
- "databases",
- "distributed-sql",
- "dsql",
- "iam-auth",
- "migration",
- "postgresql",
- "postgres",
- "schema",
- "serverless",
- "sql"
- ],
- "name": "databases-on-aws",
- "source": "./plugins/databases-on-aws",
- "tags": [
- "aws",
- "database",
- "aurora",
- "dsql",
- "serverless",
- "postgresql"
- ],
- "version": "1.11.0"
- },
- {
- "category": "deployment",
- "description": "Deploy applications to AWS with architecture recommendations, cost estimates, and IaC deployment. Generate validated AWS architecture diagrams as draw.io XML.",
- "keywords": [
- "aws",
- "aws agent skills",
- "amazon",
- "deploy",
- "cdk",
- "cloudformation",
- "infrastructure",
- "pricing",
- "diagrams",
- "draw.io",
- "architecture"
- ],
- "name": "deploy-on-aws",
- "source": "./plugins/deploy-on-aws",
- "tags": [
- "aws",
- "deploy",
- "infrastructure",
- "cdk",
- "diagrams"
- ],
- "version": "1.4.0"
- },
- {
- "category": "ai",
- "description": "Build, train, and deploy AI models with deep AWS AI/ML expertise brought directly into your coding assistants, covering the surface area of Amazon SageMaker AI.",
- "keywords": [
- "aws",
- "sagemaker",
- "machine-learning",
- "generative-ai",
- "fine-tuning",
- "training",
- "deployment",
- "inference",
- "mlops"
- ],
- "name": "sagemaker-ai",
- "source": "./plugins/sagemaker-ai",
- "tags": [
- "aws",
- "sagemaker",
- "machine-learning",
- "generative-ai"
- ],
- "version": "1.3.0"
- }
- ]
- }
README.md · README · 349 lines
- # Agent Plugins for AWS
- _Read this in other languages: [日本語](README.jp.md)_
- > [!IMPORTANT]
- > Generative AI can make mistakes. You should consider reviewing all output and costs generated by your chosen AI model and agentic coding assistant. See [AWS Responsible AI Policy](https://aws.amazon.com/ai/responsible-ai/policy/).
- ## What's new
- > [!TIP]
- > The [Agent Toolkit for AWS](https://aws.amazon.com/about-aws/whats-new/2026/05/agent-toolkit/) is now live! The Agent Toolkit for AWS is the successor to the MCP servers, plugins, and skills available on AWS Labs, and was informed by feedback from customers like you. If you're building production software using coding agents or building agents for your own customers, we recommend Agent Toolkit for AWS. It includes IAM condition keys to distinguish agent actions from human ones, CloudWatch and CloudTrail visibility, and skills that have been evaluated for accuracy and effectiveness. This repo continues to work and accept contributions. Over time, the most useful projects here will move into Agent Toolkit for AWS.
- ---
- ## Agent Plugins for AWS
- Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS. Agent plugins are currently supported by Claude Code, Codex, and Cursor.
- AI coding agents are increasingly used in software development, helping developers write, review, and deploy code more efficiently. Agent skills and the broader agent plugin packaging model are emerging as best practices for steering coding agents toward reliable outcomes without bloating model context. Instead of repeatedly pasting long AWS guidance into prompts, developers can now encode that guidance as reusable, versioned capabilities that agents invoke when relevant. This improves determinism, reduces context overhead, and makes agent behavior easier to standardize across teams. Agent plugins act as containers that package different types of expertise artifacts together. A single agent plugin can include:
- - Agent skills – Structured workflows and best-practice playbooks that guide AI through complex tasks like deployment, code review, or architecture planning. Agent skills encode domain expertise as step-by-step processes.
- - MCP servers – Connections to external services, data sources, and APIs. MCP servers give your assistant access to live documentation, pricing data, and other resources at runtime. Learn more about [MCP servers for AWS](https://github.com/awslabs/mcp).
- - Hooks – Automation and guardrails that run on developer actions. Hooks can validate changes, enforce standards, or trigger workflows automatically.
- - References – Documentation, configuration defaults, and knowledge that the agent skill can consult. References make agent skills smarter without bloating the prompt.
- As new types of expertise artifacts emerge in this space, they can be packaged into agent plugins, making the evolution transparent to developers.
- ## Best practices
- To maximize the benefits of plugin-assisted development while maintaining security and code quality, follow these essential guidelines:
- - Always review generated code before deployment (for example, against your constraints for security, cost, resilience)
- - Use plugins as accelerators, not replacements for developer judgment and expertise.
- - Keep plugins updated to benefit from the latest AWS best practices.
- - Follow the principle of least privilege when configuring AWS credentials.
- - Run security scanning tools on generated infrastructure code.
- ## Plugins
- | Plugin | Description | Status |
- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
- | **amazon-location-service** | Add maps, geocoding, routing, places search, and geospatial features to applications with Amazon Location Service | Available |
- | **aws-amplify** | Build full-stack apps with AWS Amplify Gen 2 using guided workflows for auth, data, storage, and functions | Available |
- | **aws-serverless** | Build serverless applications with Lambda, API Gateway, EventBridge, Step Functions, durable functions, and Lambda Managed Instances | Available |
- | **aws-transform** | Migrate, modernize, and upgrade codebases to AWS — .NET to .NET 8/10, mainframe COBOL to Java, VMware to EC2, SQL Server to Aurora, and language/SDK upgrades | Available |
- | **codebase-documentor-for-aws** | Analyze AWS-deployed services and codebases to generate structured technical documentation with source-of-truth citations | Available |
- | **databases-on-aws** | Database guidance for the AWS database portfolio — schema design, queries, migrations, and multi-tenant patterns | Some Services Available (Aurora DSQL) |
- | **deploy-on-aws** | Deploy applications to AWS with architecture recommendations, cost estimates, and IaC deployment | Available |
- | **migration-to-aws** | Migrate GCP infrastructure to AWS with resource discovery, architecture mapping, cost analysis, and execution planning | Available |
- | **sagemaker-ai** | Build, train, and deploy AI models with deep AWS AI/ML expertise brought directly into your coding assistants, covering the surface area of [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/ai/) | Available |
- ## Installation
- ### Claude Code
- #### Add the marketplace
- ```bash
- /plugin marketplace add awslabs/agent-plugins
- ```
- #### Install a plugin
- ```bash
- /plugin install amazon-location-service@agent-plugins-for-aws
- ```
- or
- ```bash
- /plugin install aws-amplify@agent-plugins-for-aws
- ```
- or
- ```bash
- /plugin install aws-serverless@agent-plugins-for-aws
- ```
- or
- ```bash
- /plugin install codebase-documentor-for-aws@agent-plugins-for-aws
- ```
- or
- ```bash
- /plugin install databases-on-aws@agent-plugins-for-aws
- ```
- or
- ```bash
- /plugin install deploy-on-aws@agent-plugins-for-aws
- ```
- or
- ```bash
- /plugin install sagemaker-ai@agent-plugins-for-aws
- ```
- ### Codex
- Codex supports repo-local marketplaces and plugin manifests through
- `.agents/plugins/marketplace.json` and per-plugin manifests under
- `plugins/*/.codex-plugin/plugin.json`; this repo includes both, following the
- official Codex plugin packaging guidance in
- [Build plugins](https://developers.openai.com/codex/plugins/build).
- To try these plugins in Codex from this repo:
- 1. Clone this repository locally.
- 2. Open the repo in Codex so Codex can discover
- `.agents/plugins/marketplace.json`.
- 3. Restart Codex.
- 4. Open the plugin directory, choose the `Agent Plugins for AWS` marketplace,
- and install the plugin you want to use.
- Known limitation:
- - Claude-specific automatic hooks are not yet wired into the Codex manifests.
- - The `databases-on-aws` prompt hook is included because it does not depend on
- Claude-specific environment variables.
- ### Cursor
- You can install the **deploy-on-aws** plugin from the [Cursor Marketplace](https://cursor.com/marketplace/aws). For additional information, please refer to the [Cursor plugin documentation](https://cursor.com/docs/plugins). You can also install within the Cursor application:
- - Open Cursor Settings
- - Navigate to `Plugins`
- - Search for `AWS`
- - Select the plugin you want to install and click `Add to Cursor`
- - Select the scope for the installed plugin
- - The plugin should appear under `Plugins -> Installed`
- ### Kiro (experimental)
- You can convert Claude Code plugins into Kiro format using the third-party [`@every-env/compound-plugin`](https://www.npmjs.com/package/@every-env/compound-plugin) CLI. This converts skills (SKILL.md workflow instructions and reference documents) and MCP server configs into Kiro's `.kiro/` format.
- Requires [Bun](https://bun.sh) (latest version recommended).
- > [!NOTE]
- > Hook conversion is a work in progress — hooks are currently dropped entirely during conversion because the Claude Code and Kiro hook models are not 1:1. Skills and MCP servers are fully supported.
- #### Project scope
- Writes to `.kiro/` in the current working directory:
- ```bash
- COMPOUND_PLUGIN_GITHUB_SOURCE=https://github.com/awslabs/agent-plugins \
- bunx @every-env/compound-plugin install deploy-on-aws --to kiro
- ```
- #### Global scope
- Writes to `~/.kiro/` so skills are available across all projects:
- ```bash
- COMPOUND_PLUGIN_GITHUB_SOURCE=https://github.com/awslabs/agent-plugins \
- bunx @every-env/compound-plugin install deploy-on-aws --to kiro --output ~/.kiro
- ```
- Replace `deploy-on-aws` with any plugin name from the table above (e.g., `aws-serverless`, `amazon-location-service`, `migration-to-aws`, `aws-amplify`).
- > [!TIP]
- > If a skill isn't activated automatically, you can explicitly invoke it by saying "Use the deploy skill to ..." in your prompt. Kiro's intent matching may not always auto-trigger skills.
- >
- > If `bunx` fails with a "Cannot find package" error, clear the bun cache and retry:
- >
- > ```bash
- > rm -rf ~/.bun/install/cache/@every-env
- > ```
- For issues with the conversion tool, please report them at [EveryInc/compound-engineering-plugin](https://github.com/EveryInc/compound-engineering-plugin/issues).
- ## amazon-location-service
- Guides developers through adding maps, places search, geocoding, routing, and other geospatial features with Amazon Location Service, including authentication setup, SDK integration, and best practices.
- ### Agent Skill Triggers
- | Agent Skill | Triggers |
- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
- | **amazon-location-service** | "add a map", "geocode an address", "calculate a route", "location-aware app", "Amazon Location Service", "geospatial", "places search" |
- ### MCP Servers
- | Server | Purpose |
- | ----------- | -------------------------------------- |
- | **aws-mcp** | AWS documentation and service guidance |
- ## aws-amplify
- Build full-stack apps with AWS Amplify Gen 2 using TypeScript code-first development, guided by official AWS Agent Standard Operating Procedures (SOPs).
- ### Workflow
- 1. **Backend** - Create Amplify Gen 2 resources (auth, data, storage, functions)
- 2. **Sandbox** - Deploy to sandbox for testing
- 3. **Frontend & Test** - Connect frontend to backend, verify locally
- 4. **Production** - Deploy to production
- ### Agent Skill Triggers
- | Agent Skill | Triggers |
- | -------------------- | ------------------------------------------------------------------------------------------------------------ |
- | **amplify-workflow** | "build Amplify app", "create Amplify project", "add auth to Amplify", "deploy Amplify", "full-stack Amplify" |
- ### MCP Servers
- | Server | Purpose |
- | ----------- | -------------------------------------- |
- | **aws-mcp** | AWS documentation and service guidance |
- ## aws-serverless
- Design, build, deploy, test, and debug serverless applications with AWS Lambda, API Gateway, EventBridge, Step Functions, and durable functions. Includes SAM and CDK deployment workflows, a SAM template validation hook, the AWS Lambda durable functions skill for building resilient, long-running, multi-step applications, and the Lambda Managed Instances skill for evaluating, configuring, and migrating workloads to EC2-backed Lambda.
- ### Agent Skill Triggers
- | Agent Skill | Triggers |
- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
- | **aws-lambda** | "Lambda function", "event source", "serverless application", "API Gateway", "EventBridge", "Step Functions", "serverless API", "event-driven architecture", "Lambda trigger" |
- | **aws-serverless-deployment** | "use SAM", "SAM template", "SAM init", "SAM deploy", "CDK serverless", "CDK Lambda construct", "NodejsFunction", "PythonFunction", "serverless CI/CD pipeline" |
- | **aws-lambda-durable-functions** | "lambda durable functions", "workflow orchestration", "state machines", "retry/checkpoint patterns", "long-running stateful Lambda", "saga pattern", "human-in-the-loop" |
- | **aws-lambda-managed-instances** | "Lambda Managed Instances", "LMI", "capacity provider", "multi-concurrency Lambda", "EC2-backed Lambda", "cold start elimination", "Graviton Lambda", "Lambda cost optimization with Reserved Instances" |
- ### MCP Servers
- | Server | Purpose |
- | ---------------------- | ------------------------------------------------------------------------------------ |
- | **aws-serverless-mcp** | Serverless development guidance, project scaffolding, IaC generation, and deployment |
- ### Hooks
- | Hook | Trigger | Action |
- | --------------------------- | --------------------------------------------- | --------------------------------------------- |
- | **SAM template validation** | After edits to `template.yaml`/`template.yml` | Runs `sam validate` and reports errors inline |
- ## codebase-documentor-for-aws
- Analyzes codebases to generate structured technical documentation with source-of-truth citations linking every finding back to the exact code that produced it. Uses an outline-driven pipeline to systematically analyze codebases of any size with a persistent task board for resumability.
- ### Agent Skill Triggers
- | Agent Skill | Triggers |
- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
- | **document-service** | "analyze this codebase", "generate documentation", "document this service", "I inherited this code", "help me understand this system", "draw the architecture", "what does this system look like" |
- ### MCP Servers
- | Server | Purpose |
- | ---------------- | -------------------------------------------------------------------- |
- | **awsknowledge** | AWS service descriptions, architecture guidance, documentation links |
- | **awsiac** | CDK/CloudFormation resource schema validation and IaC best practices |
- ## databases-on-aws
- Database guidance for the AWS database portfolio. Design schemas, execute queries, handle migrations, build applications, and choose the right database for your workload. Currently includes Aurora DSQL — a serverless, PostgreSQL-compatible distributed SQL database.
- ### Agent Skill Triggers
- | Agent Skill | Triggers |
- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
- | **dsql** | "Aurora DSQL", "DSQL schema", "distributed SQL database", "serverless PostgreSQL-compatible database", "create DSQL table", "migrate to DSQL" |
- ### MCP Servers
- | Server | Purpose |
- | ---------------- | -------------------------------------------------------------------------------- |
- | **awsknowledge** | AWS documentation, architecture guidance, and best practices |
- | **aurora-dsql** | Direct database operations — queries, schema, transactions (disabled by default) |
- ### Hooks
- | Hook | Trigger | Action |
- | ----------------------- | --------------------------- | -------------------------------------------------------- |
- | **Schema verification** | After `transact` operations | Prompts verification of schema changes and affected rows |
- ## deploy-on-aws
- Equips agents with the skills to accelerate AWS deployment - recommending AWS architectures and services, estimating costs, generating Infrastructure as Code (CDK or CloudFormation), and guiding you through deployment.
- ### Workflow
- 1. **Analyze** - Scan codebase for framework, database, dependencies
- 2. **Recommend** - Select AWS services with concise rationale
- 3. **Estimate** - Show cost estimates before proceeding
- 4. **Generate** - Write IaC code (CDK/CloudFormation)
- 5. **Deploy** - Execute with user confirmation
- ### Agent Skill Triggers
- | Agent Skill | Triggers |
- | ----------- | --------------------------------------------------------------------------------------------------------------------- |
- | **deploy** | "deploy to AWS", "host on AWS", "run this on AWS", "AWS architecture", "estimate AWS cost", "generate infrastructure" |
- ### MCP Servers
- | Server | Purpose |
- | ---------------- | ------------------------------------------------------------ |
- | **awsknowledge** | AWS documentation, architecture guidance, and best practices |
- | **awspricing** | Real-time AWS service pricing for cost estimates |
- | **aws-iac-mcp** | IaC best practices for CDK/CloudFormation |
- ## sagemaker-ai
- Build, train, and deploy AI models with deep AWS AI/ML expertise brought directly into your coding assistants, covering the surface area of [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/ai/). [Learn more](./plugins/sagemaker-ai/README.md).
- ## Requirements
- - Claude Code >=2.1.29 or [Cursor >= 2.5](https://cursor.com/changelog/2-5)
- - AWS CLI configured with appropriate credentials
- ## Troubleshooting
- Having issues with plugin installation or usage? See our [troubleshooting guide](./docs/TROUBLESHOOTING.md) for common solutions.
- ## Contributing
- Big shout out to our awesome contributors! Thank you for making this project better!
- Contributions of all kinds are welcome! Check out our [contributor guide](./CONTRIBUTING.md) for more information.
- ## Developer guide
- If you want to add a new plugin to the library, check out our [design guidelines](./docs/DESIGN_GUIDELINES.md) and [development guide](./docs/DEVELOPMENT_GUIDE.md).
- ## Maintainers
- Repository reviewers, maintainers, and admins can find PR review workflows, merge rules, and CI/CD documentation in the [maintainers guide](./docs/MAINTAINERS_GUIDE.md).
- ## Administrators
- GitHub environment and AWS account configuration are documented in the [administrators guide](./docs/ADMINISTRATORS_GUIDE.md).
- ## Additional Resources
- - [Introducing Agent Plugins for AWS Blog](https://aws.amazon.com/blogs/developer/introducing-agent-plugins-for-aws/)
- - [Extend Cursor with plugins ft. AWS](https://cursor.com/blog/marketplace)
- - [MCP Servers for AWS](https://github.com/awslabs/mcp)
- - [Claude Code plugins documentation](https://code.claude.com/docs/en/plugins)
- - [Cursor plugins documentation](https://cursor.com/docs/plugins)
- ## License
- This project is licensed under the Apache-2.0 License.
plugins/sagemaker-ai/skills/hyperpod-cluster-debugger/scripts/diagnose-cluster.sh · Bundled script · 1,621 lines · 1 finding marked
- #!/usr/bin/env bash
- # diagnose-cluster.sh — read-only HyperPod cluster-level diagnostic.
- # See SKILL.md and references/cluster-diagnostics-detail.md for remediation.
- #
- # Exit codes:
- # 0 No critical (P0/P1) failures; P2 warnings are informational-only.
- # 1 One or more critical failures, or a fatal prerequisite.
- # 2 Invalid argument.
- set -euo pipefail
- for cmd in aws jq python3; do
- command -v "$cmd" &>/dev/null || {
- echo "ERROR: '$cmd' is required but not found. Install it and retry."
- exit 1
- }
- done
- # unbuffer is only needed if the Slurm-controller SSM probe runs
- # (activated when the cluster's orchestrator is Slurm). Warn at startup
- # but don't exit — EKS-only users shouldn't be blocked.
- if ! command -v unbuffer &>/dev/null; then
- echo "WARN: 'unbuffer' not found. Required for the Slurm-controller SSM probe." >&2
- echo " Install via 'yum install expect' / 'apt install expect' / 'brew install expect'." >&2
- echo " EKS diagnostics will continue; Slurm-controller-only checks will be skipped." >&2
- fi
- CLUSTER=""
- REGION="${AWS_DEFAULT_REGION:-}"
- USE_COLOR=true
- VALIDATE_MODE=false
- VALIDATE_SG_IDS=""
- VALIDATE_SUBNET_IDS=""
- VALIDATE_IAM_ROLE=""
- VALIDATE_S3_URI=""
- VALIDATE_INSTANCE_TYPE=""
- while [[ $# -gt 0 ]]; do
- case "$1" in
- --cluster) [[ $# -lt 2 ]] && { echo "ERROR: --cluster needs a value"; exit 2; }
- [[ ! "$2" =~ ^(arn:aws[a-z-]*:sagemaker:[a-z0-9-]+:[0-9]{12}:cluster/[a-z0-9]{12}|[a-zA-Z0-9]([-a-zA-Z0-9]{0,62}))$ ]] && { echo "ERROR: --cluster must be a valid HyperPod cluster name or ARN (got '$2')"; exit 2; }
- CLUSTER="$2"; shift 2 ;;
- --region) [[ $# -lt 2 ]] && { echo "ERROR: --region needs a value"; exit 2; }
- [[ ! "$2" =~ ^[a-z]{2}-[a-z]+-[0-9]+$ ]] && { echo "ERROR: --region must be a valid AWS region (got '$2')"; exit 2; }
- REGION="$2"; shift 2 ;;
- --sg-ids) [[ $# -lt 2 ]] && { echo "ERROR: --sg-ids needs a value"; exit 2; }; VALIDATE_SG_IDS="$2"; shift 2 ;;
- --subnet-ids) [[ $# -lt 2 ]] && { echo "ERROR: --subnet-ids needs a value"; exit 2; }; VALIDATE_SUBNET_IDS="$2"; shift 2 ;;
- --iam-role) [[ $# -lt 2 ]] && { echo "ERROR: --iam-role needs a value"; exit 2; }; VALIDATE_IAM_ROLE="$2"; shift 2 ;;
- --s3-uri) [[ $# -lt 2 ]] && { echo "ERROR: --s3-uri needs a value"; exit 2; }; VALIDATE_S3_URI="$2"; shift 2 ;;
- --instance-type) [[ $# -lt 2 ]] && { echo "ERROR: --instance-type needs a value"; exit 2; }; VALIDATE_INSTANCE_TYPE="$2"; shift 2 ;;
- --no-color) USE_COLOR=false; shift ;;
- --validate) VALIDATE_MODE=true; shift ;;
- -h|--help)
- cat <<'EOF'
- Usage: diagnose-cluster.sh --cluster <name-or-arn> --region <region> [--no-color]
- diagnose-cluster.sh --validate --region <region> \
- --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
- [--s3-uri s3://bucket/path/] [--instance-type ml.p5.48xlarge]
- Read-only diagnostic for HyperPod cluster-level issues: provisioning, access,
- node replacement, VPC/SG, EKS config + add-ons, SSM, CloudWatch logs. Each
- [FAIL] line in the summary includes a pointer of the form
- "→ references/cluster-diagnostics-detail.md § <section>"
- so the hyperpod-cluster-debugger skill can look up the remediation runbook.
- The script never modifies cluster state and never prints remediation commands.
- Modes:
- (default) Diagnose an existing cluster.
- --validate Pre-flight config validation (validates SGs / subnets / IAM /
- VPC endpoints / optional S3 lifecycle scripts / optional per-AZ
- instance-type capacity before creating a cluster; no cluster
- needed).
- See references/cluster-diagnostics-detail.md for full remediation runbooks.
- See references/capacity-planning.md, lifecycle-scripts.md, cloudformation-errors.md
- for deep-dive companions to sections B / C / H.
- EOF
- exit 0
- ;;
- *) echo "Unknown argument: $1"; exit 2 ;;
- esac
- done
- if [[ -z "$REGION" ]]; then
- echo "ERROR: --region is required (or set AWS_DEFAULT_REGION before running)." >&2
- exit 2
- fi
- if ! "$VALIDATE_MODE"; then
- [[ -z "$CLUSTER" ]] && echo "Usage: $0 --cluster <name-or-arn> --region <region>" && exit 1
- fi
- if ! [ -t 1 ] || [ "${TERM:-}" = "dumb" ]; then
- USE_COLOR=false
- fi
- if "$USE_COLOR"; then
- RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
- CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
- else
- RED=''; GREEN=''; YELLOW=''; CYAN=''; BOLD=''; NC=''
- fi
- CALLER_IDENTITY=$(aws sts get-caller-identity --output json 2>&1) || {
- echo -e "${RED}ERROR: AWS credentials not configured or expired.${NC}"
- echo "$CALLER_IDENTITY"
- echo ""
- echo "→ references/cluster-diagnostics-detail.md § D (EKS Access / kubectl) for credential setup"
- exit 1
- }
- CALLER_ARN=$(echo "$CALLER_IDENTITY" | python3 -c "import sys,json; print(json.load(sys.stdin).get('Arn','unknown'))" 2>/dev/null || echo "unknown")
- CRITICAL_FAILURES=0
- WARNINGS=0
- ISSUES_FOUND=()
- pass() { echo -e " ${GREEN}[PASS]${NC} $1${2:+ — $2}"; }
- fail() { CRITICAL_FAILURES=$((CRITICAL_FAILURES+1)); echo -e " ${RED}[FAIL]${NC} $1${2:+ — $2}"; }
- warn() { WARNINGS=$((WARNINGS+1)); echo -e " ${YELLOW}[WARN]${NC} $1${2:+ — $2}"; }
- info() { echo -e " ${CYAN}[INFO]${NC} $1${2:+ — $2}"; }
- header() { echo ""; echo -e "${BOLD}--- $1 ---${NC}"; }
- section() { echo ""; echo -e "${BOLD}=== $1 ===${NC}"; }
- add_issue() {
- local priority="${2:-P1}"
- ISSUES_FOUND+=("${priority}|$1")
- }
- _CD_TEMP_FILES=()
- trap '[[ ${#_CD_TEMP_FILES[@]} -gt 0 ]] && rm -f "${_CD_TEMP_FILES[@]}" 2>/dev/null || true' EXIT
- # Run a shell command on a HyperPod instance via SSM. Payload is base64-encoded
- # so shell metacharacters in the command are safely passed through argv.
- ssm_run_on_node() {
- local iid="$1" grp="$2" cmd="$3"
- [[ -z "$iid" || -z "$grp" || -z "$cmd" ]] && return 1
- [[ ! "$iid" =~ ^i-[0-9a-f]{8,17}$ ]] && return 1
- [[ -z "${CLUSTER_ID:-}" ]] && return 1
- [[ ! "$grp" =~ ^[A-Za-z0-9._-]+$ ]] && return 1
- local target="sagemaker-cluster:${CLUSTER_ID}_${grp}-${iid}"
- local tmp; tmp=$(mktemp 2>/dev/null) || return 1
- chmod 600 "$tmp" 2>/dev/null || true
- _CD_TEMP_FILES+=("$tmp")
- local cmd_b64
- cmd_b64=$(printf '%s' "$cmd" | base64 | tr -d '\n') || return 1
- local remote="bash -c \"echo $cmd_b64 | base64 -d | bash\""
- Note: Quoted as data in code: Decodes base64 and runs the result in a shell
- Line 146 on GitHub
- python3 -c "import json,sys; print(json.dumps({'command':[sys.argv[1]]}))" "$remote" > "$tmp" || return 1
- # unbuffer avoids the session-manager-plugin "Cannot perform start session:
- # EOF" race. Only required on Slurm clusters (controller probe); guard at
- # call site so EKS-only users aren't blocked if unbuffer is absent.
- local _ssm_wrap=""
- command -v unbuffer >/dev/null 2>&1 && _ssm_wrap="unbuffer"
- local attempt=0 out rc
- while (( attempt < 5 )); do
- out=$($_ssm_wrap timeout 180 aws ssm start-session \
- --target "$target" \
- --document-name AWS-StartNonInteractiveCommand \
- --parameters "file://$tmp" \
- --region "$REGION" 2>&1)
- rc=$?
- # Retry transient SSM transport errors (rc=0 with EOF/plugin/timeout in stdout).
- if (( rc == 0 )) && ! echo "$out" | grep -qiE "Cannot perform start session|EOF$|SessionManagerPlugin is not found|ERROR: Unable to|i/o timeout"; then
- # Strip SSM session banners and the echoed base64 command line.
- echo "$out" | grep -vE '^(Starting session with SessionId:|Exiting session with sessionId:|\s*$)' \
- | grep -vE "^(bash -c \"echo [A-Za-z0-9+/=]+ \| base64 -d \| bash\"|echo '[A-Za-z0-9+/=]+'|[A-Za-z0-9+/=]{40,}={0,2})[[:space:]]*\|?[[:space:]]*base64?[[:space:]]*-?d?[[:space:]]*\|?[[:space:]]*bash\"?\$" || true
- return 0
- fi
- if echo "$out" | grep -qiE "ThrottlingException|RequestLimitExceeded|InternalFailure|InternalError|ServiceUnavailable|TooManyUpdates|Cannot perform start session|EOF$|SessionManagerPlugin is not found|i/o timeout"; then
- attempt=$((attempt + 1))
- sleep $((attempt * 3))
- continue
- fi
- echo "$out" >&2
- return 1
- done
- return 1
- }
- # Check SG self-referencing rules. Reads SG JSON from stdin, outputs PASS/FAIL/WARN lines.
- check_sg_self_ref() {
- local sg_id="$1"
- SG_CHECK_ID="$sg_id" python3 -c "
- import sys, json, os
- sg_id = os.environ['SG_CHECK_ID']
- sgs = json.load(sys.stdin).get('SecurityGroups', [])
- if not sgs:
- print(f'SKIP:Could not describe {sg_id}')
- sys.exit(0)
- sg = sgs[0]
- inbound_self = any(
- any(p.get('GroupId') == sg_id for p in r.get('UserIdGroupPairs', []))
- for r in sg.get('IpPermissions', [])
- )
- outbound_self = any(
- any(p.get('GroupId') == sg_id for p in r.get('UserIdGroupPairs', []))
- for r in sg.get('IpPermissionsEgress', [])
- )
- outbound_all = any(
- any(r2.get('CidrIp') == '0.0.0.0/0' for r2 in r.get('IpRanges', []))
- for r in sg.get('IpPermissionsEgress', [])
- )
- if inbound_self: print(f'PASS:inbound:SG {sg_id}: Inbound self-ref present')
- else: print(f'FAIL:inbound:SG {sg_id}: Inbound self-ref MISSING — required for inter-node communication')
- if outbound_self: print(f'PASS:outbound:SG {sg_id}: Outbound self-ref present')
- else: print(f'FAIL:outbound:SG {sg_id}: Outbound self-ref MISSING — required for EFA RDMA traffic')
- if outbound_all: print(f'PASS:internet:SG {sg_id}: Outbound 0.0.0.0/0 present')
- else: print(f'WARN:internet:SG {sg_id}: Outbound 0.0.0.0/0 missing — may be needed for AWS API calls')
- " 2>/dev/null || echo ""
- }
- # AWS API wrapper that detects permission failures
- aws_check() {
- local api_label="$1"; shift
- local result
- result=$("$@" 2>&1)
- local rc=$?
- if [[ $rc -ne 0 ]]; then
- if echo "$result" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized|AuthorizationError"; then
- warn "$api_label" "IAM permission denied — results may be incomplete"
- add_issue "Missing IAM permission for $api_label → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
- echo ""
- return 1
- fi
- echo "$result"
- return "$rc"
- fi
- echo "$result"
- }
- if "$VALIDATE_MODE"; then
- section "HyperPod Pre-Creation Validation"
- echo -e "Region: ${BOLD}${REGION}${NC}"
- echo -e "Caller: ${BOLD}${CALLER_ARN}${NC}"
- echo -e "Time: $(date -u +"%Y-%m-%dT%H:%M:%SZ")"
- if [[ -n "$VALIDATE_SG_IDS" ]]; then
- header "V1. Security Group Rules"
- for SG in $(echo "$VALIDATE_SG_IDS" | tr ',' ' '); do
- SG_JSON=$(aws_check "describe-sg-$SG" aws ec2 describe-security-groups \
- --group-ids "$SG" --region "$REGION" --output json) || continue
- _SG_CHECK_OUT=$(echo "$SG_JSON" | check_sg_self_ref "$SG")
- while IFS=: read -r level check msg; do
- [[ -z "$level" ]] && continue
- case "$level" in
- PASS) pass "$msg" ;;
- FAIL)
- fail "$msg"
- add_issue "SG $SG missing $check self-ref → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
- ;;
- WARN) warn "$msg" ;;
- esac
- done <<< "$_SG_CHECK_OUT"
- done
- fi
- if [[ -n "$VALIDATE_SUBNET_IDS" ]]; then
- header "V2. Subnet Configuration"
- IFS=',' read -ra _subnet_ids <<< "$VALIDATE_SUBNET_IDS"
- SUB_JSON=$(aws_check "describe-subnets" aws ec2 describe-subnets \
- --subnet-ids "${_subnet_ids[@]}" \
- --region "$REGION" --output json) || SUB_JSON='{"Subnets":[]}'
- echo "$SUB_JSON" | python3 -c "
- import sys, json
- subnets = json.load(sys.stdin).get('Subnets', [])
- vpcs = set()
- azs = set()
- for s in subnets:
- sid = s.get('SubnetId', '?')
- vpc = s.get('VpcId', '?')
- az = s.get('AvailabilityZone', '?')
- free = s.get('AvailableIpAddressCount', 0)
- vpcs.add(vpc)
- azs.add(az)
- status = 'LOW' if free < 10 else 'OK'
- print(f'SUBNET:{sid}:{vpc}:{az}:{free}:{status}')
- print(f'VPC_COUNT:{len(vpcs)}')
- print(f'AZ_COUNT:{len(azs)}')
- " 2>/dev/null | while IFS=: read -r tag rest; do
- case "$tag" in
- SUBNET)
- IFS=: read -r sid _vpc az free status <<< "$rest"
- if [[ "$status" == "LOW" ]]; then
- warn "Subnet $sid (AZ=$az) — only $free IPs available"
- else
- pass "Subnet $sid" "AZ=$az FreeIPs=$free"
- fi
- ;;
- VPC_COUNT)
- if [[ "$rest" -gt 1 ]]; then
- fail "Subnets are in DIFFERENT VPCs — all must be in same VPC"
- add_issue "Subnets in different VPCs → references/cluster-diagnostics-detail.md § B (Capacity & AZ)" "P0"
- else
- pass "All subnets in same VPC"
- fi
- ;;
- AZ_COUNT)
- info "Subnets span $rest availability zone(s)"
- ;;
- esac
- done
- fi
- if [[ -n "$VALIDATE_IAM_ROLE" ]]; then
- header "V3. IAM Execution Role"
- ROLE_NAME=$(echo "$VALIDATE_IAM_ROLE" | awk -F/ '{print $NF}')
- ROLE_INFO=$(aws_check "get-role" aws iam get-role --role-name "$ROLE_NAME" --output json) || ROLE_INFO=""
- if [[ -n "$ROLE_INFO" ]]; then
- pass "IAM role exists" "$ROLE_NAME"
- TRUST_SM=$(echo "$ROLE_INFO" | python3 -c "
- import sys,json
- doc=json.load(sys.stdin).get('Role',{}).get('AssumeRolePolicyDocument',{})
- stmts=doc.get('Statement',[])
- for s in stmts:
- p=s.get('Principal',{})
- svc=p.get('Service',[]) if isinstance(p.get('Service'), list) else [p.get('Service','')]
- if 'sagemaker.amazonaws.com' in svc:
- print('true')
- break
- else:
- print('false')
- " 2>/dev/null)
- if [[ "$TRUST_SM" == "true" ]]; then
- pass "Trust policy" "allows sagemaker.amazonaws.com"
- else
- fail "Trust policy" "missing sagemaker.amazonaws.com — cluster creation will fail"
- add_issue "IAM execution role trust policy missing sagemaker.amazonaws.com → references/cluster-diagnostics-detail.md § H (CloudFormation Errors / SLR)" "P0"
- fi
- POLICIES=$(aws_check "list-attached-role-policies-$ROLE_NAME" \
- aws iam list-attached-role-policies --role-name "$ROLE_NAME" \
- --query 'AttachedPolicies[*].PolicyArn' --output text) || POLICIES=""
- if [[ -n "$POLICIES" ]]; then
- if echo "$POLICIES" | grep -q "AmazonSageMakerClusterInstanceRolePolicy"; then
- pass "Managed policy" "AmazonSageMakerClusterInstanceRolePolicy attached"
- else
- warn "Managed policy" "AmazonSageMakerClusterInstanceRolePolicy not attached — cluster bootstrap will fail"
- add_issue "IAM execution role missing AmazonSageMakerClusterInstanceRolePolicy → references/cluster-diagnostics-detail.md § H (CloudFormation Errors / SLR)" "P0"
- fi
- if echo "$POLICIES" | grep -q "AmazonSSMManagedInstanceCore"; then
- pass "Managed policy" "AmazonSSMManagedInstanceCore attached (SSM access)"
- else
- warn "Managed policy" "AmazonSSMManagedInstanceCore not attached — SSM node access will not work"
- add_issue "IAM execution role missing AmazonSSMManagedInstanceCore → references/cluster-diagnostics-detail.md § F (SSM Connectivity)" "P1"
- fi
- fi
- else
- fail "IAM role" "cannot find role '$ROLE_NAME'"
- add_issue "IAM execution role not found → references/cluster-diagnostics-detail.md § H (CloudFormation Errors / SLR)" "P0"
- fi
- fi
- if [[ -n "$VALIDATE_SUBNET_IDS" ]]; then
- header "V4. VPC Endpoints"
- FIRST_SUBNET=$(echo "$VALIDATE_SUBNET_IDS" | cut -d, -f1)
- VPC_FOR_EP=$(aws ec2 describe-subnets --subnet-ids "$FIRST_SUBNET" \
- --region "$REGION" --query 'Subnets[0].VpcId' --output text 2>/dev/null || echo "")
- if [[ -n "$VPC_FOR_EP" && "$VPC_FOR_EP" != "None" ]]; then
- ENDPOINTS=$(aws ec2 describe-vpc-endpoints \
- --filters "Name=vpc-id,Values=$VPC_FOR_EP" \
- --region "$REGION" \
- --query "VpcEndpoints[?State==\`available\`].ServiceName" \
- --output text 2>/dev/null || echo "")
- for SVC in s3 ssm ssmmessages ec2messages; do
- if echo "$ENDPOINTS" | grep -qE "(^|[.])${SVC}($|[[:space:]])"; then
- pass "VPC endpoint: $SVC"
- else
- warn "VPC endpoint: $SVC" "not found — needed for private VPC clusters"
- add_issue "Missing VPC endpoint for $SVC → references/cluster-diagnostics-detail.md § C (Lifecycle Scripts)" "P2"
- fi
- done
- fi
- fi
- if [[ -n "$VALIDATE_INSTANCE_TYPE" && -n "$VALIDATE_SUBNET_IDS" ]]; then
- header "V5. Instance-Type Capacity per AZ"
- # EC2 API takes the bare type, not the ml. prefix.
- EC2_TYPE="${VALIDATE_INSTANCE_TYPE#ml.}"
- AZ_OFFERINGS=$(aws_check "describe-instance-type-offerings-$EC2_TYPE" \
- aws ec2 describe-instance-type-offerings \
- --location-type availability-zone \
- --filters "Name=instance-type,Values=${EC2_TYPE}" \
- --region "$REGION" \
- --query 'InstanceTypeOfferings[*].Location' --output text) || AZ_OFFERINGS=""
- if [[ -z "$AZ_OFFERINGS" ]]; then
- fail "Instance type $VALIDATE_INSTANCE_TYPE" "not offered in region $REGION"
- add_issue "$VALIDATE_INSTANCE_TYPE is not offered in any AZ in $REGION → references/capacity-planning.md" "P0"
- else
- info "$VALIDATE_INSTANCE_TYPE available in AZ(s): $AZ_OFFERINGS"
- IFS=',' read -ra _subnet_ids <<< "$VALIDATE_SUBNET_IDS"
- SUB_AZ_JSON=$(aws_check "describe-subnets-validate" aws ec2 describe-subnets \
- --subnet-ids "${_subnet_ids[@]}" \
- --region "$REGION" \
- --query 'Subnets[*].{SubnetId:SubnetId,AZ:AvailabilityZone}' --output json) || SUB_AZ_JSON="[]"
- MATCHED=0
- while IFS=$'\t' read -r sid az; do
- [[ -z "$sid" ]] && continue
- if echo "$AZ_OFFERINGS" | tr '\t' '\n' | grep -qx "$az"; then
- pass "Subnet $sid (AZ=$az)" "$VALIDATE_INSTANCE_TYPE is available"
- MATCHED=$((MATCHED+1))
- else
- fail "Subnet $sid (AZ=$az)" "$VALIDATE_INSTANCE_TYPE NOT offered here"
- add_issue "Subnet $sid AZ=$az does not offer $VALIDATE_INSTANCE_TYPE → references/capacity-planning.md" "P0"
- fi
- done < <(echo "$SUB_AZ_JSON" | python3 -c "
- import sys, json
- for s in json.load(sys.stdin):
- print(f\"{s.get('SubnetId','')}\t{s.get('AZ','')}\")
- " 2>/dev/null)
- if [[ $MATCHED -eq 0 ]]; then
- warn "No provided subnet is in an AZ that offers $VALIDATE_INSTANCE_TYPE — cluster creation will fail with Insufficient capacity / No subnets in the capacity AZ"
- fi
- fi
- fi
- if [[ -n "$VALIDATE_S3_URI" ]]; then
- header "V6. S3 Lifecycle Scripts"
- if [[ ! "$VALIDATE_S3_URI" =~ ^s3:// ]]; then
- fail "S3 URI" "must start with s3:// (got '$VALIDATE_S3_URI')"
- add_issue "S3 URI is not a valid s3:// URI → references/lifecycle-scripts.md" "P0"
- else
- S3_URI_NORM="${VALIDATE_S3_URI%/}/"
- info "S3 URI: $S3_URI_NORM"
- S3_LIST=$(aws_check "s3-ls-$S3_URI_NORM" \
- aws s3 ls "$S3_URI_NORM" --region "$REGION") || S3_LIST=""
- if [[ -z "$S3_LIST" ]]; then
- fail "S3 access" "cannot list $S3_URI_NORM — bucket missing, permissions denied, or empty prefix"
- add_issue "S3 URI not accessible or empty: $S3_URI_NORM → references/lifecycle-scripts.md" "P0"
- else
- pass "S3 access" "prefix is listable"
- if echo "$S3_LIST" | grep -q "on_create.sh"; then
- pass "on_create.sh" "entry script present"
- TMPFILE=$(mktemp)
- if aws s3 cp "${S3_URI_NORM}on_create.sh" "$TMPFILE" \
- --region "$REGION" --only-show-errors 2>/dev/null; then
- if file "$TMPFILE" | grep -q "CRLF"; then
- fail "on_create.sh" "has Windows CRLF line endings — will fail on Linux"
- add_issue "on_create.sh has CRLF line endings → references/lifecycle-scripts.md" "P0"
- else
- pass "on_create.sh" "Unix line endings"
- fi
- if head -1 "$TMPFILE" | grep -q "^#!"; then
- pass "on_create.sh" "shebang present"
- else
- warn "on_create.sh" "missing shebang (#!/bin/bash)"
- add_issue "on_create.sh missing shebang → references/lifecycle-scripts.md" "P1"
- fi
- else
- warn "on_create.sh" "could not download for inspection"
- fi
- rm -f "$TMPFILE"
- else
- fail "on_create.sh" "entry script NOT FOUND at $S3_URI_NORM — cluster creation will fail"
- add_issue "Missing on_create.sh at $S3_URI_NORM → references/lifecycle-scripts.md" "P0"
- fi
- if echo "$S3_LIST" | grep -q "lifecycle_script.py"; then
- pass "Orchestrator script" "lifecycle_script.py present (Slurm)"
- elif echo "$S3_LIST" | grep -q "on_create_main.sh"; then
- pass "Orchestrator script" "on_create_main.sh present (EKS)"
- else
- warn "Orchestrator script" "neither lifecycle_script.py (Slurm) nor on_create_main.sh (EKS) found at $S3_URI_NORM"
- add_issue "Missing orchestrator-specific lifecycle script at $S3_URI_NORM → references/lifecycle-scripts.md" "P1"
- fi
- fi
- fi
- fi
- echo ""
- echo -e "${BOLD}========================================${NC}"
- echo -e "${BOLD} VALIDATION SUMMARY ${NC}"
- echo -e "${BOLD}========================================${NC}"
- echo ""
- echo -e " Results: ${RED}${CRITICAL_FAILURES} critical${NC} | ${YELLOW}${WARNINGS} warnings${NC}"
- echo -e " Mode: READ-ONLY (no changes made; each [FAIL] points to a references section)"
- echo ""
- if [[ ${#ISSUES_FOUND[@]} -gt 0 ]]; then
- echo -e "${BOLD} Issues:${NC}"
- for priority in P0 P1 P2; do
- for issue in "${ISSUES_FOUND[@]}"; do
- if [[ "$issue" == "${priority}|"* ]]; then
- desc="${issue#*|}"
- case "$priority" in
- P0) echo -e " ${RED}[${priority}]${NC} $desc" ;;
- P1) echo -e " ${YELLOW}[${priority}]${NC} $desc" ;;
- P2) echo -e " [${priority}] $desc" ;;
- esac
- fi
- done
- done
- echo ""
- fi
- if [[ $CRITICAL_FAILURES -eq 0 ]]; then
- echo -e " ${GREEN}${BOLD}Pre-flight validation passed. Safe to create cluster.${NC}"
- else
- echo -e " ${RED}${BOLD}Fix P0 issues above before creating the cluster.${NC}"
- fi
- echo ""
- exit "$([[ $CRITICAL_FAILURES -eq 0 ]] && echo 0 || echo 1)"
- fi
- section "HyperPod Cluster Diagnostics (read-only)"
- echo -e "Cluster: ${BOLD}${CLUSTER}${NC}"
- echo -e "Region: ${BOLD}${REGION}${NC}"
- echo -e "Time: $(date -u +"%Y-%m-%dT%H:%M:%SZ")"
- echo -e "${CYAN} No cluster state will be modified. Each issue line below includes a${NC}"
- echo -e "${CYAN} pointer to references/cluster-diagnostics-detail.md for remediation.${NC}"
- header "1. Cluster Identity & Status"
- CLUSTER_JSON=$(aws sagemaker describe-cluster \
- --cluster-name "$CLUSTER" \
- --region "$REGION" \
- --cli-read-timeout 30 \
- --output json 2>&1) || {
- echo -e "${RED}ERROR: Could not describe cluster '$CLUSTER' in region '$REGION'${NC}"
- echo "$CLUSTER_JSON" | head -3
- echo ""
- if echo "$CLUSTER_JSON" | grep -qiE "ResourceNotFound|Cluster with name .* not found"; then
- echo "Available clusters in $REGION:"
- aws sagemaker list-clusters --region "$REGION" \
- --query 'ClusterSummaries[*].{Name:ClusterName,Status:ClusterStatus}' \
- --output table 2>/dev/null || echo " (unable to list clusters — check IAM)"
- else
- echo "Verify:"
- echo " 1. Cluster name is correct (use: aws sagemaker list-clusters --region $REGION)"
- echo " 2. Region is correct"
- echo " 3. IAM permissions include sagemaker:DescribeCluster"
- fi
- exit 1
- }
- CLUSTER_ARN=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ClusterArn',''))" 2>/dev/null)
- CLUSTER_ID=$(echo "$CLUSTER_ARN" | awk -F'/' '{print $NF}')
- if [[ -z "$CLUSTER_ID" ]]; then
- echo "ERROR: Could not extract cluster ID from ARN '$CLUSTER_ARN'. Verify the cluster name/ARN."
- exit 1
- fi
- CLUSTER_STATUS=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ClusterStatus','unknown'))" 2>/dev/null)
- ORCHESTRATOR=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); o=d.get('Orchestrator',{}); print('EKS' if 'Eks' in o else 'Slurm')" 2>/dev/null)
- NODE_RECOVERY=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- # Prefer cluster-level NodeRecovery (the API's canonical location); fall back to
- # per-InstanceGroup only when top-level is absent. Reading only per-group yields
- # 'Unknown' on every cluster because the field is null at group level when set
- # cluster-wide.
- top=d.get('NodeRecovery')
- if top:
- print(top)
- else:
- groups=d.get('InstanceGroups',[])
- recoveries={g.get('NodeRecovery') for g in groups if g.get('NodeRecovery')}
- print(','.join(sorted(recoveries)) if recoveries else 'Unknown')
- " 2>/dev/null || echo "Unknown")
- info "ARN: $CLUSTER_ARN"
- info "Cluster ID: $CLUSTER_ID"
- info "Status: $CLUSTER_STATUS"
- info "Orchestrator: $ORCHESTRATOR"
- info "NodeRecovery: $NODE_RECOVERY"
- # Flag auto-recovery disabled regardless of orchestrator.
- if [[ "$NODE_RECOVERY" == *"None"* && "$NODE_RECOVERY" == *"Automatic"* ]]; then
- warn "NodeRecovery" "mixed settings — some instance groups have recovery disabled"
- add_issue "NodeRecovery disabled on some instance groups → references/cluster-diagnostics-detail.md § G (Node Replacement)" "P2"
- elif [[ "$NODE_RECOVERY" == *"None"* ]]; then
- warn "NodeRecovery" "disabled on all instance groups — auto-replacement won't trigger"
- add_issue "NodeRecovery disabled → references/cluster-diagnostics-detail.md § G (Node Replacement)" "P2"
- fi
- CREATION_TIME=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- ct=d.get('CreationTime','')
- print(ct if ct else '')
- " 2>/dev/null || echo "")
- LAST_MODIFIED_TIME=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- lm=d.get('LastModifiedTime','')
- print(lm if lm else '')
- " 2>/dev/null || echo "")
- STUCK_THRESHOLD_SECONDS=3600
- is_stuck() {
- local creation_time="$1"
- if [[ -z "$creation_time" ]]; then echo "false"; return; fi
- CREATION_TS="$creation_time" THRESHOLD="$STUCK_THRESHOLD_SECONDS" python3 -c "
- import os
- from datetime import datetime, timezone
- ct = os.environ['CREATION_TS']
- threshold = int(os.environ['THRESHOLD'])
- try:
- ct=ct.replace('+00:00','Z').rstrip('Z')
- if '.' in ct: ct=ct[:ct.index('.')+7]
- created=datetime.fromisoformat(ct).replace(tzinfo=timezone.utc)
- elapsed=(datetime.now(timezone.utc)-created).total_seconds()
- print('true' if elapsed > threshold else 'false')
- except (ValueError, TypeError):
- # Unparseable timestamp — assume not stuck rather than abort the whole run.
- print('false')
- " 2>/dev/null || echo "false"
- }
- case "$CLUSTER_STATUS" in
- InService) pass "Cluster status" "InService" ;;
- Creating)
- STUCK=$(is_stuck "$CREATION_TIME")
- if [[ "$STUCK" == "true" ]]; then
- fail "Cluster status" "Creating for over 1 hour — likely stuck"
- add_issue "Cluster stuck in Creating > 1hr → references/cluster-diagnostics-detail.md § E (Cluster Provisioning), § H (CloudFormation)" "P0"
- else
- warn "Cluster status" "Creating — cluster is still being provisioned"
- add_issue "Cluster still creating → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P1"
- fi ;;
- Updating)
- STUCK=$(is_stuck "${LAST_MODIFIED_TIME:-$CREATION_TIME}")
- if [[ "$STUCK" == "true" ]]; then
- fail "Cluster status" "Updating — check if operation is stuck"
- add_issue "Cluster may be stuck Updating → references/cluster-diagnostics-detail.md § E (Cluster Provisioning), § H (CloudFormation)" "P1"
- else
- warn "Cluster status" "Updating — cluster operation in progress"
- fi ;;
- Failed) fail "Cluster status" "Failed — check events and CloudFormation"; add_issue "Cluster FAILED → references/cluster-diagnostics-detail.md § E (Cluster Provisioning), § H (CloudFormation)" "P0" ;;
- Deleting)
- STUCK=$(is_stuck "${LAST_MODIFIED_TIME:-$CREATION_TIME}")
- if [[ "$STUCK" == "true" ]]; then
- warn "Cluster status" "Deleting for extended time — may be blocked by VPC ENI dependencies"
- add_issue "Cluster stuck Deleting → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P1"
- else
- warn "Cluster status" "Deleting"
- fi ;;
- RollingBack) warn "Cluster status" "RollingBack — update is being rolled back"; add_issue "Cluster RollingBack → references/cluster-diagnostics-detail.md § J (AMI & Cluster Updates)" "P1" ;;
- *RollbackFailed*|*MaintenanceFailed*)
- fail "Cluster status" "$CLUSTER_STATUS — cluster is stuck in a non-recoverable state"
- add_issue "Cluster stuck in $CLUSTER_STATUS → references/cluster-diagnostics-detail.md § J (AMI & Cluster Updates)" "P0" ;;
- *) warn "Cluster status" "$CLUSTER_STATUS" ;;
- esac
- EKS_NAME=""
- if [[ "$ORCHESTRATOR" == "EKS" ]]; then
- EKS_NAME=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- arn=d.get('Orchestrator',{}).get('Eks',{}).get('ClusterArn','')
- print(arn.split('/')[-1] if arn else '')
- " 2>/dev/null || echo "")
- if [[ -n "$EKS_NAME" ]]; then
- info "EKS Cluster: $EKS_NAME"
- fi
- fi
- header "2. Instance Groups & Node Health"
- echo "$CLUSTER_JSON" | python3 -c "
- import sys, json
- d = json.load(sys.stdin)
- groups = d.get('InstanceGroups', [])
- if not groups:
- print(' No instance groups found')
- else:
- for g in groups:
- name = g.get('InstanceGroupName', '?')
- itype = g.get('InstanceType', '?')
- target = g.get('TargetCount', 0)
- current = g.get('CurrentCount', 0)
- status = g.get('Status', g.get('InstanceGroupStatus', '?'))
- threads = g.get('ThreadsPerCore', '?')
- # TargetStateCount is the count the service is working toward when a
- # resize is in flight; print when it differs from TargetCount.
- tstate = g.get('TargetStateCount', None)
- # Note: NodeRecovery is a cluster-level field in the DescribeCluster
- # response, not per-group; shown on the cluster header line above.
- print(f' {name}: type={itype} target={target} current={current} status={status} threads/core={threads}')
- if tstate is not None and tstate != target:
- print(f' TargetStateCount={tstate} (resize in progress)')
- if current < target:
- print(f' Current count ({current}) < target ({target}) — instances may still be provisioning or failed')
- " 2>/dev/null
- # Check node-level details. Paginate — default page is small and large clusters
- # silently truncate, which would break dangling-node reconciliation below.
- fetch_all_cluster_nodes_cd() {
- local merged='[]' token='' page_json combined i=0
- local max_pages=200 # 200 × 100 = 20 000 nodes, supports 7k+ clusters
- while (( i < max_pages )); do
- if [[ -n "$token" ]]; then
- page_json=$(aws sagemaker list-cluster-nodes \
- --cluster-name "$CLUSTER" --region "$REGION" \
- --max-results 100 --next-token "$token" \
- --cli-read-timeout 30 --output json 2>&1) || break
- else
- page_json=$(aws sagemaker list-cluster-nodes \
- --cluster-name "$CLUSTER" --region "$REGION" \
- --max-results 100 \
- --cli-read-timeout 30 --output json 2>&1) || break
- fi
- if echo "$page_json" | grep -qiE "AccessDenied|not authorized|UnauthorizedAccess"; then
- echo "__AUTH_DENIED__"
- return 1
- fi
- # Merge via stdin (NUL-delimited blobs) instead of argv — argv is capped at
- # ARG_MAX (~128KB on Linux), which fails at ~500 nodes of accumulated JSON.
- # Large clusters (7k+) need this path to avoid silent truncation.
- combined=$(printf '%s\0%s' "$merged" "$page_json" | python3 -c "
- import sys, json
- blob = sys.stdin.buffer.read()
- try:
- a, b = blob.split(b'\0', 1)
- merged = json.loads(a)
- page = json.loads(b)
- except (json.JSONDecodeError, ValueError):
- sys.exit(2)
- merged.extend(page.get('ClusterNodeSummaries', []))
- print(json.dumps(merged))
- print(page.get('NextToken', ''))
- " 2>/dev/null) || break
- merged=$(printf '%s\n' "$combined" | sed -n '1p')
- token=$(printf '%s\n' "$combined" | sed -n '2p')
- i=$((i+1))
- [[ -z "$token" ]] && break
- done
- if (( i == max_pages )) && [[ -n "$token" ]]; then
- # Surface truncation via a marker file — this function runs inside $(...)
- # (command substitution subshell), so add_issue would be lost. The parent
- # shell checks for the marker after the call returns.
- echo "WARN: list-cluster-nodes truncated at ${max_pages} pages (~$((max_pages*100)) nodes). Diagnostic sample is incomplete for very large clusters." >&2
- : > "${_NODE_TRUNC_MARKER:-/dev/null}" 2>/dev/null || true
- fi
- printf '%s' "$merged" | python3 -c "
- import sys, json
- try:
- print(json.dumps({'ClusterNodeSummaries': json.loads(sys.stdin.read())}))
- except json.JSONDecodeError:
- print('{\"ClusterNodeSummaries\":[]}')
- " 2>/dev/null || echo '{"ClusterNodeSummaries":[]}'
- }
- _NODE_TRUNC_MARKER=$(mktemp 2>/dev/null) && _CD_TEMP_FILES+=("$_NODE_TRUNC_MARKER") || _NODE_TRUNC_MARKER=""
- export _NODE_TRUNC_MARKER
- rm -f "$_NODE_TRUNC_MARKER" 2>/dev/null || true
- NODE_LIST=$(fetch_all_cluster_nodes_cd)
- if [[ "$NODE_LIST" == "__AUTH_DENIED__" ]]; then
- warn "list-cluster-nodes" "IAM permission denied — add sagemaker:ListClusterNodes to your role"
- add_issue "Missing IAM permission for sagemaker:ListClusterNodes → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
- NODE_LIST='{"ClusterNodeSummaries":[]}'
- fi
- # Parent-shell follow-up for the truncation marker set inside the subshell.
- if [[ -n "$_NODE_TRUNC_MARKER" && -e "$_NODE_TRUNC_MARKER" ]]; then
- add_issue "Node list truncated at 200 pages (~20000 nodes); diagnostic sample incomplete → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P2"
- fi
- TOTAL_NODES=$(echo "$NODE_LIST" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('ClusterNodeSummaries',[])))" 2>/dev/null || echo 0)
- info "Total nodes reported: $TOTAL_NODES"
- UNHEALTHY_NODES=$(echo "$NODE_LIST" | python3 -c "
- import sys, json
- nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
- unhealthy = [n for n in nodes if n.get('InstanceStatus', {}).get('Status', '') not in ('Running', 'Pending')]
- if unhealthy:
- for n in unhealthy:
- nid = n.get('InstanceId', '?')
- group = n.get('InstanceGroupName', '?')
- status = n.get('InstanceStatus', {}).get('Status', '?')
- msg = n.get('InstanceStatus', {}).get('Message', '')
- print(f' {nid} ({group}): {status} {msg}')
- print(f'UNHEALTHY_COUNT={len(unhealthy)}')
- else:
- print('UNHEALTHY_COUNT=0')
- " 2>/dev/null || echo "UNHEALTHY_COUNT=0")
- UNHEALTHY_COUNT=$(echo "$UNHEALTHY_NODES" | grep "^UNHEALTHY_COUNT=" | cut -d= -f2)
- [[ -z "$UNHEALTHY_COUNT" ]] && UNHEALTHY_COUNT=0
- echo "$UNHEALTHY_NODES" | grep -v "^UNHEALTHY_COUNT=" || true
- if [[ "$UNHEALTHY_COUNT" -gt 0 ]]; then
- warn "Node health" "$UNHEALTHY_COUNT unhealthy node(s)"
- add_issue "$UNHEALTHY_COUNT unhealthy node(s) → references/cluster-diagnostics-detail.md § G (Node Replacement); delegate to hyperpod-node-debugger" "P1"
- echo "$NODE_LIST" | python3 -c "
- import sys, json
- from collections import defaultdict
- nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
- groups = defaultdict(lambda: {'total': 0, 'unhealthy': 0})
- for n in nodes:
- g = n.get('InstanceGroupName', 'unknown')
- groups[g]['total'] += 1
- st = n.get('InstanceStatus', {}).get('Status', '')
- if st not in ('Running', 'Pending', ''):
- groups[g]['unhealthy'] += 1
- for g, c in groups.items():
- if c['unhealthy'] > 0:
- pct = int(c['unhealthy'] / c['total'] * 100) if c['total'] > 0 else 0
- print(f' [WARN] Group {g}: {c[\"unhealthy\"]}/{c[\"total\"]} unhealthy ({pct}%)')
- " 2>/dev/null
- elif [[ "$TOTAL_NODES" -eq 0 && "$CLUSTER_STATUS" == "InService" ]]; then
- warn "Node health" "Cluster InService but 0 nodes reported"
- add_issue "Cluster InService but no nodes → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P1"
- else
- pass "Node health" "$TOTAL_NODES node(s), $UNHEALTHY_COUNT unhealthy"
- fi
- header "3. Cluster Events (Recent)"
- # Paginate up to 5 pages (500 events) so the event scan covers incident windows
- # longer than the default page. Long-lived clusters with rolling replacements
- # regularly generate >100 events.
- fetch_cluster_events_cd() {
- local merged='[]' token='' page_json combined i=0 denied=0
- while (( i < 5 )); do
- if [[ -n "$token" ]]; then
- page_json=$(aws sagemaker list-cluster-events \
- --cluster-name "$CLUSTER" --region "$REGION" \
- --max-results 100 --next-token "$token" \
- --cli-read-timeout 30 --output json 2>&1) || break
- else
- page_json=$(aws sagemaker list-cluster-events \
- --cluster-name "$CLUSTER" --region "$REGION" \
- --max-results 100 \
- --cli-read-timeout 30 --output json 2>&1) || break
- fi
- if echo "$page_json" | grep -qi "AccessDenied\|not authorized"; then
- denied=1
- break
- fi
- combined=$(python3 -c "
- import sys, json
- try:
- prev = json.loads(sys.argv[1])
- page = json.loads(sys.argv[2])
- except json.JSONDecodeError:
- # Malformed page response — stop paginating; caller falls through on break.
- sys.exit(2)
- prev.extend(page.get('ClusterEventSummaries', []))
- print(json.dumps(prev))
- print(page.get('NextToken',''))
- " "$merged" "$page_json" 2>/dev/null) || break
- merged=$(printf '%s\n' "$combined" | sed -n '1p')
- token=$(printf '%s\n' "$combined" | sed -n '2p')
- i=$((i+1))
- [[ -z "$token" ]] && break
- done
- if (( denied )); then
- echo "__AUTH_DENIED__"
- return 1
- fi
- python3 -c "import sys, json; print(json.dumps({'ClusterEventSummaries': json.loads(sys.argv[1])}))" "$merged" \
- 2>/dev/null || echo '{"ClusterEventSummaries":[]}'
- }
- EVENTS_JSON=$(fetch_cluster_events_cd)
- if [[ "$EVENTS_JSON" == "__AUTH_DENIED__" ]]; then
- warn "list-cluster-events" "IAM permission denied — add sagemaker:ListClusterEvents to your role"
- EVENTS_JSON='{"ClusterEventSummaries":[]}'
- fi
- EVENT_COUNT=$(echo "$EVENTS_JSON" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('ClusterEventSummaries',[])))" 2>/dev/null || echo 0)
- if [[ "$EVENT_COUNT" -eq 0 ]]; then
- info "No cluster events found"
- if [[ "$ORCHESTRATOR" == "Slurm" ]]; then
- info "(Cluster events may not be available for HyperPod Slurm clusters)"
- fi
- else
- echo "$EVENTS_JSON" | python3 -c "
- import sys, json
- events = json.load(sys.stdin).get('ClusterEventSummaries', [])
- # Issue pattern mapping
- ISSUE_PATTERNS = {
- 'EFA health checks': 'EFA health check failure → references/cluster-diagnostics-detail.md § A',
- 'Insufficient capacity': 'Capacity error → references/cluster-diagnostics-detail.md § B',
- 'No subnets in the capacity': 'AZ/subnet mismatch → references/cluster-diagnostics-detail.md § B',
- 'Lifecycle scripts did not run': 'Lifecycle script failure → references/cluster-diagnostics-detail.md § C',
- 'Lifecycle scripts execution timed out': 'Lifecycle script timeout → references/cluster-diagnostics-detail.md § C',
- 'network misconfiguration': 'Network misconfiguration → references/cluster-diagnostics-detail.md § A + § B',
- 'hardware failure': 'Hardware failure → delegate to node-debugger',
- 'Failed to provision': 'Provisioning failure → references/cluster-diagnostics-detail.md § B or § E',
- 'replace': 'Node replacement activity → references/cluster-diagnostics-detail.md § G',
- 'reboot': 'Node reboot activity → references/cluster-diagnostics-detail.md § G',
- }
- for e in events[:20]:
- ts = str(e.get('EventTime', '?'))[:19]
- etype = e.get('EventType', '?')
- msg = e.get('Message', '?')[:120]
- print(f' [{ts}] {etype}: {msg}')
- msg_lower = (e.get('Message','') or '').lower()
- for pattern, hint in ISSUE_PATTERNS.items():
- if pattern.lower() in msg_lower:
- print(f' [ISSUE] {hint}')
- break
- " 2>/dev/null
- fi
- header "4. VPC & Security Group Configuration"
- SUBNET_IDS=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- print(' '.join(d.get('VpcConfig',{}).get('Subnets',[])))
- " 2>/dev/null || echo "")
- SG_IDS=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- print(' '.join(d.get('VpcConfig',{}).get('SecurityGroupIds',[])))
- " 2>/dev/null || echo "")
- if [[ -z "$SUBNET_IDS" ]]; then
- warn "VpcConfig" "No VpcConfig found in cluster"
- else
- info "Subnets: $SUBNET_IDS"
- info "Security Groups: $SG_IDS"
- IFS=' ' read -ra _subnet_ids_arr <<< "$SUBNET_IDS"
- SUBNET_JSON=$(aws ec2 describe-subnets \
- --subnet-ids "${_subnet_ids_arr[@]}" \
- --region "$REGION" \
- --cli-read-timeout 30 \
- --output json 2>&1) || {
- SUB_ERR="$SUBNET_JSON"
- if echo "$SUB_ERR" | grep -qi "AccessDenied\|UnauthorizedOperation\|not authorized"; then
- warn "describe-subnets" "IAM permission denied — add ec2:DescribeSubnets to your role"
- fi
- SUBNET_JSON='{"Subnets":[]}'
- }
- _SUBNET_CHECK=$(echo "$SUBNET_JSON" | python3 -c "
- import sys, json
- subnets = json.load(sys.stdin).get('Subnets', [])
- vpcs = set()
- for s in subnets:
- sid = s.get('SubnetId', '?')
- vpc = s.get('VpcId', '?')
- az = s.get('AvailabilityZone', '?')
- free = s.get('AvailableIpAddressCount', 0)
- flag = ' LOW IPs' if free < 10 else ''
- print(f' {sid}: VPC={vpc} AZ={az} FreeIPs={free}{flag}')
- vpcs.add(vpc)
- if len(vpcs) > 1:
- print('MULTI_VPC=true')
- print('VPC_LIST=' + ','.join(vpcs))
- else:
- print('MULTI_VPC=false')
- v = vpcs.pop() if vpcs else '?'
- print('VPC_ID=' + v)
- " 2>/dev/null || echo "")
- while IFS= read -r line; do
- if [[ "$line" == "MULTI_VPC=true" ]]; then
- fail "Subnet VPC alignment" "Subnets are in DIFFERENT VPCs — all must be in the same VPC"
- add_issue "Subnets in different VPCs → references/cluster-diagnostics-detail.md § B (Capacity & AZ)" "P0"
- fi
- if [[ "$line" != MULTI_VPC=* && "$line" != VPC_ID=* && "$line" != VPC_LIST=* ]]; then
- echo "$line"
- fi
- done <<< "$_SUBNET_CHECK"
- # SG self-referencing rules are an EFA requirement.
- # shellcheck disable=SC2086 # intentional word-split on space-separated SG IDs
- for SG in $SG_IDS; do
- SG_RESULT=$(aws ec2 describe-security-groups \
- --group-ids "$SG" \
- --region "$REGION" \
- --cli-read-timeout 30 \
- --output json 2>&1)
- if echo "$SG_RESULT" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
- warn "describe-security-groups" "IAM permission denied for $SG — SG check skipped"
- continue
- fi
- SG_JSON="${SG_RESULT}"
- [[ -z "$SG_JSON" || "$SG_JSON" == *"error"* ]] && SG_JSON='{"SecurityGroups":[]}'
- _SG_CHECK=$(echo "$SG_JSON" | check_sg_self_ref "$SG")
- while IFS= read -r line; do
- [[ -z "$line" ]] && continue
- level=$(echo "$line" | cut -d: -f1)
- msg=$(echo "$line" | cut -d: -f2-)
- case "$level" in
- PASS) pass "$msg" ;;
- FAIL) fail "$msg"
- if echo "$msg" | grep -q "Inbound self-ref MISSING"; then
- add_issue "Security group $SG inbound self-ref MISSING → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
- elif echo "$msg" | grep -q "Outbound self-ref MISSING"; then
- add_issue "Security group $SG outbound self-ref MISSING → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
- elif echo "$msg" | grep -q "Outbound 0.0.0.0/0 missing"; then
- add_issue "Security group $SG outbound 0.0.0.0/0 MISSING → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
- else
- add_issue "Security group $SG rule missing → references/cluster-diagnostics-detail.md § A (EFA Health Checks)" "P0"
- fi
- ;;
- WARN) warn "$msg" ;;
- SKIP) info "$msg" ;;
- esac
- done <<< "$_SG_CHECK"
- done
- fi
- header "4b. Instance Quotas"
- INSTANCE_TYPES=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- types=set(g.get('InstanceType','') for g in d.get('InstanceGroups',[]))
- print(' '.join(t for t in types if t))
- " 2>/dev/null || echo "")
- if [[ -n "$INSTANCE_TYPES" ]]; then
- # One paginated list-service-quotas call, cached across all instance types.
- # The API is account/region rate-limited and throttles if called per-type.
- QUOTA_ALL=""
- QUOTA_ERR=""
- _next=""
- for _pg in 1 2 3 4 5; do
- if [[ -n "$_next" ]]; then
- _raw=$(aws service-quotas list-service-quotas \
- --service-code sagemaker --region "$REGION" \
- --cli-read-timeout 15 --starting-token "$_next" \
- --output json 2>&1 || true)
- else
- _raw=$(aws service-quotas list-service-quotas \
- --service-code sagemaker --region "$REGION" \
- --cli-read-timeout 15 \
- --output json 2>&1 || true)
- fi
- # Order matters: test for specific errors first, then fall through to
- # generic "not JSON" check, so throttled responses don't get misclassified.
- if echo "$_raw" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
- QUOTA_ERR="denied"; break
- elif echo "$_raw" | grep -qiE "TooManyRequestsException|ThrottlingException|RequestLimitExceeded|exceeded the rate"; then
- QUOTA_ERR="throttled"; break
- elif ! echo "$_raw" | head -c 1 | grep -q '{'; then
- QUOTA_ERR="api-error"; break
- fi
- _pg_quotas=$(echo "$_raw" | python3 -c "import sys,json; d=json.load(sys.stdin); print(json.dumps(d.get('Quotas',[])))" 2>/dev/null || echo "[]")
- if [[ "$_pg_quotas" != "[]" ]]; then
- if [[ -z "$QUOTA_ALL" ]]; then
- QUOTA_ALL="$_pg_quotas"
- else
- QUOTA_ALL=$(python3 -c "import sys,json; a=json.loads(sys.argv[1]); b=json.loads(sys.argv[2]); print(json.dumps(a+b))" "$QUOTA_ALL" "$_pg_quotas")
- fi
- fi
- _next=$(echo "$_raw" | python3 -c "import sys,json; print(json.load(sys.stdin).get('NextToken','') or '')" 2>/dev/null || echo "")
- [[ -z "$_next" ]] && break
- done
- case "$QUOTA_ERR" in
- denied) warn "list-service-quotas" "IAM permission denied — quota check skipped" ;;
- throttled) warn "list-service-quotas" "Throttled — quota check skipped (retry later)" ;;
- api-error) warn "list-service-quotas" "API call failed — quota check skipped" ;;
- esac
- if [[ -n "$QUOTA_ALL" && -z "$QUOTA_ERR" ]]; then
- for ITYPE in $INSTANCE_TYPES; do
- QUOTA_VAL=$(python3 -c "
- import sys, json
- quotas = json.loads(sys.argv[1])
- itype = sys.argv[2]
- # Match quotas that reference the instance type AND HyperPod
- matches = [q for q in quotas if itype in q.get('QuotaName','') and 'HyperPod' in q.get('QuotaName','')]
- if matches:
- q = matches[0]
- print(f\"{q.get('QuotaName','?')}: {int(q.get('Value',0))}\")
- else:
- print('NOT_FOUND')
- " "$QUOTA_ALL" "$ITYPE" 2>/dev/null || echo "NOT_FOUND")
- if [[ "$QUOTA_VAL" == "NOT_FOUND" ]]; then
- info "Quota for $ITYPE: not found in the SageMaker quota list (check Service Quotas console)"
- else
- info "Quota: $QUOTA_VAL"
- fi
- done
- fi
- else
- info "No instance types found in cluster config"
- fi
- if [[ "$ORCHESTRATOR" == "EKS" && -n "$EKS_NAME" ]]; then
- header "5. EKS Configuration"
- EKS_AUTH=$(aws eks describe-cluster \
- --name "$EKS_NAME" \
- --region "$REGION" \
- --query 'cluster.accessConfig.authenticationMode' \
- --output text 2>/dev/null || echo "unknown")
- if [[ "$EKS_AUTH" == "CONFIG_MAP" ]]; then
- warn "EKS auth mode" "CONFIG_MAP-only — access entries require API or API_AND_CONFIG_MAP"
- add_issue "EKS auth mode is CONFIG_MAP — access entries unavailable until switched (see EKS access-entries docs) → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P2"
- elif [[ "$EKS_AUTH" == "API" || "$EKS_AUTH" == "API_AND_CONFIG_MAP" ]]; then
- pass "EKS auth mode" "$EKS_AUTH"
- else
- warn "EKS auth mode" "Could not determine ($EKS_AUTH)"
- fi
- # Check access entries for current identity. AWS CLI paginates JSON output by
- # token, so paginate explicitly to handle accounts with many principals.
- info "Current IAM identity: $CALLER_ARN"
- fetch_all_access_entries() {
- local merged='[]' token='' page_json combined i=0
- while (( i < 20 )); do
- if [[ -n "$token" ]]; then
- page_json=$(aws eks list-access-entries --cluster-name "$EKS_NAME" --region "$REGION" \
- --next-token "$token" --output json 2>/dev/null) || break
- else
- page_json=$(aws eks list-access-entries --cluster-name "$EKS_NAME" --region "$REGION" \
- --output json 2>/dev/null) || break
- fi
- combined=$(python3 -c "
- import sys, json
- prev = json.loads(sys.argv[1])
- page = json.loads(sys.argv[2])
- prev.extend(page.get('accessEntries', []))
- print(json.dumps(prev))
- print(page.get('nextToken',''))
- " "$merged" "$page_json" 2>/dev/null) || break
- merged=$(printf '%s\n' "$combined" | sed -n '1p')
- token=$(printf '%s\n' "$combined" | sed -n '2p')
- i=$((i+1))
- [[ -z "$token" ]] && break
- done
- echo "$merged"
- }
- ACCESS_ENTRIES=$(fetch_all_access_entries)
- [[ -z "$ACCESS_ENTRIES" ]] && ACCESS_ENTRIES='[]'
- ENTRY_COUNT=$(echo "$ACCESS_ENTRIES" | python3 -c "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0)
- info "Access entries: $ENTRY_COUNT configured"
- # Strip session name for role-based ARNs
- CALLER_BASE=$(echo "$CALLER_ARN" | python3 -c "
- import sys
- arn = sys.stdin.read().strip()
- # Convert assumed-role ARN to role ARN for matching
- # arn:aws:sts::ACCOUNT:assumed-role/ROLE/SESSION -> arn:aws:iam::ACCOUNT:role/ROLE
- if ':assumed-role/' in arn:
- parts = arn.split(':')
- role_path = parts[-1].replace('assumed-role/', 'role/')
- role_path = '/'.join(role_path.split('/')[:2]) # remove session name
- parts[-1] = role_path
- parts[2] = 'iam'
- parts[3] = '' # IAM ARNs have no region
- print(':'.join(parts))
- else:
- print(arn)
- " 2>/dev/null || echo "$CALLER_ARN")
- HAS_ACCESS=$(echo "$ACCESS_ENTRIES" | CALLER_BASE_ENV="$CALLER_BASE" python3 -c "
- import sys, json, os
- entries = json.load(sys.stdin)
- caller = os.environ['CALLER_BASE_ENV']
- found = any(caller in str(e) for e in entries)
- print('true' if found else 'false')
- " 2>/dev/null || echo "false")
- if [[ "$HAS_ACCESS" == "true" ]]; then
- pass "EKS access entry" "current identity has an access entry"
- else
- warn "EKS access entry" "current identity ($CALLER_BASE) may not have an access entry — kubectl may fail"
- add_issue "Current IAM identity may lack EKS access → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
- fi
- if command -v kubectl &>/dev/null; then
- KUBECTL_TEST=$(kubectl cluster-info 2>&1 || true)
- if echo "$KUBECTL_TEST" | grep -q "Kubernetes control plane\|running at"; then
- pass "kubectl connectivity" "can reach EKS API server"
- if kubectl get namespace aws-hyperpod &>/dev/null 2>&1; then
- pass "aws-hyperpod namespace" "exists"
- else
- warn "aws-hyperpod namespace" "missing → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)"
- fi
- # Node count. Note: `wc -l` never fails; avoid `|| echo 0` which would produce "0\n0".
- K8S_NODE_COUNT=$(kubectl get nodes --no-headers 2>/dev/null | wc -l | tr -d ' ')
- K8S_NODE_COUNT=${K8S_NODE_COUNT:-0}
- info "Kubernetes nodes visible: $K8S_NODE_COUNT"
- if [[ "$K8S_NODE_COUNT" -eq 0 && "$TOTAL_NODES" -gt 0 ]]; then
- warn "K8s nodes" "0 K8s nodes but $TOTAL_NODES HyperPod nodes — nodes may not have registered with EKS"
- add_issue "Nodes not visible in kubectl → references/cluster-diagnostics-detail.md § E (Cluster Provisioning)" "P1"
- fi
- HEALTH_LABELS=$(kubectl get nodes -o custom-columns='NODE:.metadata.name,HEALTH:.metadata.labels.sagemaker\.amazonaws\.com/node-health-status' --no-headers 2>/dev/null || true)
- if [[ -n "$HEALTH_LABELS" ]]; then
- UNHEALTHY_K8S=$(echo "$HEALTH_LABELS" | grep -v "<none>" | grep -viE "Schedulable$" || true)
- if [[ -n "$UNHEALTHY_K8S" ]]; then
- warn "EKS node health labels" "non-schedulable nodes detected:"
- echo "$UNHEALTHY_K8S" | while IFS= read -r line; do info " $line"; done
- add_issue "EKS nodes with health issues → delegate to hyperpod-node-debugger skill; references/cluster-diagnostics-detail.md § G (Node Replacement)" "P1"
- else
- pass "EKS node health labels" "all nodes schedulable"
- fi
- fi
- # Dangling node detection — nodes visible in EKS but not in HyperPod list
- # (or vice versa). Happens after failed scale-up, rollback, or orphaned
- # kubelet registrations.
- if [[ "$K8S_NODE_COUNT" -gt 0 && "$TOTAL_NODES" -gt 0 ]]; then
- HP_INSTANCES=$(echo "$NODE_LIST" | python3 -c "
- import sys,json
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- for n in nodes:
- iid=n.get('InstanceId','')
- if iid: print(iid)
- " 2>/dev/null | sort -u)
- EKS_INSTANCES=$(kubectl get nodes -l sagemaker.amazonaws.com/compute-type=hyperpod \
- -o jsonpath='{range .items[*]}{.spec.providerID}{"\n"}{end}' 2>/dev/null \
- | awk -F/ '{print $NF}' | grep -E '^i-' | sort -u || true)
- if [[ -n "$HP_INSTANCES" && -n "$EKS_INSTANCES" ]]; then
- DANGLING=$(comm -13 <(echo "$HP_INSTANCES") <(echo "$EKS_INSTANCES"))
- ORPHANED=$(comm -23 <(echo "$HP_INSTANCES") <(echo "$EKS_INSTANCES"))
- if [[ -n "$DANGLING" ]]; then
- warn "Dangling nodes" "visible in EKS but not in HyperPod ($(echo "$DANGLING" | wc -l))"
- echo "$DANGLING" | head -5 | while IFS= read -r iid; do info " EKS-only: $iid"; done
- add_issue "Dangling EKS nodes (present in kubectl, absent from list-cluster-nodes) → references/cluster-diagnostics-detail.md § K (Dangling Nodes & Cleanup)" "P1"
- fi
- if [[ -n "$ORPHANED" ]]; then
- warn "Orphaned HyperPod nodes" "visible in HyperPod but not in EKS ($(echo "$ORPHANED" | wc -l))"
- echo "$ORPHANED" | head -5 | while IFS= read -r iid; do info " HyperPod-only: $iid"; done
- add_issue "HyperPod nodes not registered in EKS → references/cluster-diagnostics-detail.md § E (Cluster Provisioning); delegate to hyperpod-node-debugger" "P1"
- fi
- [[ -z "$DANGLING" && -z "$ORPHANED" ]] && pass "Node reconciliation" "EKS and HyperPod views match"
- fi
- fi
- # EKS add-on health — VPC CNI, CoreDNS, kube-proxy failures break pod networking.
- # Add-on count is small in practice (<10) so a single page of 100 is always sufficient.
- if [[ -n "$EKS_NAME" ]]; then
- ADDON_JSON=$(aws eks list-addons --cluster-name "$EKS_NAME" --region "$REGION" \
- --max-results 100 --output json 2>/dev/null || echo '{"addons":[]}')
- ADDON_NAMES=$(echo "$ADDON_JSON" | python3 -c "
- import sys,json
- print('\n'.join(json.load(sys.stdin).get('addons',[])))
- " 2>/dev/null)
- DEGRADED_ADDONS=""
- while IFS= read -r addon; do
- [[ -z "$addon" ]] && continue
- A_STATUS=$(aws eks describe-addon --cluster-name "$EKS_NAME" --addon-name "$addon" \
- --region "$REGION" --query 'addon.status' --output text 2>/dev/null || echo "UNKNOWN")
- if [[ "$A_STATUS" != "ACTIVE" && "$A_STATUS" != "UPDATING" ]]; then
- DEGRADED_ADDONS+="$addon($A_STATUS) "
- fi
- done <<< "$ADDON_NAMES"
- if [[ -n "$DEGRADED_ADDONS" ]]; then
- warn "EKS add-ons" "not ACTIVE: $DEGRADED_ADDONS"
- add_issue "EKS add-on(s) degraded: $DEGRADED_ADDONS → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
- else
- [[ -n "$ADDON_NAMES" ]] && pass "EKS add-ons" "$(echo "$ADDON_NAMES" | wc -l) add-on(s) ACTIVE"
- fi
- fi
- # aws-auth ConfigMap legacy check — deprecated but still load-bearing if cluster auth mode
- # is API_AND_CONFIG_MAP or CONFIG_MAP. Misconfigured entries here can shadow access entries.
- if [[ -n "$EKS_NAME" ]]; then
- AUTH_MODE=$(aws eks describe-cluster --name "$EKS_NAME" --region "$REGION" \
- --query 'cluster.accessConfig.authenticationMode' --output text 2>/dev/null || echo "")
- if [[ "$AUTH_MODE" == "CONFIG_MAP" || "$AUTH_MODE" == "API_AND_CONFIG_MAP" ]]; then
- if kubectl -n kube-system get configmap aws-auth >/dev/null 2>&1; then
- AUTH_ENTRIES=$(kubectl -n kube-system get configmap aws-auth -o jsonpath='{.data.mapRoles}' 2>/dev/null | grep -c "^" || true)
- AUTH_ENTRIES=${AUTH_ENTRIES:-0}
- info "aws-auth ConfigMap: $AUTH_ENTRIES mapRoles entries (auth mode: $AUTH_MODE)"
- if [[ "$AUTH_MODE" == "API_AND_CONFIG_MAP" ]]; then
- warn "aws-auth ConfigMap" "both ConfigMap and access entries in use — ConfigMap entries can shadow access entries; recommend migrating to API-only mode"
- fi
- fi
- fi
- fi
- else
- warn "kubectl connectivity" "cannot reach EKS API — check kubeconfig and access entries"
- add_issue "kubectl cannot reach EKS → references/cluster-diagnostics-detail.md § D (EKS Access / kubectl)" "P1"
- fi
- else
- info "kubectl not installed — skipping Kubernetes checks"
- fi
- else
- header "5. Slurm Checks"
- info "Orchestrator: Slurm"
- # Warn/issue emitted in section 1; this branch is the PASS-only confirmation.
- if [[ "$NODE_RECOVERY" == *"Automatic"* ]] && [[ "$NODE_RECOVERY" != *"None"* ]]; then
- pass "NodeRecovery" "enabled on all instance groups"
- fi
- if command -v session-manager-plugin &>/dev/null && [[ -n "$CLUSTER_ID" ]]; then
- header "5b. Slurm Controller Health (via SSM)"
- HEAD_NODE_ID=$(echo "$NODE_LIST" | python3 -c "
- import sys,json
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- for n in nodes:
- g=n.get('InstanceGroupName','').lower()
- if any(x in g for x in ['controller','head','master','login']):
- print(n.get('InstanceId',''))
- break
- else:
- if nodes:
- print(nodes[0].get('InstanceId',''))
- " 2>/dev/null || echo "")
- if [[ -n "$HEAD_NODE_ID" ]]; then
- HEAD_GROUP=$(echo "$NODE_LIST" | HEAD_NODE_ID_ENV="$HEAD_NODE_ID" python3 -c "
- import sys,json,os
- target_id = os.environ['HEAD_NODE_ID_ENV']
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- for n in nodes:
- if n.get('InstanceId','') == target_id:
- print(n.get('InstanceGroupName',''))
- break
- " 2>/dev/null || echo "")
- if [[ -z "$HEAD_GROUP" ]]; then
- warn "Controller node" "could not resolve instance-group name — SSM check skipped"
- HEAD_NODE_ID=""
- fi
- fi
- if [[ -n "$HEAD_NODE_ID" ]]; then
- SSM_TARGET="sagemaker-cluster:${CLUSTER_ID}_${HEAD_GROUP}-${HEAD_NODE_ID}"
- info "Controller node: $HEAD_NODE_ID ($HEAD_GROUP)"
- info "SSM target: $SSM_TARGET"
- _slurm_nonce=$(date +%s%N 2>/dev/null || echo "$RANDOM")
- # Validate nonce is numeric to prevent injection in remote command
- if [[ ! "$_slurm_nonce" =~ ^[0-9]+$ ]]; then
- _slurm_nonce="$$"
- fi
- SLURM_SH=$(cat <<EOF
- scontrol show config >/dev/null 2>&1
- if [ \$? -eq 0 ]; then echo SLURM_OK_${_slurm_nonce}; else echo SLURM_DOWN_${_slurm_nonce}; fi
- echo NODES_START_${_slurm_nonce}
- sinfo -o '%N %T %30E' --noheader 2>/dev/null | head -20
- echo NODES_END_${_slurm_nonce}
- echo JOBS_START_${_slurm_nonce}
- squeue -o '%i %j %T %R' --noheader 2>/dev/null | grep -iE 'COMPLETING|CONFIGURING|PENDING' | head -10 || true
- echo JOBS_END_${_slurm_nonce}
- echo MUNGE_${_slurm_nonce}
- systemctl is-active munge 2>/dev/null || echo munge_inactive
- echo END_${_slurm_nonce}
- EOF
- )
- STDOUT=$(ssm_run_on_node "$HEAD_NODE_ID" "$HEAD_GROUP" "$SLURM_SH" || echo "")
- if [[ -n "$STDOUT" ]]; then
- if echo "$STDOUT" | grep -q "SLURM_OK_${_slurm_nonce}"; then
- pass "slurmctld" "responsive"
- elif echo "$STDOUT" | grep -q "SLURM_DOWN_${_slurm_nonce}"; then
- fail "slurmctld" "not responding — all Slurm operations blocked"
- add_issue "slurmctld down on controller → references/cluster-operations.md § 8 Slurm — controller operations" "P0"
- fi
- SLURM_DOWN_NODES=$(echo "$STDOUT" | sed -n "/^NODES_START_${_slurm_nonce}\$/,/^NODES_END_${_slurm_nonce}\$/p" | grep -v "^NODES_" | grep -iE "down|drain|fail" || true)
- if [[ -n "$SLURM_DOWN_NODES" ]]; then
- warn "Slurm nodes with issues:"
- echo "$SLURM_DOWN_NODES" | while IFS= read -r line; do info " $line"; done
- S_DOWN_COUNT=$(echo "$SLURM_DOWN_NODES" | grep -c . ; :)
- S_DOWN_COUNT=${S_DOWN_COUNT:-0}
- add_issue "$S_DOWN_COUNT Slurm node(s) down/drained → references/cluster-diagnostics-detail.md § G (Node Replacement); delegate to hyperpod-node-debugger" "P1"
- else
- pass "Slurm nodes" "all idle/alloc/mixed"
- fi
- STUCK_JOBS=$(echo "$STDOUT" | sed -n "/^JOBS_START_${_slurm_nonce}\$/,/^JOBS_END_${_slurm_nonce}\$/p" | grep -v "^JOBS_" || true)
- if [[ -n "$STUCK_JOBS" ]]; then
- warn "Stuck Slurm jobs detected:"
- echo "$STUCK_JOBS" | while IFS= read -r line; do info " $line"; done
- add_issue "Stuck Slurm jobs → references/cluster-operations.md § 8 Slurm — controller operations" "P1"
- fi
- if echo "$STDOUT" | sed -n "/^MUNGE_${_slurm_nonce}\$/,/^END_${_slurm_nonce}\$/p" | grep -q "munge_inactive"; then
- fail "munge" "authentication service not running — Slurm auth will fail"
- add_issue "munge service inactive on controller → references/cluster-operations.md § 8 Slurm — controller operations" "P0"
- fi
- else
- info "Could not get output from SSM on controller — check ssm:StartSession permission, session-manager-plugin, or node reachability"
- fi
- else
- info "Could not identify controller node from node list"
- fi
- else
- info "SSM plugin not available — Slurm checks require SSM access to controller"
- info "Install SSM plugin to enable Slurm health checks"
- fi
- fi
- header "6. SSM Readiness"
- if command -v session-manager-plugin &>/dev/null; then
- if SSM_VERSION=$(session-manager-plugin --version 2>/dev/null); then
- pass "SSM plugin installed" "version: $SSM_VERSION"
- else
- warn "SSM plugin" "installed but --version failed — plugin may be corrupt"
- add_issue "SSM plugin installed but broken → references/cluster-diagnostics-detail.md § F (SSM Connectivity)" "P1"
- fi
- else
- warn "SSM plugin" "not installed — required for node access (install session-manager-plugin)"
- add_issue "SSM plugin not installed → references/cluster-diagnostics-detail.md § F (SSM Connectivity)" "P2"
- fi
- if [[ -n "$CLUSTER_ID" && "$TOTAL_NODES" -gt 0 ]]; then
- FIRST_NODE=$(echo "$NODE_LIST" | python3 -c "
- import sys, json
- nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
- if nodes:
- n = nodes[0]
- nid = n.get('InstanceId', '?')
- group = n.get('InstanceGroupName', '?')
- print(f'{group}-{nid}')
- " 2>/dev/null || echo "")
- if [[ -n "$FIRST_NODE" ]]; then
- info "SSM target format: sagemaker-cluster:${CLUSTER_ID}_${FIRST_NODE}"
- info "To connect: aws ssm start-session --target sagemaker-cluster:${CLUSTER_ID}_${FIRST_NODE} --region $REGION"
- fi
- fi
- if [[ -n "$SUBNET_IDS" ]]; then
- header "6b. VPC Endpoints"
- FIRST_SUBNET=$(echo "$SUBNET_IDS" | awk '{print $1}')
- VPC_FOR_ENDPOINTS=$(aws ec2 describe-subnets \
- --subnet-ids "$FIRST_SUBNET" \
- --region "$REGION" \
- --cli-read-timeout 15 \
- --query 'Subnets[0].VpcId' \
- --output text 2>/dev/null || echo "")
- if [[ -n "$VPC_FOR_ENDPOINTS" && "$VPC_FOR_ENDPOINTS" != "None" ]]; then
- EP_RESULT=$(aws ec2 describe-vpc-endpoints \
- --filters "Name=vpc-id,Values=$VPC_FOR_ENDPOINTS" \
- --region "$REGION" \
- --cli-read-timeout 15 \
- --query "VpcEndpoints[?State==\`available\`].ServiceName" \
- --output text 2>&1)
- if echo "$EP_RESULT" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
- warn "describe-vpc-endpoints" "IAM permission denied — VPC endpoint check skipped"
- EP_RESULT=""
- fi
- ENDPOINTS="${EP_RESULT}"
- # s3 → Lifecycle scripts (S3 bucket download path)
- # ssm/ssmmessages/ec2messages → SSM connectivity (§ F)
- for SVC in s3 ssm ssmmessages ec2messages; do
- if echo "$ENDPOINTS" | grep -qE "(^|[.])${SVC}($|[[:space:]])"; then
- pass "VPC endpoint: $SVC"
- else
- warn "VPC endpoint: $SVC" "not found — required only if the cluster subnet has no NAT/IGW path out"
- case "$SVC" in
- s3) add_issue "VPC endpoint not found for s3 → references/cluster-diagnostics-detail.md § C (Lifecycle Scripts)" "P2" ;;
- ssm|ssmmessages|ec2messages)
- add_issue "VPC endpoint not found for $SVC → references/cluster-diagnostics-detail.md § F (SSM Connectivity)" "P2" ;;
- esac
- fi
- done
- else
- info "Could not determine VPC ID for endpoint check"
- fi
- fi
- header "7. CloudWatch Logs"
- if [[ -n "$CLUSTER_ID" ]]; then
- # CW log groups follow /aws/sagemaker/Clusters/<CLUSTER_NAME>/<CLUSTER_ID>,
- # where <CLUSTER_NAME> is the human-readable name (not the ARN short-id).
- CLUSTER_NAME_FOR_LOGS=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys, json
- try:
- d = json.load(sys.stdin)
- n = d.get('ClusterName', '')
- print(n if n else '')
- except Exception:
- print('')
- " 2>/dev/null)
- # Fall back to the value the caller supplied, unless it looks like an ARN.
- if [[ -z "$CLUSTER_NAME_FOR_LOGS" ]]; then
- if [[ "$CLUSTER" == arn:aws:* ]]; then
- CLUSTER_NAME_FOR_LOGS="$CLUSTER_ID" # best-effort; will probe the prefix below
- else
- CLUSTER_NAME_FOR_LOGS="$CLUSTER"
- fi
- fi
- LOG_GROUP="/aws/sagemaker/Clusters/${CLUSTER_NAME_FOR_LOGS}/${CLUSTER_ID}"
- LOG_RESULT=$(aws logs describe-log-groups \
- --log-group-name-prefix "$LOG_GROUP" \
- --region "$REGION" \
- --query 'logGroups[0].logGroupName' \
- --output text 2>&1)
- if echo "$LOG_RESULT" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
- warn "describe-log-groups" "IAM permission denied — CloudWatch log check skipped"
- LOG_RESULT="None"
- fi
- LOG_EXISTS="${LOG_RESULT:-None}"
- if [[ "$LOG_EXISTS" != "None" && -n "$LOG_EXISTS" ]]; then
- pass "CloudWatch log group" "$LOG_GROUP"
- # Use the server-side prefix filter; clusters with hundreds of nodes have
- # hundreds of streams and the default first-page result truncates.
- count_log_streams_by_prefix() {
- local prefix="$1"
- local merged='[]' token='' page_json combined i=0
- while (( i < 20 )); do
- if [[ -n "$token" ]]; then
- page_json=$(aws logs describe-log-streams \
- --log-group-name "$LOG_GROUP" --region "$REGION" \
- --log-stream-name-prefix "$prefix" --limit 50 --next-token "$token" \
- --output json 2>/dev/null) || break
- else
- page_json=$(aws logs describe-log-streams \
- --log-group-name "$LOG_GROUP" --region "$REGION" \
- --log-stream-name-prefix "$prefix" --limit 50 \
- --output json 2>/dev/null) || break
- fi
- combined=$(python3 -c "
- import sys, json
- prev = json.loads(sys.argv[1])
- page = json.loads(sys.argv[2])
- prev.extend(s.get('logStreamName','') for s in page.get('logStreams', []))
- print(json.dumps(prev))
- print(page.get('nextToken',''))
- " "$merged" "$page_json" 2>/dev/null) || break
- merged=$(printf '%s\n' "$combined" | sed -n '1p')
- token=$(printf '%s\n' "$combined" | sed -n '2p')
- i=$((i+1))
- [[ -z "$token" ]] && break
- done
- echo "$merged" | python3 -c "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0
- }
- LC_COUNT=$(count_log_streams_by_prefix "LifecycleConfig")
- HM_COUNT=$(count_log_streams_by_prefix "SagemakerHealthMonitoringAgent")
- info "Lifecycle log streams: $LC_COUNT"
- info "Health monitoring log streams: $HM_COUNT"
- if [[ "$LC_COUNT" -eq 0 && "$CLUSTER_STATUS" != "Creating" ]]; then
- warn "Lifecycle logs" "no lifecycle log streams found — scripts may not have run"
- fi
- else
- warn "CloudWatch log group" "not found: $LOG_GROUP"
- info "Logs may not be available if cluster creation failed early"
- info "Check IAM execution role has CloudWatch Logs write permissions"
- add_issue "CloudWatch log group not found → references/cluster-diagnostics-detail.md § C (Lifecycle Scripts)" "P2"
- fi
- fi
- echo ""
- echo -e "${BOLD}========================================${NC}"
- echo -e "${BOLD} DIAGNOSTIC SUMMARY ${NC}"
- echo -e "${BOLD}========================================${NC}"
- echo ""
- echo -e " Cluster: ${BOLD}${CLUSTER}${NC} (${ORCHESTRATOR})"
- echo -e " Status: ${CLUSTER_STATUS}"
- echo -e " Results: ${RED}${CRITICAL_FAILURES} critical${NC} | ${YELLOW}${WARNINGS} warnings${NC}"
- echo -e " Mode: READ-ONLY (no changes made; each [FAIL] points to a references section)"
- echo ""
- if [[ ${#ISSUES_FOUND[@]} -gt 0 ]]; then
- echo -e "${BOLD} Issues Found (prioritized):${NC}"
- for priority in P0 P1 P2; do
- has_priority=false
- for issue in "${ISSUES_FOUND[@]}"; do
- if [[ "$issue" == "${priority}|"* ]]; then
- if ! "$has_priority"; then
- case "$priority" in
- P0) echo -e " ${RED}${BOLD}[$priority — Fix Immediately]${NC}" ;;
- P1) echo -e " ${YELLOW}${BOLD}[$priority — Fix Soon]${NC}" ;;
- P2) echo -e " ${BOLD}[$priority — Informational]${NC}" ;;
- esac
- has_priority=true
- fi
- echo -e " → ${issue#*|}"
- fi
- done
- done
- echo ""
- fi
- if [[ $CRITICAL_FAILURES -eq 0 && $WARNINGS -eq 0 ]]; then
- echo -e " ${GREEN}${BOLD}All cluster-level checks passed.${NC}"
- echo " If issues persist, try:"
- echo " - hyperpod-node-debugger skill for per-node issues"
- echo " - hyperpod-nccl skill for NCCL/training issues"
- elif [[ $CRITICAL_FAILURES -eq 0 ]]; then
- echo -e " ${YELLOW}${BOLD}No critical issues, but $WARNINGS warning(s) found.${NC}"
- echo " Review [WARN] items above."
- else
- echo -e " ${RED}${BOLD}$CRITICAL_FAILURES critical issue(s) found.${NC}"
- echo " Fix [FAIL] items above. See SKILL.md for detailed resolution steps."
- fi
- echo ""
- exit "$([[ $CRITICAL_FAILURES -eq 0 ]] && echo 0 || echo 1)"
plugins/sagemaker-ai/skills/hyperpod-nccl/scripts/nccl-diagnose.sh · Bundled script · 2,604 lines · 3 findings marked
- #!/usr/bin/env bash
- # nccl-diagnose.sh — read-only NCCL diagnostic for SageMaker HyperPod.
- # Supports both EKS and Slurm orchestrators (auto-detected).
- # Hardware checks run on cluster nodes via SSM, not locally.
- #
- # This script never modifies cluster state. It collects diagnostic signals and
- # attaches a reference pointer (→ references/<file>.md § <section>) to each
- # finding. The calling skill (hyperpod-nccl) reads this output alongside the
- # referenced sections to guide the user through the remediation.
- #
- # USAGE:
- # bash nccl-diagnose.sh [OPTIONS]
- #
- # OPTIONS:
- # --cluster <name> HyperPod cluster name (required)
- # --region <region> AWS region (required)
- # --orchestrator <eks|slurm> Force orchestrator (default: auto-detect)
- # --namespace <ns> [EKS] K8s namespace to scope (default: all)
- # --job <job-name> [EKS] Specific job to diagnose
- # --node <instance-id> Specific node instance ID for SSM checks
- # --sample-nodes <N> How many nodes to SSM into (default: 3)
- # --verbose Show extra debug output
- # --no-color Disable ANSI colors (also auto-off when not a TTY)
- # --help Show this help
- #
- # ARCHITECTURE:
- # LOCAL checks (run on this machine):
- # - AWS API calls: cluster status, SG rules, cluster events, node list
- # - kubectl calls: K8s node readiness, pod status, logs, NetworkPolicies
- # ON-NODE checks (run via SSM on actual cluster compute nodes):
- # - GPU health (nvidia-smi, XID errors, NVLink)
- # - EFA / libfabric availability
- # - NCCL library presence
- # - Network interfaces and MTU
- # - Memory / /dev/shm / memlock limits
- # - Active training processes
- # - dmesg hardware errors
- # SCALE strategy for 100s of nodes:
- # - AWS API checks cover ALL nodes cheaply via list-cluster-nodes
- # - K8s checks cover ALL nodes cheaply via kubectl
- # - SSM hardware checks sample --sample-nodes (default: 3) compute nodes
- # - CloudWatch log analysis covers ALL nodes at scale (no per-node SSM needed)
- #
- # EXAMPLES:
- # bash nccl-diagnose.sh --cluster my-cluster --region us-east-1
- # bash nccl-diagnose.sh --cluster my-cluster --region us-east-1 \
- # --namespace nccl-test --job my-job --sample-nodes 5
- # bash nccl-diagnose.sh --cluster my-cluster --region us-east-1 \
- # --node i-0123456789abcdef0
- #
- # EXIT CODES:
- # 0 No critical (P0/P1) issues; P2 informational findings are allowed.
- # 1 One or more critical issues, or a fatal prerequisite is missing.
- # 2 Invalid argument.
- set -euo pipefail
- _TEMP_FILES=()
- cleanup() {
- # Guard against empty-array + set -u on older bash (4.2 on AL2).
- [[ ${#_TEMP_FILES[@]} -gt 0 ]] && rm -f "${_TEMP_FILES[@]}" 2>/dev/null || true
- }
- trap cleanup EXIT
- # Auto-disable colors when stdout is not a TTY or TERM=dumb (agent-piped output).
- if [ -t 1 ] && [ "${TERM:-}" != "dumb" ]; then
- RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
- BLUE='\033[0;34m'; BOLD='\033[1m'; RESET='\033[0m'
- else
- RED=''; GREEN=''; YELLOW=''; BLUE=''; BOLD=''; RESET=''
- fi
- info() { echo -e "${BLUE}[INFO]${RESET} $*"; }
- success() { echo -e "${GREEN}[PASS]${RESET} $*"; }
- warn() { echo -e "${YELLOW}[WARN]${RESET} $*"; }
- error() { echo -e "${RED}[FAIL]${RESET} $*"; }
- header() { echo -e "\n${BOLD}${BLUE}═══════════════════════════════════════════════${RESET}"
- echo -e "${BOLD}${BLUE} $*${RESET}"
- echo -e "${BOLD}${BLUE}═══════════════════════════════════════════════${RESET}"; }
- section() { echo -e "\n${BOLD}-- $* --${RESET}"; }
- debug() { $VERBOSE && echo -e "[DEBUG] $*" >&2 || true; }
- CLUSTER_NAME=""
- REGION="${AWS_DEFAULT_REGION:-}"
- ORCHESTRATOR=""
- NAMESPACE=""
- JOB_NAME=""
- NODE_ID=""
- SAMPLE_NODES=3
- VERBOSE=false
- ISSUES_FOUND=0
- ISSUE_DETAILS=()
- add_issue_detail() {
- local priority="${2:-P1}"
- ISSUE_DETAILS+=("${priority}|$1")
- }
- K8S_CONNECTED=false
- SSM_CLUSTER_ID=""
- SSM_NODES=()
- usage() {
- # --help exits 0; invalid invocation exits 2 via usage 2.
- grep "^# USAGE:" -A 40 "$0" | grep "^#" | sed 's/^# \?//' | head -25
- exit "${1:-0}"
- }
- while [[ $# -gt 0 ]]; do
- case "$1" in
- --cluster) [[ $# -lt 2 ]] && { error "--cluster needs a value"; exit 2; }
- [[ ! "$2" =~ ^(arn:aws[a-z-]*:sagemaker:[a-z0-9-]+:[0-9]{12}:cluster/[a-z0-9]{12}|[a-zA-Z0-9]([-a-zA-Z0-9]{0,62}))$ ]] && { error "--cluster must be a valid HyperPod cluster name or ARN (got '$2')"; exit 2; }
- CLUSTER_NAME="$2"; shift 2 ;;
- --region) [[ $# -lt 2 ]] && { error "--region needs a value"; exit 2; }
- [[ ! "$2" =~ ^[a-z]{2}-[a-z]+-[0-9]+$ ]] && { error "--region must be a valid AWS region (got '$2')"; exit 2; }
- REGION="$2"; shift 2 ;;
- --orchestrator) [[ $# -lt 2 ]] && { error "--orchestrator needs a value"; exit 2; }
- [[ "$2" != "eks" && "$2" != "slurm" ]] && { error "--orchestrator must be 'eks' or 'slurm' (got '$2')"; exit 2; }
- ORCHESTRATOR="$2"; shift 2 ;;
- --namespace) [[ $# -lt 2 ]] && { error "--namespace needs a value"; exit 2; }
- [[ ! "$2" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] && { error "--namespace must be a valid K8s namespace (got '$2')"; exit 2; }
- NAMESPACE="$2"; shift 2 ;;
- --job) [[ $# -lt 2 ]] && { error "--job needs a value"; exit 2; }
- [[ ! "$2" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] && { error "--job must be a valid K8s name (got '$2')"; exit 2; }
- JOB_NAME="$2"; shift 2 ;;
- --node) [[ $# -lt 2 ]] && { error "--node needs a value"; exit 2; }
- [[ ! "$2" =~ ^i-[0-9a-f]{8,17}$ ]] && { error "--node must be an EC2 instance ID (got '$2')"; exit 2; }
- NODE_ID="$2"; shift 2 ;;
- --sample-nodes) [[ $# -lt 2 ]] && { error "--sample-nodes needs a value"; exit 2; }; SAMPLE_NODES="$2"; shift 2 ;;
- --verbose) VERBOSE=true; shift ;;
- --no-color) RED=''; GREEN=''; YELLOW=''; BLUE=''; BOLD=''; RESET=''; shift ;;
- --help|-h) usage 0 ;;
- *) echo "Unknown option: $1" >&2; usage 2 ;;
- esac
- done
- [[ -z "$CLUSTER_NAME" ]] && { error "Missing required: --cluster"; exit 1; }
- [[ -z "$REGION" ]] && { error "--region is required (or set AWS_DEFAULT_REGION before running)"; exit 2; }
- if ! [[ "$SAMPLE_NODES" =~ ^[0-9]+$ ]] || [[ "$SAMPLE_NODES" -lt 1 ]]; then
- error "--sample-nodes must be a positive integer (got: '$SAMPLE_NODES')"
- exit 1
- fi
- if [[ "$SAMPLE_NODES" -gt 50 ]]; then
- warn "--sample-nodes=$SAMPLE_NODES is very high (max recommended: 50). Capping at 50."
- SAMPLE_NODES=50
- fi
- # Paginate a sagemaker list-* call. Usage:
- # sagemaker_list_paginated list-cluster-nodes ClusterNodeSummaries [extra args...]
- # Returns {"<SummaryKey>": [...]} on stdout. Caps at 20 000 items; emits a
- # stderr warning if truncated. Returns an empty result on AccessDenied.
- sagemaker_list_paginated() {
- local api="$1" summary_key="$2"
- shift 2
- local merged='[]' token='' page_json combined i=0
- local max_pages=200
- while (( i < max_pages )); do
- local page_args=(--cluster-name "$CLUSTER_NAME" --region "$REGION" \
- --max-results 100 --cli-read-timeout 30 --output json "$@")
- # Validate token format before sending — avoid BadRequest on garbage.
- if [[ -n "$token" ]]; then
- if [[ "$token" =~ ^[a-zA-Z0-9/+]*={0,2}$ ]] && [[ -n "$token" ]]; then
- page_args+=(--next-token "$token")
- else
- break
- fi
- fi
- page_json=$(aws sagemaker "$api" "${page_args[@]}" 2>&1) || break
- if echo "$page_json" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized"; then
- break
- fi
- # Merge via stdin (NUL-delimited) to avoid ARG_MAX truncation at ~500
- # entries. summary_key stays in argv since it's small.
- combined=$(printf '%s\0%s' "$merged" "$page_json" | python3 -c "
- import sys, json
- blob = sys.stdin.buffer.read()
- try:
- a, b = blob.split(b'\0', 1)
- prev = json.loads(a)
- page = json.loads(b)
- except (json.JSONDecodeError, ValueError):
- sys.exit(2)
- prev.extend(page.get(sys.argv[1], []))
- print(json.dumps(prev))
- print(page.get('NextToken', ''))
- " "$summary_key" 2>/dev/null) || break
- merged=$(printf '%s\n' "$combined" | sed -n '1p')
- token=$(printf '%s\n' "$combined" | sed -n '2p')
- i=$((i+1))
- [[ -z "$token" ]] && break
- done
- if (( i == max_pages )) && [[ -n "$token" ]]; then
- echo "WARN: sagemaker_list_paginated($api): truncated at ${max_pages} pages (~$((max_pages*100)) items). Result may be incomplete for very large clusters." >&2
- fi
- printf '%s' "$merged" | python3 -c "
- import sys, json
- try:
- print(json.dumps({sys.argv[1]: json.loads(sys.stdin.read())}))
- except json.JSONDecodeError:
- print('{\"%s\":[]}' % sys.argv[1])
- " "$summary_key" 2>/dev/null || echo "{\"$summary_key\":[]}"
- }
- detect_orchestrator() {
- if [[ -n "$ORCHESTRATOR" ]]; then
- info "Orchestrator forced: $ORCHESTRATOR"; return
- fi
- header "Detecting Orchestrator Type"
- local orch_type
- orch_type=$(aws sagemaker describe-cluster \
- --cluster-name "$CLUSTER_NAME" --region "$REGION" \
- --query 'Orchestrator' --output text 2>/dev/null || echo "")
- if echo "$orch_type" | grep -qi "eks\|kubernetes"; then
- ORCHESTRATOR="eks"
- elif echo "$orch_type" | grep -qi "slurm"; then
- ORCHESTRATOR="slurm"
- elif kubectl cluster-info &>/dev/null 2>&1; then
- ORCHESTRATOR="eks"; info "Auto-detected: EKS (kubectl responds)"
- elif command -v sinfo &>/dev/null && sinfo &>/dev/null 2>&1; then
- ORCHESTRATOR="slurm"; info "Auto-detected: Slurm (sinfo responds)"
- elif command -v squeue &>/dev/null; then
- ORCHESTRATOR="slurm"; info "Auto-detected: Slurm (squeue found)"
- else
- ORCHESTRATOR="eks"
- warn "Could not auto-detect orchestrator — defaulting to 'eks'"
- warn "Override with: --orchestrator slurm"
- fi
- success "Orchestrator: $(printf '%s' "$ORCHESTRATOR" | tr '[:lower:]' '[:upper:]')"
- }
- check_prerequisites() {
- header "Checking Prerequisites"
- local missing=()
- local tool_path
- for tool in aws jq python3 unbuffer; do
- if tool_path=$(command -v "$tool" 2>/dev/null) && [[ -n "$tool_path" ]]; then
- success "$tool: $tool_path"
- else
- error "$tool NOT found — required"
- missing+=("$tool")
- fi
- done
- if [[ "$ORCHESTRATOR" == "eks" ]]; then
- if tool_path=$(command -v kubectl 2>/dev/null) && [[ -n "$tool_path" ]]; then
- success "kubectl: $tool_path"
- else
- error "kubectl NOT found — required for EKS"
- missing+=("kubectl")
- fi
- elif [[ "$ORCHESTRATOR" == "slurm" ]]; then
- local slurm_ok=false
- for t in sinfo squeue scontrol; do
- command -v "$t" &>/dev/null && { success "$t found (Slurm CLI OK)"; slurm_ok=true; break; }
- done
- $slurm_ok || warn "Slurm CLI not found locally — will use SSM for Slurm commands"
- fi
- if [[ ${#missing[@]} -gt 0 ]]; then
- error "Install: ${missing[*]}"
- # unbuffer ships in the `expect` package.
- if printf '%s\n' "${missing[@]}" | grep -qx unbuffer; then
- error " unbuffer: 'yum install expect' / 'apt install expect' / 'brew install expect'"
- fi
- exit 1
- fi
- if aws sts get-caller-identity --region "$REGION" &>/dev/null; then
- local id
- id=$(aws sts get-caller-identity --region "$REGION" --query 'Arn' --output text)
- success "AWS credentials: $id"
- else
- error "AWS credentials invalid or expired"; exit 1
- fi
- # Inspect both stdout (node list) and stderr (error message).
- # Empty stdout with non-zero exit is an auth / transport failure, not
- # a healthy cluster with zero nodes.
- if [[ "$ORCHESTRATOR" == "eks" ]]; then
- local kubectl_out kubectl_err tmpfile
- tmpfile=$(mktemp /tmp/kubectl-check-XXXXXX.err)
- _TEMP_FILES+=("$tmpfile")
- kubectl_out=$(kubectl get nodes --no-headers 2>"$tmpfile" || true)
- kubectl_err=$(cat "$tmpfile" 2>/dev/null || echo "")
- rm -f "$tmpfile"
- debug "kubectl stdout: '$kubectl_out'"
- debug "kubectl stderr: '$kubectl_err'"
- if echo "$kubectl_err" | grep -qiE \
- "Unauthorized|forbidden|You must be logged in|certificate|no configuration|Unable to connect|server.*refused"; then
- error "kubectl NOT authenticated to EKS cluster"
- error " $(echo "$kubectl_err" | head -1)"
- warn " K8s checks (2, 2b, 5, 5b, 6, 7, 9) will be SKIPPED"
- K8S_CONNECTED=false
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "kubectl not authenticated to EKS cluster → references/operations.md § 3 SSM target format (HyperPod)" "P1"
- elif echo "$kubectl_err" | grep -qiE \
- "connection refused|no such host|dial tcp|context deadline exceeded|EOF"; then
- error "kubectl cannot reach EKS API server → references/operations.md § 1 Getting cluster names (kubeconfig setup)"
- error " $(echo "$kubectl_err" | head -1)"
- warn " K8s checks (2, 2b, 5, 5b, 6, 7, 9) will be SKIPPED — check VPN/network connectivity"
- K8S_CONNECTED=false
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "kubectl cannot reach EKS API server → references/operations.md § 1 Getting cluster names (kubeconfig setup)" "P1"
- elif [[ -z "$kubectl_out" && -z "$kubectl_err" ]]; then
- warn "kubectl returned no output — kubeconfig may point to wrong cluster"
- warn " → references/operations.md § 1 Getting cluster names"
- K8S_CONNECTED=true # Allow K8s checks — cluster may simply have no nodes yet
- elif [[ -n "$kubectl_err" && -z "$kubectl_out" ]]; then
- error "kubectl error: $(echo "$kubectl_err" | head -1)"
- warn " K8s checks (2, 2b, 5, 5b, 6, 7, 9) will be SKIPPED"
- K8S_CONNECTED=false
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "kubectl error — K8s checks skipped → references/operations.md § 1 Getting cluster names (kubeconfig setup)" "P1"
- else
- local node_count
- node_count=$(echo "$kubectl_out" | wc -l | tr -d ' ')
- success "kubectl authenticated — $node_count node(s) visible"
- K8S_CONNECTED=true
- fi
- fi
- }
- check_cluster_health() {
- header "Check 1: HyperPod Cluster Health"
- local cluster_json
- cluster_json=$(aws sagemaker describe-cluster \
- --cluster-name "$CLUSTER_NAME" --region "$REGION" \
- --output json 2>&1) || {
- if echo "$cluster_json" | grep -qiE "ResourceNotFound|Cluster with name .* not found|ValidationException"; then
- error "Cluster '$CLUSTER_NAME' not found in region '$REGION'"
- echo "$cluster_json" | head -3
- echo ""
- echo "Available clusters in $REGION:"
- aws sagemaker list-clusters --region "$REGION" \
- --query 'ClusterSummaries[*].{Name:ClusterName,Status:ClusterStatus}' \
- --output table 2>/dev/null || echo " (unable to list clusters — check IAM)"
- exit 1
- fi
- if echo "$cluster_json" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
- warn "Permission denied: sagemaker:DescribeCluster — check IAM policy"
- fi
- cluster_json="{}"
- }
- local cluster_state
- cluster_state=$(echo "$cluster_json" | python3 -c \
- "import sys,json; print(json.load(sys.stdin).get('ClusterStatus','UNKNOWN'))" 2>/dev/null \
- || echo "UNKNOWN")
- case "$cluster_state" in
- InService)
- success "Cluster status: $cluster_state" ;;
- UNKNOWN|None|"")
- warn "Cluster status: could not retrieve"
- warn " Ensure --cluster is the HyperPod cluster name and IAM has sagemaker:DescribeCluster" ;;
- Creating|Updating|RollingBack|SystemUpdating)
- warn "Cluster status: $cluster_state (operation in progress — NCCL checks may be partial)"
- add_issue_detail "Cluster in transient state $cluster_state — rerun after it completes → hyperpod-cluster-debugger skill if it stays stuck" "P2" ;;
- Deleting|DeleteFailed)
- error "Cluster status: $cluster_state (cluster is being torn down)"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "Cluster is ${cluster_state} → hyperpod-cluster-debugger skill" "P0" ;;
- Failed|ClusterMaintenanceRollbackFailed)
- error "Cluster status: $cluster_state (expected: InService)"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "Cluster status ${cluster_state} → hyperpod-cluster-debugger skill" "P0" ;;
- *)
- warn "Cluster status: $cluster_state (unrecognized state)"
- add_issue_detail "Unrecognized cluster state '${cluster_state}' → hyperpod-cluster-debugger skill" "P1" ;;
- esac
- # NodeRecovery — affects whether failed nodes are auto-replaced.
- # Prefer top-level NodeRecovery (the canonical location); InstanceGroups[*].NodeRecovery
- # is null when cluster-level setting is applied, so per-group-only reads always return 'Unknown'.
- local node_recovery
- node_recovery=$(echo "$cluster_json" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- top=d.get('NodeRecovery')
- if top:
- print(top)
- else:
- igs = d.get('InstanceGroups',[])
- modes = sorted({ig.get('NodeRecovery') for ig in igs if ig.get('NodeRecovery')})
- print(','.join(modes) if modes else 'Unknown')
- " 2>/dev/null || echo "Unknown")
- if echo "$node_recovery" | grep -q "Automatic"; then
- success "NodeRecovery: $node_recovery (auto-repair enabled)"
- elif echo "$node_recovery" | grep -qi "^Unknown$"; then
- info "NodeRecovery: could not retrieve (needs sagemaker:DescribeCluster)"
- elif echo "$node_recovery" | grep -qi "^None$"; then
- warn "NodeRecovery: None — failed nodes won't auto-replace → references/operations.md § 6 HyperPod node health labels (EKS)"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "NodeRecovery disabled (set to 'None') — failed nodes won't auto-replace → references/operations.md § 6 HyperPod node health labels (EKS)" "P2"
- else
- warn "NodeRecovery: $node_recovery — failed nodes won't auto-replace → references/operations.md § 6 HyperPod node health labels (EKS)"
- fi
- # All instance groups — count nodes per group, surface any unhealthy count.
- # Paginated because clusters >50 nodes would otherwise be diagnosed on a partial sample.
- local node_summary
- node_summary=$(sagemaker_list_paginated list-cluster-nodes ClusterNodeSummaries)
- local node_output
- node_output=$(echo "$node_summary" | python3 -c "
- import sys,json
- nodes = json.load(sys.stdin).get('ClusterNodeSummaries',[])
- total = len(nodes)
- by_status = {}
- for n in nodes:
- s = n.get('InstanceStatus',{}).get('Status','Unknown')
- by_status[s] = by_status.get(s,0) + 1
- print(f' Total nodes: {total}')
- for s,c in sorted(by_status.items()):
- tag = '[PASS]' if s == 'Running' else '[FAIL]'
- print(f' {tag} {s}: {c}')
- failed = [n for n in nodes if n.get('InstanceStatus',{}).get('Status') not in ('Running','Pending')]
- for n in failed[:10]:
- msg = n.get('InstanceStatus',{}).get('Message','')
- print(f' -> {n[\"InstanceId\"]} ({n[\"InstanceGroupName\"]}): {msg[:120]}')
- print(f'FAILED_COUNT={len(failed)}')
- " 2>/dev/null || echo "FAILED_COUNT=0")
- local fc
- fc=$(echo "$node_output" | grep "^FAILED_COUNT=" | cut -d= -f2 || echo 0)
- # `|| true` on grep — no-match returns 1 and pipefail aborts the function.
- echo "$node_output" | { grep -v "^FAILED_COUNT=" || true; } | while IFS= read -r line; do
- if echo "$line" | grep -q "\[FAIL\]"; then
- error "$line"
- else
- echo "$line"
- fi
- done
- if [[ "${fc:-0}" -gt 0 ]]; then
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "${fc} node(s) in failed/non-Running state → hyperpod-node-debugger skill" "P1"
- fi
- # Pre-flight: if the cluster has no GPU/EFA-capable instance groups, NCCL
- # diagnostics don't apply — exit clearly instead of emitting mixed INFO/SKIP.
- local gpu_groups
- gpu_groups=$(echo "$cluster_json" | python3 -c "
- import sys, json, re
- d = json.load(sys.stdin)
- igs = d.get('InstanceGroups', [])
- gpu_efa_re = re.compile(r'^ml\.(p4d|p4de|p5|p5e|p5en|p6|trn1|trn2|g5\.48xlarge|g6\.48xlarge|g6e\.48xlarge)', re.I)
- matches = [ig.get('InstanceGroupName','?') + ':' + ig.get('InstanceType','?')
- for ig in igs if gpu_efa_re.match(ig.get('InstanceType',''))]
- print('|'.join(matches))
- " 2>/dev/null || echo "")
- if [[ -z "$gpu_groups" ]] && [[ "$(echo "$cluster_json" | python3 -c 'import sys,json; print(len(json.load(sys.stdin).get("InstanceGroups",[])))' 2>/dev/null)" -gt 0 ]]; then
- warn "No GPU/EFA-capable instance groups in this cluster — NCCL is not applicable"
- warn " NCCL is only meaningful on multi-GPU instances with EFA (p4d/p4de/p5/p5e/p5en/p6/trn1/trn2/g5.48xlarge/g6.48xlarge/g6e.48xlarge)"
- warn " The rest of the diagnostic will still run, but most checks will return INFO/SKIP on CPU-only fleets"
- fi
- }
- check_cluster_events() {
- header "Check 3: Cluster Events (infrastructure signals)"
- # HyperPod cluster events report infrastructure-level state only:
- # lifecycle, bootstrap, EFA health-check, capacity, replacement, reboot,
- # software update. They do NOT carry NCCL / GPU / training-level signals —
- # those come from pod logs, CloudWatch, and on-node probes (checks 6–8).
- # ListClusterEvents response shape: array under `Events` with fields
- # EventId / ClusterArn / ClusterName / InstanceGroupName / ResourceType /
- # EventTime / Description (verified live; no Severity field).
- local events_json
- events_json=$(sagemaker_list_paginated list-cluster-events Events)
- local events
- events=$(echo "$events_json" | python3 -c "
- import sys, json
- summaries = json.load(sys.stdin).get('Events', [])
- proj = [{'Time': e.get('EventTime',''),
- 'Grp': e.get('InstanceGroupName','') or e.get('ResourceType',''),
- 'Msg': e.get('Description','') or ''} for e in summaries]
- print(json.dumps(proj))
- " 2>/dev/null || echo "[]")
- local infra_events
- infra_events=$(echo "$events" | python3 -c "
- import sys, json
- data = json.load(sys.stdin)
- # Match real HyperPod event messages that could block or degrade distributed training.
- keywords = [
- 'efa health checks', # 'EFA health checks did not run successfully'
- 'lifecycle script', # 'Lifecycle scripts did not run successfully' / 'execution timed out'
- 'bootstrap failed', # 'Instance bootstrap failed likely because of customer network misconfiguration'
- 'network misconfiguration', # appears in bootstrap-failed events
- 'insufficient capacity', # 'Insufficient capacity' / 'No subnets in the capacity AZ'
- 'failed to provision', # provisioning events
- 'hardware failure', # rare; surfaces via events when SMHP detects
- 'replacement', # node replacement activity
- 'reboot', # node reboot activity
- 'rollback', # AMI upgrade rollback
- ]
- found = [e for e in data if any(k in e.get('Msg','').lower() for k in keywords)]
- for e in found[:20]:
- print(f\"[{e.get('Grp','?')}] {str(e.get('Time','?'))[:19]} | {e.get('Msg','?')[:140]}\")
- print(f'COUNT={len(found)}')
- " 2>/dev/null || echo "COUNT=0")
- local count
- count=$(echo "$infra_events" | grep "^COUNT=" | cut -d= -f2 || echo 0)
- local lines
- lines=$(echo "$infra_events" | grep -v "^COUNT=" || true)
- if [[ -z "$lines" || "${count:-0}" -eq 0 ]]; then
- success "No infrastructure events that would block NCCL"
- if [[ "$ORCHESTRATOR" == "slurm" ]]; then
- info "(Cluster events may not be populated for HyperPod Slurm clusters — rely on pod-/job-log checks instead.)"
- fi
- else
- warn "Infrastructure events potentially affecting NCCL (last 100):"
- echo "$lines" | while IFS= read -r line; do
- if echo "$line" | grep -qiE "error|fail|timeout|rollback"; then
- error " $line"
- else
- warn " $line"
- fi
- done
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "Infrastructure-level events found — review and cross-reference with cluster-debugger if root-cause is cluster-wide → references/debugging-guide.md (match event text to section)" "P1"
- fi
- }
- check_security_groups() {
- header "Check 4: Security Group Rules (EFA / NCCL Communication)"
- local cluster_json
- cluster_json=$(aws sagemaker describe-cluster \
- --cluster-name "$CLUSTER_NAME" --region "$REGION" \
- --output json 2>/dev/null || echo "{}")
- # DescribeCluster.VpcConfig returns SecurityGroupIds + Subnets (not SubnetIds).
- # VpcId is not on VpcConfig; derive from a subnet if needed downstream.
- local sgs subnets
- sgs=$(echo "$cluster_json" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- print(','.join(d.get('VpcConfig',{}).get('SecurityGroupIds',[])))
- " 2>/dev/null || echo "")
- subnets=$(echo "$cluster_json" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- print(','.join(d.get('VpcConfig',{}).get('Subnets',[])))
- " 2>/dev/null || echo "")
- info "SGs: ${sgs:-none} | Subnets: ${subnets:-none}"
- if [[ -z "$sgs" ]]; then
- warn "No security groups in cluster VPC config — cannot verify NCCL rules"
- warn " (DescribeCluster may need sagemaker:DescribeCluster permission)"
- return
- fi
- IFS=',' read -ra sg_list <<< "$sgs"
- for sg in "${sg_list[@]}"; do
- [[ -z "$sg" ]] && continue
- section "SG: $sg"
- local sg_json
- sg_json=$(aws ec2 describe-security-groups \
- --group-ids "$sg" --region "$REGION" \
- --query 'SecurityGroups[0]' --output json 2>&1) || {
- if echo "$sg_json" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
- warn "Permission denied: ec2:DescribeSecurityGroups — check IAM policy"
- fi
- sg_json="{}"
- }
- local self_in self_out all_out
- read -r self_in self_out all_out < <(echo "$sg_json" | python3 -c "
- import sys,json
- sg=json.load(sys.stdin)
- gid=sg.get('GroupId','')
- def has_self(rules):
- return any(any(p.get('GroupId')==gid for p in r.get('UserIdGroupPairs',[])) for r in rules)
- def has_all_out(rules):
- return any(r.get('IpProtocol')=='-1' and any(x.get('CidrIp')=='0.0.0.0/0' for x in r.get('IpRanges',[])) for r in rules)
- print('YES' if has_self(sg.get('IpPermissions',[])) else 'NO',
- 'YES' if has_self(sg.get('IpPermissionsEgress',[])) else 'NO',
- 'YES' if has_all_out(sg.get('IpPermissionsEgress',[])) else 'NO')
- " 2>/dev/null || echo "UNKNOWN UNKNOWN UNKNOWN")
- if [[ "$self_in" == "YES" ]]; then
- success " Inbound self-reference: PRESENT (inter-node communication OK)"
- else
- error " Inbound self-reference: MISSING — NCCL inter-node comm WILL FAIL"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "SG $sg missing inbound self-referencing rule → references/operations.md § 8 NCCL-specific remediations (Security group self-reference)" "P0"
- fi
- if [[ "$self_out" == "YES" ]]; then
- success " Outbound self-reference: PRESENT (EFA traffic OK)"
- else
- error " Outbound self-reference: MISSING — EFA traffic WILL FAIL"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "SG $sg missing outbound self-referencing rule → references/operations.md § 8 NCCL-specific remediations (Security group self-reference)" "P0"
- fi
- if [[ "$all_out" == "YES" ]]; then
- success " Outbound 0.0.0.0/0: PRESENT (API/internet OK)"
- else
- warn " Outbound 0.0.0.0/0: MISSING — may block SageMaker/S3 API calls"
- fi
- done
- }
- check_k8s_nodes() {
- header "Check 2: Kubernetes Node Readiness"
- local raw_nodes total not_ready
- raw_nodes=$(kubectl get nodes --no-headers 2>/dev/null || true)
- total=$(echo "$raw_nodes" | awk 'NF{c++} END{print c+0}')
- not_ready=$(echo "$raw_nodes" | { grep -vE " Ready" || true; } | awk 'NF{c++} END{print c+0}')
- info "Total K8s nodes: $total"
- if [[ "$not_ready" -eq 0 ]]; then
- success "All $total nodes are Ready"
- else
- error "$not_ready/$total nodes NOT Ready"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "$not_ready/$total K8s nodes not Ready → hyperpod-node-debugger skill" "P1"
- echo "$raw_nodes" | { grep -vE " Ready" || true; } | while read -r line; do
- error " Not Ready: $line"
- done
- fi
- section "HyperPod Health Labels (all nodes)"
- # Uses the 4 documented node-health-status values plus deep-health-check-status
- local health_output
- health_output=$(kubectl get nodes -o json 2>/dev/null | python3 -c "
- import sys, json
- data = json.load(sys.stdin)
- issues = 0
- for node in data.get('items', []):
- name = node['metadata']['name']
- labels = node['metadata'].get('labels', {})
- health = labels.get('sagemaker.amazonaws.com/node-health-status', '')
- deep = labels.get('sagemaker.amazonaws.com/deep-health-check-status', '')
- fault_type = labels.get('sagemaker.amazonaws.com/fault-types', '')
- fault_reason = labels.get('sagemaker.amazonaws.com/fault-reasons', '')
- ok = health in ('', 'Schedulable') and deep in ('', 'Passed') and not fault_type
- tag = '[PASS]' if ok else '[FAIL]'
- if not ok:
- issues += 1
- line = f' {tag} {name}: health={health or \"(none)\"}'
- if deep: line += f' deep={deep}'
- if fault_type: line += f' fault={fault_type}'
- print(line)
- if health == 'Unschedulable':
- print(' -> Running deep health checks (~2h), temporarily unavailable')
- elif health == 'UnschedulablePendingReplacement':
- print(' -> Failed health checks — needs replacement (NodeRecovery=Automatic will auto-replace)')
- elif health == 'UnschedulablePendingReboot':
- print(' -> Unhealthy — rebooting to re-run health checks')
- if deep == 'InProgress': print(' -> Deep health check in progress')
- elif deep == 'Failed': print(' -> Deep health check FAILED — node will be replaced')
- if fault_type: print(f' -> Fault: {fault_type} | {fault_reason}')
- print(f'ISSUES={issues}')
- " 2>/dev/null || echo "ISSUES=0")
- local health_issues
- health_issues=$(echo "$health_output" | grep "^ISSUES=" | cut -d= -f2 || echo 0)
- echo "$health_output" | { grep -v "^ISSUES=" || true; } | while IFS= read -r line; do
- echo -e "$line"
- done
- # Using `if` instead of `[[ ... ]] && ...` — the short-circuit form returns
- # non-zero when the test is false, which aborts the script under `set -e`
- # and silently skips every remaining check (pods, env vars, hardware).
- if [[ "${health_issues:-0}" -gt 0 ]]; then
- ISSUES_FOUND=$((ISSUES_FOUND + health_issues))
- fi
- }
- check_pod_status() {
- local ns_flag ns_label
- if [[ -n "$NAMESPACE" ]]; then
- ns_flag=(-n "$NAMESPACE")
- ns_label="'$NAMESPACE'"
- else
- ns_flag=(-A)
- ns_label="all namespaces"
- fi
- header "Check 5: Pod / Job Status ($ns_label)"
- local job_filter=()
- [[ -n "$JOB_NAME" ]] && job_filter=(-l "job-name=$JOB_NAME")
- # `${arr[@]+"${arr[@]}"}` — expand only if defined; plain `${arr[@]}`
- # trips `set -u` on empty arrays under bash 4.2 (AL2 default).
- local pods_json
- pods_json=$(kubectl get pods "${ns_flag[@]}" ${job_filter[@]+"${job_filter[@]}"} -o json 2>/dev/null \
- || echo '{"items":[]}')
- local pod_output
- pod_output=$(python3 -c "
- import sys, json
- d = json.load(sys.stdin)
- items = d['items']
- total = len(items)
- failed = [p for p in items if p.get('status',{}).get('phase') in ('Failed','Unknown')]
- pending = [p for p in items if p.get('status',{}).get('phase') == 'Pending']
- crashes = []
- for p in items:
- for cs in p.get('status',{}).get('containerStatuses',[]):
- if cs.get('restartCount',0)>2 or cs.get('state',{}).get('waiting',{}).get('reason') \
- in ('CrashLoopBackOff','OOMKilled','Error'):
- crashes.append(p)
- break
- print(f'TOTAL={total}')
- print(f'FAILED={len(failed)}')
- print(f'PENDING={len(pending)}')
- print(f'CRASH={len(crashes)}')
- for p in failed[:5]:
- name = p['metadata']['name']
- ns = p['metadata']['namespace']
- msg = p.get('status',{}).get('message','')[:150]
- print(f'FAILED_POD={ns}/{name}: {msg}')
- for p in pending[:5]:
- name = p['metadata']['name']
- ns = p['metadata']['namespace']
- for c in p.get('status',{}).get('conditions',[]):
- if c.get('status')=='False':
- print(f'PENDING_POD={ns}/{name}: {c.get(\"message\",\"\")[:120]}')
- for p in crashes[:5]:
- name = p['metadata']['name']
- ns = p['metadata']['namespace']
- for cs in p.get('status',{}).get('containerStatuses',[]):
- r = cs.get('state',{}).get('waiting',{}).get('reason','CrashLoop')
- print(f'CRASH_POD={ns}/{name}: {r} restarts={cs.get(\"restartCount\",0)}')
- break
- " <<< "$pods_json" 2>/dev/null || echo "TOTAL=0
- FAILED=0
- PENDING=0
- CRASH=0")
- # Parse counts outside of pipe to avoid subshell variable loss
- local p_total p_failed p_pending p_crash
- p_total=$(echo "$pod_output" | grep "^TOTAL=" | cut -d= -f2 || echo 0)
- p_failed=$(echo "$pod_output" | grep "^FAILED=" | cut -d= -f2 || echo 0)
- p_pending=$(echo "$pod_output" | grep "^PENDING=" | cut -d= -f2 || echo 0)
- p_crash=$(echo "$pod_output" | grep "^CRASH=" | cut -d= -f2 || echo 0)
- info " Total pods: ${p_total:-0}"
- if [[ "${p_failed:-0}" -gt 0 ]]; then
- error " Failed/Unknown pods: $p_failed"; ISSUES_FOUND=$((ISSUES_FOUND+1)); add_issue_detail "$p_failed Failed/Unknown pod(s) → references/debugging-guide.md § 20 Pending / CrashLoopBackOff / Init-Container Failures" "P1"
- else
- success " No failed pods"
- fi
- if [[ "${p_pending:-0}" -gt 0 ]]; then
- warn " Pending pods: $p_pending"; ISSUES_FOUND=$((ISSUES_FOUND+1)); add_issue_detail "$p_pending Pending pod(s) → references/debugging-guide.md § 20 Pending / CrashLoopBackOff / Init-Container Failures" "P1"
- else
- success " No pending pods"
- fi
- if [[ "${p_crash:-0}" -gt 0 ]]; then
- error " CrashLoop/OOM pods: $p_crash"; ISSUES_FOUND=$((ISSUES_FOUND+1)); add_issue_detail "$p_crash CrashLoopBackOff/OOM pod(s) → references/debugging-guide.md § 20 Pending / CrashLoopBackOff / Init-Container Failures" "P1"
- else
- success " No crashloop pods"
- fi
- # `|| true` — grep returns 1 on no-match; with `pipefail` that kills the
- # whole function, silently skipping the rest of the diagnostic.
- echo "$pod_output" | { grep "^FAILED_POD=" || true; } | while IFS= read -r line; do error " ${line#FAILED_POD=}"; done
- echo "$pod_output" | { grep "^PENDING_POD=" || true; } | while IFS= read -r line; do warn " ${line#PENDING_POD=}"; done
- echo "$pod_output" | { grep "^CRASH_POD=" || true; } | while IFS= read -r line; do error " ${line#CRASH_POD=}"; done
- }
- # Checks EKS-specific prerequisites that cause NCCL failures before training starts:
- # - Headless service for MASTER_ADDR DNS resolution
- # - Init container failures blocking training containers
- # - /dev/shm volume mount (K8s default 64MB is too small for NCCL)
- check_nccl_infra_prereqs() {
- header "Check 5b: NCCL Infrastructure Prerequisites"
- local ns_flag ns_label
- if [[ -n "$NAMESPACE" ]]; then
- ns_flag=(-n "$NAMESPACE")
- ns_label="'$NAMESPACE'"
- else
- ns_flag=(-A)
- ns_label="all namespaces"
- fi
- local job_filter=()
- [[ -n "$JOB_NAME" ]] && job_filter=(-l "job-name=$JOB_NAME")
- # MASTER_ADDR DNS resolution requires a headless service (ClusterIP: None)
- # Without it, pods get DNS like "10-0-1-5.default.pod.cluster.local" which
- # doesn't resolve from other pods → rendezvous timeout
- section "Headless Service (MASTER_ADDR DNS)"
- local headless_svcs
- headless_svcs=$(kubectl get svc "${ns_flag[@]}" -o json 2>/dev/null | python3 -c "
- import sys, json
- data = json.load(sys.stdin)
- found = []
- for svc in data.get('items', []):
- spec = svc.get('spec', {})
- if spec.get('clusterIP') == 'None':
- name = svc['metadata']['name']
- ns = svc['metadata']['namespace']
- sel = spec.get('selector', {})
- found.append(f'{ns}/{name} selector={sel}')
- print(f'COUNT={len(found)}')
- for f in found[:10]:
- print(f)
- " 2>/dev/null || echo "COUNT=0")
- local hl_count
- hl_count=$(echo "$headless_svcs" | grep "^COUNT=" | cut -d= -f2 || echo 0)
- if [[ "${hl_count:-0}" -gt 0 ]]; then
- success "Headless service(s) found (${hl_count}) — MASTER_ADDR DNS can resolve"
- echo "$headless_svcs" | { grep -v "^COUNT=" || true; } | while IFS= read -r line; do
- [[ -n "$line" ]] && info " $line"
- done
- else
- warn "No headless services found in $ns_label"
- warn " If MASTER_ADDR uses a hostname, DNS resolution will fail"
- warn " Example: spec.clusterIP: None, spec.selector: {app: my-training-job}"
- fi
- # Init containers must complete before training container starts.
- # Common failures: S3 data download, config fetch, health check wait
- section "Init Container Status"
- local init_issues
- init_issues=$(kubectl get pods "${ns_flag[@]}" ${job_filter[@]+"${job_filter[@]}"} -o json 2>/dev/null | python3 -c "
- import sys, json
- data = json.load(sys.stdin)
- issues = 0
- for pod in data.get('items', []):
- name = pod['metadata']['name']
- ns = pod['metadata']['namespace']
- for ics in pod.get('status', {}).get('initContainerStatuses', []):
- state = ics.get('state', {})
- if 'waiting' in state:
- reason = state['waiting'].get('reason', '')
- msg = state['waiting'].get('message', '')[:100]
- if reason in ('CrashLoopBackOff', 'Error', 'ImagePullBackOff', 'ErrImagePull'):
- print(f'FAIL:{ns}/{name}: init container \"{ics[\"name\"]}\" {reason}: {msg}')
- issues += 1
- elif 'terminated' in state and state['terminated'].get('exitCode', 0) != 0:
- reason = state['terminated'].get('reason', 'Error')
- print(f'FAIL:{ns}/{name}: init container \"{ics[\"name\"]}\" exited {state[\"terminated\"][\"exitCode\"]}: {reason}')
- issues += 1
- print(f'ISSUES={issues}')
- " 2>/dev/null || echo "ISSUES=0")
- local init_count
- init_count=$(echo "$init_issues" | grep "^ISSUES=" | cut -d= -f2 || echo 0)
- if [[ "${init_count:-0}" -gt 0 ]]; then
- error " $init_count init container failure(s) — training containers cannot start"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "$init_count failed init container(s) blocking training → references/debugging-guide.md § 20 Pending / CrashLoopBackOff / Init-Container Failures" "P1"
- echo "$init_issues" | { grep "^FAIL:" || true; } | while IFS= read -r line; do
- error " ${line#FAIL:}"
- done
- else
- success "No init container failures"
- fi
- # K8s default /dev/shm = 64MB. NCCL needs ≥1GB. Without emptyDir mount,
- # training gets "failed to extend /dev/shm/nccl-*" or SIGBUS.
- section "/dev/shm Volume Mount"
- if [[ -n "$JOB_NAME" ]]; then
- local ns="${NAMESPACE:-default}"
- local shm_check
- shm_check=$(kubectl get pods -n "$ns" -l "job-name=$JOB_NAME" -o json 2>/dev/null | python3 -c "
- import sys, json
- data = json.load(sys.stdin)
- pods = data.get('items', [])
- if not pods:
- print('NO_PODS')
- else:
- pod = pods[0]
- vols = pod.get('spec', {}).get('volumes', [])
- has_dshm = any(
- v.get('emptyDir', {}).get('medium') == 'Memory'
- for v in vols
- if any(vm.get('mountPath') == '/dev/shm'
- for c in pod.get('spec', {}).get('containers', [])
- for vm in c.get('volumeMounts', [])
- if vm.get('name') == v.get('name'))
- )
- if has_dshm:
- print('OK')
- else:
- print('MISSING')
- " 2>/dev/null || echo "UNKNOWN")
- case "$shm_check" in
- OK) success "/dev/shm mounted as emptyDir Memory — NCCL shared memory OK" ;;
- MISSING) warn "/dev/shm NOT mounted as emptyDir Memory (K8s default = 64MB)"
- warn " NCCL will fail with 'failed to extend /dev/shm/nccl-*' or Bus error"
- warn " volumes: [{name: dshm, emptyDir: {medium: Memory, sizeLimit: '10Gi'}}]"
- warn " volumeMounts: [{name: dshm, mountPath: /dev/shm}]"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "/dev/shm not mounted as emptyDir Memory → references/debugging-guide.md § 17 RDMA Memory Registration Failure" "P1" ;;
- NO_PODS) info "No pods found for job '$JOB_NAME' — /dev/shm check skipped" ;;
- *) info "/dev/shm mount status unknown" ;;
- esac
- else
- info "/dev/shm check requires --job flag (skipped)"
- fi
- }
- analyze_nccl_logs() {
- header "Check 6: NCCL Log Pattern Analysis"
- local job_filter=()
- [[ -n "$JOB_NAME" ]] && job_filter=(-l "job-name=$JOB_NAME")
- local pod_entries
- if [[ -n "$NAMESPACE" ]]; then
- pod_entries=$(kubectl get pods -n "$NAMESPACE" ${job_filter[@]+"${job_filter[@]}"} --no-headers 2>/dev/null \
- | awk -v ns="$NAMESPACE" '{print ns"/"$1}' || echo "")
- else
- pod_entries=$(kubectl get pods -A ${job_filter[@]+"${job_filter[@]}"} --no-headers 2>/dev/null \
- | awk '{print $1"/"$2}' \
- | grep -vE "^(kube-system|kube-public|kube-node-lease|aws-hyperpod)/" || true)
- fi
- if [[ -z "$pod_entries" ]]; then
- info "No workload pods found to analyze logs"
- return
- fi
- local NCCL_PAT_KEYS=(
- "Timeout waiting for"
- "Connection refused"
- "Address already in use"
- "NCCL WARN Connect to"
- "network is unreachable"
- "Error in Store"
- "DistStoreError"
- "RendezvousConnectionError"
- "RendezvousTimeout"
- "Name or service not known"
- "getaddrinfo failed"
- "Watchdog timeout"
- "unhandled system error"
- "unhandled cuda error"
- "peer access is not supported"
- "NCCL WARN Cuda failure"
- "fi_getinfo failed"
- "NCCL_OFI_RDMA"
- "Call to ibv_reg_mr failed"
- "NET/OFI Using TCP"
- "Failed to load NCCL"
- "libnccl-net.so"
- "OOMKilled"
- "CUDA out of memory"
- "cudaMalloc failed"
- "failed to extend /dev/shm"
- "Bus error"
- "NCCL function not found"
- "Incompatible NCCL version"
- "Could not find interface"
- "world_size mismatch"
- "doesn't have NCCL built in"
- "CUDA_VISIBLE_DEVICES"
- "unlink shared memory"
- "Call to ncclCommAbort"
- "MNNVL topology"
- "ENOMEM"
- "invalid alignment"
- )
- local NCCL_PAT_VALS=(
- "TIMEOUT_RENDEZVOUS:Rendezvous timed out — peer ranks not responding"
- "CONN_REFUSED:TCP refused — check MASTER_ADDR/MASTER_PORT"
- "PORT_CONFLICT:Port already in use — change MASTER_PORT"
- "CONNECT_FAIL:NCCL peer connection failed — check SG/NetworkPolicy"
- "NET_UNREACHABLE:Network unreachable — VPC/routing issue"
- "STORE_ERR:Distributed store error — usually rendezvous timeout"
- "STORE_ERR:Distributed store error (PyTorch 2.x) — usually rendezvous timeout"
- "RDZV_CONN_ERR:Torch elastic rendezvous connection failed — check MASTER_ADDR DNS + SG"
- "RDZV_TIMEOUT:Torch elastic rendezvous timed out — peers not reachable"
- "DNS_FAIL:DNS resolution failed for MASTER_ADDR — check headless service or /etc/hosts"
- "DNS_FAIL:DNS resolution failed — headless service missing or CoreDNS issue"
- "WATCHDOG_TIMEOUT:AllReduce watchdog expired — straggler or OOM"
- "SYSTEM_ERROR:NCCL system error — GPU/EFA hardware issue"
- "CUDA_ERROR:CUDA runtime error — GPU driver crash or hardware fault"
- "P2P_FAIL:GPU peer access blocked — ACS enabled or IOMMU misconfigured"
- "CUDA_ERROR:CUDA failure inside NCCL — GPU hardware or driver issue"
- "EFA_INIT_FAIL:EFA libfabric init failed — EFA not available or wrong NCCL_SOCKET_IFNAME"
- "OFI_ERROR:aws-ofi-nccl plugin error — check EFA driver and OFI NCCL version"
- "RDMA_REG_FAIL:EFA/RDMA memory registration failed — memlock limit too low"
- "EFA_TCP_FALLBACK:NCCL fell back to TCP instead of EFA — 10-100x slower than expected"
- "NCCL_LOAD_FAIL:Failed to load NCCL library — libnccl.so missing or LD_LIBRARY_PATH wrong"
- "OFI_LOAD_FAIL:Failed to load aws-ofi-nccl plugin — libnccl-net.so not found"
- "OOM_KILL:Container killed (OOM) — reduce batch size or increase memory limit"
- "CUDA_OOM:GPU out of memory — reduce batch size or model size"
- "CUDA_OOM:GPU cudaMalloc failed — reduce batch size or model size"
- "SHM_FULL:NCCL shared memory /dev/shm full — mount emptyDir with 10Gi sizeLimit"
- "SHM_FULL:/dev/shm too small or SIGBUS — mount emptyDir with 10Gi sizeLimit"
- "NCCL_VERSION_MISMATCH:NCCL version mismatch across nodes — mixed container images"
- "NCCL_VERSION_MISMATCH:NCCL version mismatch across nodes — mixed container images"
- "IFACE_NOT_FOUND:NCCL_SOCKET_IFNAME points to missing interface"
- "WORLD_SIZE_MISMATCH:WORLD_SIZE doesn't match running process count"
- "NCCL_NOT_BUILT:PyTorch compiled without NCCL — rebuild with USE_NCCL=1 or use AWS DLC image"
- "CUDA_VIS_DEV:CUDA_VISIBLE_DEVICES misconfigured — GPUs not visible to training process"
- "SHM_STALE:Stale NCCL shared memory from previous run — systemd RemoveIPC=yes or manual cleanup"
- "NCCL_COMM_ABORT:NCCL communicator aborted — check for straggler node or hardware fault"
- "MNNVL_TOPO_FAIL:NCCL MNNVL topology search failed — memlock=unlimited + stack=unlimited causes 2MB thread stack; fix: ulimit -l 8388608 -s 8192"
- "ENOMEM:Memory registration/allocation failed — check memlock limits and available GPU memory"
- "CUDA_ALIGN_ERR:CUDA memory alignment error — possible driver/NCCL version incompatibility"
- )
- local issues_in_logs=false
- while IFS= read -r entry; do
- local ns pod
- ns="${entry%%/*}"; pod="${entry#*/}"
- section "Logs: $ns/$pod"
- local logs
- # Use --tail=500 to catch patterns even in longer outputs.
- # For Failed/Error pods, also check --previous (logs from the crashed container instance).
- local pod_phase
- pod_phase=$(kubectl get pod -n "$ns" "$pod" -o jsonpath='{.status.phase}' 2>/dev/null || echo "")
- logs=$(kubectl logs -n "$ns" "$pod" --tail=500 2>/dev/null || echo "")
- if [[ -z "$logs" ]]; then
- logs=$(kubectl logs -n "$ns" "$pod" --previous --tail=500 2>/dev/null || echo "")
- elif [[ "$pod_phase" == "Failed" ]]; then
- local prev_logs
- prev_logs=$(kubectl logs -n "$ns" "$pod" --previous --tail=500 2>/dev/null || echo "")
- [[ -n "$prev_logs" ]] && logs="${logs}"$'\n'"${prev_logs}"
- fi
- if [[ -z "$logs" ]]; then
- info " No logs available"
- continue
- fi
- local _pi
- for _pi in "${!NCCL_PAT_KEYS[@]}"; do
- local pattern="${NCCL_PAT_KEYS[$_pi]}"
- if echo "$logs" | grep -qi "$pattern"; then
- local meaning="${NCCL_PAT_VALS[$_pi]}"
- local code="${meaning%%:*}"
- local desc="${meaning#*:}"
- error " DETECTED [$code]: $desc"
- echo "$logs" | { grep -i "$pattern" || true; } | tail -3 | while IFS= read -r logline; do
- echo -e " ${YELLOW}> $logline${RESET}"
- done
- issues_in_logs=true
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "NCCL log pattern [$code] in pod $pod: $desc → references/error-patterns-quick-ref.md" "P1"
- fi
- done
- if echo "$logs" | grep -qiE "BASELINE TEST PASSED|AllReduce SUCCESS|Training complete"; then
- success " Pod $pod: completed successfully"
- fi
- done <<< "$pod_entries"
- $issues_in_logs || success "No NCCL error patterns found in pod logs"
- }
- check_nccl_env_vars() {
- header "Check 7: NCCL Environment Variable Audit"
- local job_filter=()
- [[ -n "$JOB_NAME" ]] && job_filter=(-l "job-name=$JOB_NAME")
- local ns="${NAMESPACE:-default}"
- local first_pod
- first_pod=$(kubectl get pods -n "$ns" ${job_filter[@]+"${job_filter[@]}"} --no-headers 2>/dev/null \
- | grep -E " Running " | head -1 | awk '{print $1}' || echo "")
- if [[ -z "$first_pod" ]]; then
- info "No Running pods found for env var audit (only meaningful during active training)"
- return
- fi
- info "Checking env vars in Running pod: $ns/$first_pod"
- local pod_env
- pod_env=$(kubectl exec -n "$ns" "$first_pod" -- env 2>/dev/null || echo "")
- if [[ -z "$pod_env" ]]; then
- warn "Could not exec into $first_pod"
- return
- fi
- # Capture Python output; the sentinel line feeds issue accounting below.
- local env_audit_out env_warn_count
- env_audit_out=$(python3 - <<'PYEOF' "$pod_env"
- import sys
- pod_env = sys.argv[1] if len(sys.argv) > 1 else ""
- env_map = {}
- for line in pod_env.strip().split('\n'):
- if '=' in line:
- k, _, v = line.partition('=')
- env_map[k.strip()] = v.strip()
- # (rec_value, severity, description)
- # severity WARN = counts as issue; INFO = advisory only
- checks = {
- 'MASTER_ADDR': (None, 'WARN', 'Must be rank-0 pod hostname/IP'),
- 'MASTER_PORT': ('29500', 'WARN', 'Must match across all ranks'),
- 'WORLD_SIZE': (None, 'WARN', 'Must equal total processes'),
- 'RANK': (None, 'WARN', 'Must be unique 0..WORLD_SIZE-1'),
- 'NCCL_SOCKET_IFNAME': ('^lo,docker,efa,veth,virbr', 'WARN', 'Exclude non-VPC interfaces (loopback/docker/EFA control/veth)'),
- 'NCCL_TIMEOUT': ('1200', 'WARN', 'Default 600s too short for large clusters'),
- 'FI_PROVIDER': ('efa', 'INFO', 'Set to efa on EFA instances; omit for CPU-only'),
- 'FI_EFA_USE_DEVICE_RDMA': ('1', 'INFO', 'Required for full EFA RDMA performance'),
- 'NCCL_DEBUG': ('WARN', 'INFO', 'Enable for diagnostics (use WARN not INFO in prod)'),
- }
- print(" {:<28} {:<22} {}".format('Variable','Value','Status'))
- print(" " + "-"*68)
- warn_count = 0
- for var,(rec,sev,desc) in checks.items():
- val = env_map.get(var)
- if val:
- print(f" [SET] {var:<26} {val:<22}")
- elif sev == 'WARN':
- warn_count += 1
- print(f" [WARN] {var:<26} {'(not set)':<22} <- {desc}")
- else:
- print(f" [INFO] {var:<26} {'(not set)':<22} <- {desc}")
- nccl_debug = env_map.get('NCCL_DEBUG', '')
- if nccl_debug.upper() == 'INFO':
- warn_count += 1
- print("\n [WARN] NCCL_DEBUG=INFO detected in production job — verbose logging adds runtime overhead; set to WARN for production")
- elif nccl_debug.upper() == 'TRACE':
- warn_count += 1
- print("\n [WARN] NCCL_DEBUG=TRACE detected — TRACE prints replayable trace info on every NCCL call (per the NCCL env-var doc); large overhead and gigabytes of logs per rank, set to WARN immediately")
- # NCCL_TIMEOUT value validation (formula: nodes * 5 + 600)
- nccl_timeout_str = env_map.get('NCCL_TIMEOUT', '')
- world_size_str = env_map.get('WORLD_SIZE', '0')
- try:
- world_size = int(world_size_str)
- except ValueError:
- world_size = 0
- if nccl_timeout_str and world_size > 0:
- try:
- nccl_timeout = int(nccl_timeout_str)
- recommended = world_size * 5 + 600
- if nccl_timeout < recommended:
- warn_count += 1
- print(f"\n [WARN] NCCL_TIMEOUT={nccl_timeout}s may be too low for {world_size} ranks (recommended >= {recommended}s)")
- except ValueError:
- pass
- # Large cluster checks (256+ nodes)
- if world_size > 256:
- warn_count += 1
- print(f"\n [WARN] WORLD_SIZE={world_size} (large cluster) — verify memlock and stack ulimits")
- if warn_count == 0:
- print("\n [PASS] All critical NCCL env vars configured")
- else:
- print(f"\n [WARN] {warn_count} critical NCCL env var(s) not set or misconfigured")
- # Sentinel line consumed by the caller — DO NOT remove.
- print(f"__WARN_COUNT__={warn_count}")
- PYEOF
- )
- echo "$env_audit_out" | grep -v '^__WARN_COUNT__='
- env_warn_count=$(echo "$env_audit_out" | grep '^__WARN_COUNT__=' | cut -d= -f2)
- if [[ "${env_warn_count:-0}" =~ ^[0-9]+$ ]] && (( env_warn_count > 0 )); then
- ISSUES_FOUND=$((ISSUES_FOUND + env_warn_count))
- add_issue_detail "${env_warn_count} NCCL env var issue(s) in pod ${ns}/${first_pod} → references/operations.md § 5 NCCL environment variable reference" "P1"
- fi
- }
- # EFA device plugin + NCCL version consistency. kubectl-only, no active job needed.
- check_efa_k8s() {
- header "Check 2b: EFA K8s Device Plugin & NCCL Version Consistency"
- # Without this DaemonSet, pods can't request vpc.amazonaws.com/efa resources
- # and EFA interfaces won't be mounted into training containers.
- local efa_ds
- efa_ds=$(kubectl get daemonset -A 2>/dev/null | grep -iE "efa|aws-efa" | head -3 || echo "")
- if [[ -n "$efa_ds" ]]; then
- success "EFA device plugin DaemonSet found:"
- echo "$efa_ds" | while IFS= read -r line; do info " $line"; done
- else
- # Missing plugin is a FAIL only if any pod requests vpc.amazonaws.com/efa.
- local ns_flag=(); if [[ -n "$NAMESPACE" ]]; then ns_flag=(-n "$NAMESPACE"); else ns_flag=(-A); fi
- local efa_requested
- efa_requested=$(kubectl get pods "${ns_flag[@]}" -o json 2>/dev/null | python3 -c "
- import sys, json
- data = json.load(sys.stdin)
- for pod in data.get('items', []):
- for c in pod.get('spec', {}).get('containers', []):
- lims = c.get('resources', {}).get('limits', {})
- if 'vpc.amazonaws.com/efa' in lims:
- ns = pod['metadata']['namespace']
- name = pod['metadata']['name']
- count = lims['vpc.amazonaws.com/efa']
- print(f' {ns}/{name}: requests {count} EFA interface(s)')
- " 2>/dev/null || echo "")
- if [[ -n "$efa_requested" ]]; then
- error "Pods request EFA resources but EFA device plugin DaemonSet NOT found!"
- error " EFA interfaces will NOT be mounted into training containers"
- echo "$efa_requested"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "EFA device plugin DaemonSet missing → references/operations.md § 5 NCCL environment variable reference / references/debugging-guide.md § 6 EFA Configuration" "P0"
- else
- info "EFA device plugin not detected (OK if no pods request vpc.amazonaws.com/efa)"
- fi
- fi
- # Mixed NCCL versions across nodes → 'NCCL function not found' at init.
- # Two independent probes:
- # - torch.cuda.nccl.version(): works only if PyTorch is installed.
- # - libnccl.so on disk: authoritative — this is what actually loads at
- # runtime, works for any image (PyTorch, JAX, raw NCCL, custom).
- if [[ -n "$JOB_NAME" ]]; then
- section "NCCL Version Consistency (job: $JOB_NAME)"
- local ns="${NAMESPACE:-default}"
- local job_pods
- job_pods=$(kubectl get pods -n "$ns" -l "job-name=$JOB_NAME" --no-headers 2>/dev/null \
- | grep -E " Running " | awk '{print $1}' | head -4 || echo "")
- if [[ -z "$job_pods" ]]; then
- info "No Running pods in job '$JOB_NAME' — version check skipped"
- else
- # Read-only probe: find libnccl.so*, extract embedded version string,
- # fall back to SONAME filename parsing when `strings` is unavailable.
- # Variables below are expanded inside the remote pod via `kubectl exec
- # sh -c`, NOT locally — the quoted heredoc prevents local expansion.
- local lib_probe
- lib_probe=$(cat <<'REMOTE_PROBE'
- NCCL_LIB=$(find /usr/local/cuda/lib64 /usr/lib /usr/lib64 /usr/lib/x86_64-linux-gnu /opt/nccl/lib /opt/amazon/ofi-nccl/lib -maxdepth 3 -name "libnccl.so*" -type f 2>/dev/null | head -1)
- if [ -z "$NCCL_LIB" ]; then echo "not-found"; exit 0; fi
- VER=$(strings "$NCCL_LIB" 2>/dev/null | grep -oE "NCCL version [0-9]+\.[0-9]+\.[0-9]+" | head -1 | sed "s/NCCL version //")
- [ -z "$VER" ] && VER=$(basename "$(readlink -f "$NCCL_LIB")" 2>/dev/null | grep -oE "[0-9]+\.[0-9]+\.[0-9]+" | head -1)
- [ -z "$VER" ] && VER="present-no-version"
- echo "$VER"
- REMOTE_PROBE
- )
- local torch_versions=()
- local lib_versions=()
- for pod in $job_pods; do
- local tver lver
- tver=$(kubectl exec -n "$ns" "$pod" -- \
- python3 -c "import torch; print(torch.cuda.nccl.version())" 2>/dev/null \
- || echo "unavailable")
- lver=$(kubectl exec -n "$ns" "$pod" -- sh -c "$lib_probe" 2>/dev/null \
- || echo "unavailable")
- info " $pod: torch.nccl=$tver libnccl.so=$lver"
- torch_versions+=("$tver")
- lib_versions+=("$lver")
- done
- local unique_torch unique_lib
- unique_torch=$(printf '%s\n' "${torch_versions[@]}" | grep -v unavailable | sort -u | wc -l | tr -d ' ')
- unique_lib=$(printf '%s\n' "${lib_versions[@]}" \
- | grep -vE "unavailable|not-found|present-no-version" | sort -u | wc -l | tr -d ' ')
- if [[ "$unique_torch" -gt 1 ]]; then
- error "NCCL VERSION MISMATCH (torch.cuda.nccl.version) across pods — will cause 'NCCL function not found' at init!"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "NCCL version mismatch across pods (torch) → references/debugging-guide.md § 10 NCCL Version Mismatch" "P1"
- fi
- if [[ "$unique_lib" -gt 1 ]]; then
- error "libnccl.so VERSION MISMATCH across pods — mixed NCCL libraries will cause symbol errors at init!"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "libnccl.so version mismatch across pods → references/debugging-guide.md § 10 NCCL Version Mismatch" "P1"
- fi
- if [[ "$unique_torch" -le 1 ]] && [[ "$unique_lib" -le 1 ]]; then
- if [[ "$unique_torch" -eq 1 ]] || [[ "$unique_lib" -eq 1 ]]; then
- success "NCCL version consistent across ${#lib_versions[@]} pod(s)"
- else
- info "NCCL version unavailable (neither torch nor libnccl.so could be probed)"
- fi
- fi
- fi
- fi
- }
- check_network_policies() {
- header "Check 9: Kubernetes NetworkPolicy Scan"
- local np_flag np_label
- if [[ -n "$NAMESPACE" ]]; then
- np_flag=(-n "$NAMESPACE")
- np_label="'$NAMESPACE'"
- else
- np_flag=(-A)
- np_label="all namespaces"
- fi
- local policies
- policies=$(kubectl get networkpolicy "${np_flag[@]}" 2>/dev/null || echo "")
- if [[ -z "$policies" ]] || echo "$policies" | grep -q "No resources found"; then
- success "No NetworkPolicies in $np_label — all traffic allowed"
- return
- fi
- # Informational — only raise a finding when the per-policy scan below
- # identifies one that actually blocks all ingress/egress. Narrow allow-list
- # policies (e.g. operator-scoped ingress) are common and not a defect.
- info "NetworkPolicies found in $np_label — review each for NCCL impact:"
- echo "$policies"
- local scope_flag
- local scope_flag=()
- if [[ -n "$NAMESPACE" ]]; then scope_flag=(-n "$NAMESPACE"); else scope_flag=(-A); fi
- kubectl get networkpolicy "${scope_flag[@]}" -o json 2>/dev/null | python3 -c "
- import sys, json
- data = json.load(sys.stdin)
- for pol in data.get('items', []):
- name = pol['metadata']['name']
- ns = pol['metadata']['namespace']
- spec = pol.get('spec', {})
- types = spec.get('policyTypes', [])
- ingress = spec.get('ingress', [])
- egress = spec.get('egress', [])
- print(f' Policy: {ns}/{name} | Types: {types}')
- if 'Ingress' in types and not ingress:
- print(f' [FAIL] BLOCKS ALL INBOUND — will break NCCL rendezvous and AllReduce!')
- if 'Egress' in types and not egress:
- print(f' [FAIL] BLOCKS ALL OUTBOUND — will break NCCL communication!')
- if ('Ingress' not in types) and ('Egress' not in types):
- print(f' [INFO] Policy has no policyTypes — acts as allow-all')
- " 2>/dev/null
- local scope_flag2
- local scope_flag2=()
- if [[ -n "$NAMESPACE" ]]; then
- scope_flag2=(-n "$NAMESPACE")
- else
- scope_flag2=(-A)
- fi
- local blocking_list
- blocking_list=$(kubectl get networkpolicy "${scope_flag2[@]}" -o json 2>/dev/null | python3 -c "
- import sys, json
- try:
- data = json.load(sys.stdin)
- except json.JSONDecodeError:
- # kubectl returned non-JSON (empty stdin, error text, or version-skew output).
- # Skip this check rather than aborting the overall diagnostic run.
- sys.exit(0)
- for pol in data.get('items', []):
- name = pol['metadata']['name']
- ns = pol['metadata']['namespace']
- spec = pol.get('spec', {})
- types = spec.get('policyTypes', [])
- ingress = spec.get('ingress', [])
- egress = spec.get('egress', [])
- blocks = ('Ingress' in types and not ingress) or ('Egress' in types and not egress)
- if blocks:
- print(f'{ns}/{name}')
- " 2>/dev/null || echo "")
- if [[ -n "$blocking_list" ]]; then
- while IFS= read -r bp; do
- [[ -z "$bp" ]] && continue
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "Blocking NetworkPolicy $bp may prevent NCCL traffic → references/operations.md § 8 NCCL-specific remediations (NetworkPolicy)" "P1"
- done <<< "$blocking_list"
- fi
- }
- # Populates SSM_CLUSTER_ID and SSM_NODES array (up to SAMPLE_NODES entries).
- # Each entry is "INSTANCE_ID GROUP_NAME".
- # Prefers worker/compute nodes over controller/head nodes.
- # Respects --node <INSTANCE_ID> if provided.
- resolve_cluster_nodes_for_ssm() {
- SSM_CLUSTER_ID=""
- SSM_NODES=()
- local cluster_arn
- cluster_arn=$(aws sagemaker describe-cluster \
- --cluster-name "$CLUSTER_NAME" --region "$REGION" \
- --query 'ClusterArn' --output text 2>/dev/null || echo "")
- if [[ -z "$cluster_arn" || "$cluster_arn" == "None" ]]; then
- debug "resolve_cluster_nodes_for_ssm: describe-cluster returned empty ARN"
- return 1
- fi
- SSM_CLUSTER_ID=$(echo "$cluster_arn" | awk -F'/' '{print $NF}')
- local nodes_json
- nodes_json=$(sagemaker_list_paginated list-cluster-nodes ClusterNodeSummaries)
- if [[ -n "$NODE_ID" ]]; then
- local grp
- grp=$(echo "$nodes_json" | python3 -c "
- import sys, json
- target = sys.argv[1]
- nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
- for n in nodes:
- if n.get('InstanceId') == target:
- print(n.get('InstanceGroupName','worker'))
- break
- " "$NODE_ID" 2>/dev/null | head -1)
- [[ -z "$grp" ]] && grp="worker"
- SSM_NODES=("$NODE_ID $grp")
- return 0
- fi
- local all_nodes
- all_nodes=$(echo "$nodes_json" | python3 -c "
- import sys, json
- print(json.dumps(json.load(sys.stdin).get('ClusterNodeSummaries', [])))
- " 2>/dev/null || echo '[]')
- # For NCCL diagnostics, hardware probes (nvidia-smi, fi_info -p efa,
- # neuron-ls) only produce meaningful signal on GPU / accelerator nodes.
- # Prioritize by type: GPU/Neuron first, other Running compute next, then
- # fall back to any Running node so the script still reports on a cluster
- # that has only CPU nodes.
- local picked
- picked=$(echo "$all_nodes" | python3 -c "
- import sys, json
- nodes = json.load(sys.stdin)
- sample = $SAMPLE_NODES
- # Instance-type prefixes that carry NVIDIA GPUs or AWS Trainium/Inferentia.
- # A node's instance type shows up in ClusterNodeSummaries as e.g. 'ml.p5.48xlarge'.
- GPU_PREFIXES = ('ml.p3', 'ml.p3dn', 'ml.p4d', 'ml.p4de', 'ml.p5', 'ml.p5e',
- 'ml.p5en', 'ml.p6', 'ml.g4dn', 'ml.g5', 'ml.g6', 'ml.g6e', 'ml.g7e')
- NEURON_PREFIXES = ('ml.trn1', 'ml.trn2', 'ml.inf2')
- ACCEL_PREFIXES = GPU_PREFIXES + NEURON_PREFIXES
- def is_utility_group(name):
- n = (name or '').lower()
- return any(x in n for x in ('controller', 'head', 'master'))
- def itype(n):
- return n.get('InstanceType', '') or ''
- running = [n for n in nodes if n.get('InstanceStatus', {}).get('Status', '') == 'Running']
- # Tier 1: running + accelerator type + not a controller group
- tier1 = [n for n in running if itype(n).startswith(ACCEL_PREFIXES) and not is_utility_group(n.get('InstanceGroupName', ''))]
- # Tier 2: running + non-controller (may be CPU-only compute)
- tier2 = [n for n in running if n not in tier1 and not is_utility_group(n.get('InstanceGroupName', ''))]
- # Tier 3: anything else running (utility / controller nodes, last resort)
- tier3 = [n for n in running if n not in tier1 and n not in tier2]
- results = []
- for n in tier1 + tier2 + tier3:
- if len(results) >= sample:
- break
- results.append(n['InstanceId'] + ' ' + n['InstanceGroupName'])
- for r in results:
- print(r)
- " 2>/dev/null || echo "")
- if [[ -z "$picked" ]]; then
- debug "resolve_cluster_nodes_for_ssm: no Running nodes found"
- return 1
- fi
- while IFS= read -r line; do
- [[ -n "$line" ]] && SSM_NODES+=("$line")
- done <<< "$picked"
- return 0
- }
- # Usage: _ssm_run INSTANCE_ID GROUP_NAME CLUSTER_ID SCRIPT_BODY
- # Returns the stdout of the remote script, or empty on failure.
- _ssm_run() {
- local instance_id="$1"
- local group_name="$2"
- local cluster_id="$3"
- local script_body="$4"
- # Validate inputs before interpolating into the SSM target string.
- [[ -z "$instance_id" || -z "$group_name" || -z "$cluster_id" || -z "$script_body" ]] && return 1
- [[ ! "$instance_id" =~ ^i-[0-9a-f]{8,17}$ ]] && return 1
- [[ ! "$group_name" =~ ^[A-Za-z0-9._-]+$ ]] && return 1
- [[ ! "$cluster_id" =~ ^[A-Za-z0-9._-]+$ ]] && return 1
- local target="sagemaker-cluster:${cluster_id}_${group_name}-${instance_id}"
- local tmpfile
- tmpfile=$(mktemp "${TMPDIR:-/tmp}/nccl-ssm-XXXXXX.json") || return 1
- chmod 600 "$tmpfile" 2>/dev/null || true
- _TEMP_FILES+=("$tmpfile")
- # AWS-StartNonInteractiveCommand collapses newlines in a single command
- # element, so embed the multi-line script as a base64 payload.
- local cmd_b64
- cmd_b64=$(printf '%s' "$script_body" | base64 | tr -d '\n') || { rm -f "$tmpfile"; return 1; }
- local remote="bash -c \"echo $cmd_b64 | base64 -d | bash\""
- Note: Quoted as data in code: Decodes base64 and runs the result in a shell
- Line 1,500 on GitHub
- python3 -c "import json,sys; print(json.dumps({'command':[sys.argv[1]]}))" "$remote" > "$tmpfile" 2>/dev/null || { rm -f "$tmpfile"; return 1; }
- # session-manager-plugin races to close before flushing its last stdout
- # block; `unbuffer` (from the `expect` package) gives it a PTY and avoids
- # the resulting "Cannot perform start session: EOF". Required — see the
- # prerequisite check at script startup.
- # Retry transient SSM session errors (EOF, throttling, i/o timeout).
- # Do not retry AccessDenied / UnauthorizedOperation — permanent IAM denials.
- local out attempt=0
- while (( attempt < 5 )); do
- out=$(unbuffer timeout 180 aws ssm start-session \
- --target "$target" \
- --region "$REGION" \
- --document-name AWS-StartNonInteractiveCommand \
- --parameters "file://$tmpfile" 2>&1 || echo "")
- # Fatal (don't retry) — permanent IAM or agent state.
- if echo "$out" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized to perform|TargetNotConnected"; then
- break
- fi
- if ! echo "$out" | grep -qiE "Cannot perform start session|EOF$|SessionManagerPlugin is not found|i/o timeout|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable"; then
- break
- fi
- attempt=$((attempt + 1))
- sleep $((attempt * 3))
- done
- rm -f "$tmpfile"
- # Strip SSM session banners and the echoed base64 command line.
- echo "$out" | grep -vE '^(Starting session with SessionId:|Exiting session with sessionId:|\s*$)' \
- | grep -vE "^(bash -c \"echo [A-Za-z0-9+/=]+ \| base64 -d \| bash\"|echo '[A-Za-z0-9+/=]+'|[A-Za-z0-9+/=]{40,}={0,2})[[:space:]]*\|?[[:space:]]*base64?[[:space:]]*-?d?[[:space:]]*\|?[[:space:]]*bash\"?\$" || true
- }
- # Self-contained bash script executed on each HyperPod compute node via SSM.
- # Covers GPU, EFA, NCCL library, network, memory, and process health.
- IFS= read -r -d '' _NODE_DIAG_SCRIPT <<'NODE_SCRIPT' || true
- #!/bin/bash
- # HyperPod NCCL Node Hardware Diagnostics
- # Runs ON the compute node via SSM — NOT on the local machine.
- export PATH="/opt/amazon/efa/bin:/usr/local/cuda/bin:$PATH"
- echo "=== NODE DIAGNOSTICS ==="
- echo "Host: $(hostname)"
- echo "Date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
- echo "Kernel: $(uname -r)"
- # Instance type via IMDS (v2)
- IMDS_TOKEN=$(curl -sf -m 3 -X PUT "http://169.254.169.254/latest/api/token" \
- Dismissed after review: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
- Line 1,547 on GitHub
- -H "X-aws-ec2-metadata-token-ttl-seconds: 60" 2>/dev/null || echo "")
- if [ -n "$IMDS_TOKEN" ]; then
- INSTANCE_TYPE=$(curl -sf -m 3 -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
- "http://169.254.169.254/latest/meta-data/instance-type" 2>/dev/null || echo "unknown")
- AZ=$(curl -sf -m 3 -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
- "http://169.254.169.254/latest/meta-data/placement/availability-zone" 2>/dev/null || echo "unknown")
- else
- INSTANCE_TYPE="unknown"
- AZ="unknown"
- fi
- echo "Instance: ${INSTANCE_TYPE} | AZ: ${AZ}"
- echo ""
- echo "--- GPU ---"
- # Require both the binary AND at least one GPU visible. nvidia-smi is preinstalled
- # on some non-GPU instance types (t3/c5) but returns "No devices were found" —
- # reporting that as [FAIL] would be a false positive on controllers/logins.
- if command -v nvidia-smi &>/dev/null && nvidia-smi -L 2>/dev/null | grep -q "^GPU"; then
- nvidia-smi --query-gpu=index,name,driver_version,memory.used,memory.total,temperature.gpu,utilization.gpu \
- --format=csv,noheader 2>/dev/null \
- && echo "" \
- || echo "[FAIL] nvidia-smi query failed"
- # XID errors indicate hardware faults that will cause NCCL to abort.
- # Modern A100/H100 drivers log XIDs to dmesg but NOT to nvidia-smi -q,
- # so check both sources — verified on-hardware with A100 driver 580.126
- # where an injected XID 31 appeared in dmesg but was invisible to -q.
- XID_DMESG=$(dmesg 2>/dev/null | grep -E 'NVRM: Xid' | tail -5)
- XID_SMI=$(nvidia-smi -q 2>/dev/null | grep -E '^[[:space:]]*Xid' | head -5)
- if [ -n "$XID_DMESG" ] || [ -n "$XID_SMI" ]; then
- echo "[FAIL] GPU XID ERRORS DETECTED (hardware fault — NCCL will abort):"
- [ -n "$XID_DMESG" ] && echo "$XID_DMESG"
- [ -n "$XID_SMI" ] && echo "$XID_SMI"
- else
- echo "[PASS] No GPU XID errors"
- fi
- # Only surface nonzero ECC counts. 'ECC Errors' section header and
- # 'Uncorrectable ... : 0' lines fire on every healthy GPU.
- ECC=$(nvidia-smi -q 2>/dev/null | awk '
- /Uncorrectable/ { if ($NF ~ /^[0-9]+$/ && $NF+0 > 0) print }
- ' | head -5)
- [ -n "$ECC" ] && echo "[FAIL] GPU uncorrectable ECC errors detected: $ECC" || echo "[PASS] No ECC errors"
- # Row-remap state — marginal GPU memory. Pending rows need a reset to finalize;
- # Failed means exceeded remap capacity (bad memory). Silent degrader that
- # default DCGM medium + memtest in some driver versions miss entirely.
- REMAP=$(nvidia-smi --query-remapped-rows=gpu_bus_id,remapped_rows.pending,remapped_rows.failure \
- --format=csv,noheader 2>/dev/null)
- if [ -n "$REMAP" ]; then
- PENDING_SUM=$(echo "$REMAP" | awk -F, '{gsub(/ /,""); s+=$2} END {print s+0}')
- FAILED_COUNT=$(echo "$REMAP" | awk -F, '{gsub(/ /,""); if ($3=="Yes" || $3=="1") c++} END {print c+0}')
- if [ "$FAILED_COUNT" -gt 0 ]; then
- echo "[FAIL] GPU row-remap FAILED on $FAILED_COUNT device(s) — bad memory, replace GPU"
- elif [ "$PENDING_SUM" -gt 0 ]; then
- echo "[FAIL] GPU row-remap PENDING ($PENDING_SUM row(s)) — marginal memory; reset/reboot to finalize"
- echo " If pending persists across reboots, firmware may be stuck — replace GPU"
- else
- echo "[PASS] GPU row-remap: no pending or failed rows"
- fi
- fi
- # DCGM health — complements XID/ECC above. Parse Fail/Warn verdicts only
- # (Pass is not authoritative on DCGM <= 3.3.9 due to memtest bug).
- if command -v dcgmi >/dev/null 2>&1; then
- DCGM_OUT=$(dcgmi health --check -j 2>/dev/null || dcgmi health --check 2>/dev/null || echo "")
- if echo "$DCGM_OUT" | grep -qiE '"overall_health"\s*:\s*"(Fail|Warn)"|HEALTH_RESULT_FAIL|HEALTH_RESULT_WARN|Health Monitor Report.*(Fail|Warn)'; then
- echo "[FAIL] DCGM health check reports Fail/Warn — inspect with 'dcgmi health --check'"
- fi
- fi
- # DCGM nvvs log presence — HyperPod deep-health-check writes here.
- if [ -d /var/log/nvidia-dcgm ]; then
- NVVS_LATEST=$(find /var/log/nvidia-dcgm -maxdepth 1 -name 'nvvs*.log' -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -1 | awk '{print $2}')
- if [ -n "$NVVS_LATEST" ]; then
- if tail -n 200 "$NVVS_LATEST" 2>/dev/null | grep -qiE 'row ?remap.*(pending|fail)|FAIL: |Error: '; then
- echo "[FAIL] DCGM nvvs log contains failure / row-remap signals: $NVVS_LATEST"
- fi
- fi
- fi
- # NVLink — important for p4d/p5 multi-GPU NCCL bandwidth.
- # Output format across driver versions:
- # - 'Link N: X GB/s' (active, driver 470+)
- # - 'Link N: Active' (older drivers)
- # - 'error'/'fail'/'inactive' keywords when degraded
- NVLINK=$(nvidia-smi nvlink --status 2>/dev/null | head -200)
- if echo "$NVLINK" | grep -qiE "error|fail|inactive"; then
- echo "[FAIL] NVLink errors/inactive links detected (replace node):"
- echo "$NVLINK" | grep -iE "error|fail|inactive"
- else
- ACTIVE_COUNT=$(echo "$NVLINK" | grep -cE "Link [0-9]+:[[:space:]]+([0-9]+ GB/s|Active)" || true)
- if [ "${ACTIVE_COUNT:-0}" -gt 0 ]; then
- echo "[PASS] NVLink: $ACTIVE_COUNT active link(s)"
- else
- echo "[INFO] NVLink not available (expected on single-GPU or non-NVLink instances)"
- fi
- fi
- # GPU P2P topology — critical for intra-node NCCL AllReduce performance
- echo ""
- echo "--- GPU P2P Topology (nvidia-smi topo) ---"
- nvidia-smi topo -m 2>/dev/null | head -25 | while IFS= read -r line; do
- if echo "$line" | grep -qiE "NV[0-9]|NVLink"; then
- echo " [PASS] $line"
- elif echo "$line" | grep -qiE "PIX|PXB|PHB|SOC"; then
- echo " [WARN] $line <- PCIe path (slower than NVLink)"
- else
- echo " [INFO] $line"
- fi
- done
- # PCI ACS — intercepts GPU Direct P2P → 10-50x slower intra-node AllReduce or hang
- echo ""
- echo "--- PCI ACS (Access Control Services) ---"
- if command -v lspci &>/dev/null; then
- ACS_ENABLED=$(lspci -vvv 2>/dev/null | grep -A20 "PCI bridge\|Root Port\|Upstream Port" \
- | grep "ACSCtl:" | { grep -c "SrcValid+" 2>/dev/null; true; })
- if [ "$ACS_ENABLED" -gt 0 ] 2>/dev/null; then
- echo "[FAIL] ACS enabled on $ACS_ENABLED PCI bridge(s) — GPU Direct P2P blocked!"
- echo " Symptom: 'NCCL WARN P2P not supported between dev X and dev Y'"
- echo " Impact: 10-50x slower intra-node AllReduce"
- else
- echo "[PASS] ACS not enabled on PCI bridges — GPU Direct P2P unobstructed"
- fi
- else
- echo "[INFO] lspci not available — install pciutils to check ACS"
- fi
- IOMMU=$(dmesg 2>/dev/null | grep -iE "iommu.*enabled|dmar.*enabled" | head -2 || \
- grep -oE "intel_iommu=[^ ]+|iommu=[^ ]+" /proc/cmdline 2>/dev/null | head -1 || echo "")
- if [ -n "$IOMMU" ]; then
- echo "[WARN] IOMMU may be enabled: $IOMMU"
- echo " On baremetal: disable VT-d/IOMMU in BIOS for best GPU Direct P2P"
- echo " In VMs: normal — use ATS on network adapters"
- else
- echo "[PASS] IOMMU: not detected as enabled"
- fi
- [ "${NCCL_P2P_DISABLE:-0}" = "1" ] && \
- echo "[WARN] NCCL_P2P_DISABLE=1 set — workaround active, performance degraded" || true
- # nvidia-peermem — GPU Direct RDMA to NIC (required for EFA↔GPU on p4d/p5)
- echo ""
- echo "--- nvidia-peermem (GPU Direct RDMA) ---"
- if lsmod 2>/dev/null | grep -q "nvidia_peermem\|nv_peer_mem"; then
- echo "[PASS] nvidia-peermem loaded — GPU Direct RDMA to EFA/NIC enabled"
- else
- # Kernel 5.12+ uses DMA-BUF instead of nvidia-peermem.
- KVER_MAJOR=$(uname -r | cut -d. -f1)
- KVER_MINOR=$(uname -r | cut -d. -f2)
- if [ "$KVER_MAJOR" -gt 5 ] || { [ "$KVER_MAJOR" -eq 5 ] && [ "$KVER_MINOR" -ge 12 ]; } 2>/dev/null; then
- echo "[INFO] nvidia-peermem not loaded; kernel $(uname -r) supports DMA-BUF (auto-detected)"
- else
- echo "[WARN] nvidia-peermem NOT loaded — EFA↔GPU copies go through CPU"
- fi
- fi
- else
- if command -v nvidia-smi &>/dev/null; then
- echo "[INFO] nvidia-smi installed but no GPU devices visible — likely a CPU-only node (controller/login)"
- else
- echo "[INFO] nvidia-smi not found — CPU-only node or GPU driver not installed"
- fi
- fi
- echo ""
- echo "--- EFA ---"
- if lsmod 2>/dev/null | grep -q "^efa "; then
- EFA_MOD_VER=$(modinfo efa 2>/dev/null | grep "^version:" | awk '{print $2}' || echo "unknown")
- echo "[PASS] EFA kernel module loaded (version: ${EFA_MOD_VER})"
- else
- EFA_DEVS=$(ls /dev/infiniband/uverbs* 2>/dev/null || echo "")
- EFA_IFACES=$(ip -br link show 2>/dev/null | grep -cE "^efa" || echo 0)
- if [ -n "$EFA_DEVS" ] || [ "$EFA_IFACES" -gt 0 ] 2>/dev/null; then
- echo "[FAIL] EFA devices present but kernel module NOT loaded — NCCL EFA will fail"
- else
- echo "[INFO] EFA kernel module not loaded (expected on non-EFA instances)"
- fi
- fi
- FI_CMD=""
- command -v fi_info &>/dev/null && FI_CMD="fi_info"
- [ -z "$FI_CMD" ] && [ -f /opt/amazon/efa/bin/fi_info ] && FI_CMD="/opt/amazon/efa/bin/fi_info"
- if [ -n "$FI_CMD" ]; then
- EFA_OUTPUT=$($FI_CMD -p efa 2>&1)
- if echo "$EFA_OUTPUT" | grep -q "provider: efa"; then
- EFA_COUNT=$(echo "$EFA_OUTPUT" | { grep -c "provider: efa" 2>/dev/null; true; })
- echo "[PASS] EFA provider available: $EFA_COUNT interface(s)"
- echo "$EFA_OUTPUT" | grep "device:" | head -5
- # Validate EFA count against expected per-instance-type counts. A subset
- # of NICs silently failing to attach is a top NCCL failure mode (training
- # runs at reduced bandwidth with no error). Counts per AWS EC2 docs.
- IMDS_TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
- Dismissed after review: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
- Line 1,743 on GitHub
- -H "X-aws-ec2-metadata-token-ttl-seconds: 60" --connect-timeout 2 2>/dev/null || echo "")
- if [ -n "$IMDS_TOKEN" ]; then
- INST_TYPE=$(curl -s -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
- http://169.254.169.254/latest/meta-data/instance-type --connect-timeout 2 2>/dev/null || echo "")
- # Counts only included where AWS publishes them in the EC2 EFA
- # docs (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html).
- # For other types the doc lists bandwidth but not card count, so we
- # skip the check rather than guess.
- case "$INST_TYPE" in
- p5.48xlarge|p5e.48xlarge) EXPECTED_EFA=32 ;;
- p5en.48xlarge) EXPECTED_EFA=16 ;;
- p4d.24xlarge) EXPECTED_EFA=4 ;;
- p6-b200.48xlarge) EXPECTED_EFA=8 ;;
- p6-b300.48xlarge) EXPECTED_EFA=17 ;;
- p6e-gb200.36xlarge) EXPECTED_EFA=17 ;;
- *) EXPECTED_EFA=0 ;;
- esac
- if [ "$EXPECTED_EFA" -gt 0 ] 2>/dev/null; then
- if [ "$EFA_COUNT" -lt "$EXPECTED_EFA" ] 2>/dev/null; then
- echo "[FAIL] EFA count mismatch on ${INST_TYPE}: got ${EFA_COUNT}, expected ${EXPECTED_EFA}"
- echo " A subset of NICs failed to attach — NCCL will run at reduced bandwidth"
- else
- echo "[PASS] EFA count matches ${INST_TYPE} expected value (${EXPECTED_EFA})"
- fi
- else
- echo "[INFO] EFA count validation skipped — no expected value for ${INST_TYPE:-unknown}"
- fi
- fi
- else
- # Determine whether EFA is expected — absence on non-EFA instance types
- # (t3, c5, controllers) is normal, not a failure.
- INST_TYPE_CHECK="${INST_TYPE:-}"
- case "$INST_TYPE_CHECK" in
- p4d.*|p4de.*|p5.*|p5e.*|p5en.*|p6*|trn1.*|trn2.*)
- echo "[FAIL] EFA provider NOT available on ${INST_TYPE_CHECK}"
- echo " fi_info -p efa returned no results"
- echo " Required for NCCL on this instance type — training will fall back to TCP (very slow)"
- ;;
- *)
- echo "[INFO] EFA provider not available — expected on non-EFA instance type (${INST_TYPE_CHECK:-unknown})"
- ;;
- esac
- fi
- TCP_COUNT=$($FI_CMD -p tcp 2>/dev/null | { grep -c "provider: tcp" 2>/dev/null; true; })
- LF_VER=$($FI_CMD --version 2>&1 | grep libfabric | sed 's/.*: //' | head -1)
- echo " libfabric: ${LF_VER:-unknown} | TCP fallback endpoints: $TCP_COUNT"
- else
- echo "[INFO] fi_info not found — EFA tools not installed (OK for non-EFA instances)"
- fi
- [ -f /opt/amazon/efa_installed_packages ] && \
- grep "# EFA installer version" /opt/amazon/efa_installed_packages | head -1 \
- || echo "[INFO] /opt/amazon/efa_installed_packages not found"
- # aws-ofi-nccl — bridges NCCL and EFA, required for GPU training on EFA instances
- OFI_LIB=$(find /opt/amazon/ofi-nccl /usr/local/lib /usr/lib /opt/aws-ofi-nccl/lib \
- -maxdepth 4 -name "libnccl-net.so" 2>/dev/null | head -1)
- NCCL_NET_PLUGIN_ENV="${NCCL_NET_PLUGIN:-}"
- if [ -n "$NCCL_NET_PLUGIN_ENV" ]; then
- [ -f "$NCCL_NET_PLUGIN_ENV" ] && \
- echo "[PASS] NCCL_NET_PLUGIN=$NCCL_NET_PLUGIN_ENV (file exists)" || \
- echo "[FAIL] NCCL_NET_PLUGIN=$NCCL_NET_PLUGIN_ENV — FILE NOT FOUND! NCCL EFA will fail"
- elif [ -n "$OFI_LIB" ]; then
- echo "[PASS] aws-ofi-nccl plugin: $OFI_LIB"
- else
- # FAIL only if FI_PROVIDER=efa is set; otherwise informational.
- [ "${FI_PROVIDER:-}" = "efa" ] && \
- echo "[FAIL] FI_PROVIDER=efa but aws-ofi-nccl plugin not found — NCCL EFA will fail" || \
- echo "[INFO] aws-ofi-nccl not found (required for EFA+NCCL; not needed for CPU-only)"
- fi
- # Hugepages — improve EFA/RDMA memory registration performance
- HP_2M=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 2>/dev/null || echo 0)
- HP_1G=$(cat /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages 2>/dev/null || echo 0)
- if [ "$HP_2M" -gt 0 ] 2>/dev/null; then
- HP_FREE=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages 2>/dev/null || echo 0)
- echo "[PASS] 2MB hugepages: ${HP_2M} total, ${HP_FREE} free"
- [ "$HP_FREE" -eq 0 ] && echo "[WARN] All hugepages in use — RDMA may have reduced performance"
- elif [ "$HP_1G" -gt 0 ] 2>/dev/null; then
- echo "[PASS] 1GB hugepages: ${HP_1G} allocated"
- else
- echo "[INFO] No hugepages configured (set vm.nr_hugepages=512 for optimal EFA RDMA)"
- fi
- echo ""
- echo "--- NCCL ---"
- NCCL_LIB=$(find /usr/local/cuda*/lib* /usr/lib /opt/nccl/lib 2>/dev/null \
- -maxdepth 4 -name "libnccl.so*" 2>/dev/null | head -3)
- if [ -n "$NCCL_LIB" ]; then
- echo "[PASS] NCCL library found:"
- echo "$NCCL_LIB" | while read -r l; do echo " $l"; done
- else
- echo "[INFO] NCCL library not found (install NCCL for distributed GPU training)"
- fi
- NCCL_HDR=$(find /usr/local/cuda*/include /usr/include /opt/nccl/include 2>/dev/null \
- -maxdepth 3 -name "nccl.h" 2>/dev/null | head -1)
- if [ -n "$NCCL_HDR" ]; then
- NCCL_VER=$(grep -E "NCCL_MAJOR|NCCL_MINOR|NCCL_PATCH" "$NCCL_HDR" 2>/dev/null \
- | awk '{print $3}' | tr '\n' '.' | sed 's/\.$//')
- [ -n "$NCCL_VER" ] && echo " NCCL version: $NCCL_VER"
- fi
- CUDA_DRV=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | head -1 | tr -d ' ' || echo "")
- if [ -n "$CUDA_DRV" ] && [ -n "$NCCL_VER" ]; then
- DRV_MAJOR=$(echo "$CUDA_DRV" | cut -d. -f1)
- NCCL_MAJOR=$(echo "$NCCL_VER" | cut -d. -f1)
- NCCL_MINOR=$(echo "$NCCL_VER" | cut -d. -f2)
- # NCCL 2.20+ requires CUDA driver >= 525
- if { [ "$NCCL_MAJOR" -gt 2 ] || { [ "$NCCL_MAJOR" -eq 2 ] && [ "${NCCL_MINOR:-0}" -ge 20 ]; }; } && [ "$DRV_MAJOR" -gt 0 ] && [ "$DRV_MAJOR" -lt 525 ] 2>/dev/null; then
- echo "[WARN] NCCL $NCCL_VER may require CUDA driver >= 525; found $CUDA_DRV"
- echo " Symptom: 'NCCL function not found' on mixed-version nodes"
- fi
- fi
- echo ""
- echo "--- Network Interfaces ---"
- ip -br addr show 2>/dev/null | while IFS= read -r line; do
- IFACE=$(echo "$line" | awk '{print $1}')
- STATE=$(echo "$line" | awk '{print $2}')
- ADDR=$(echo "$line" | awk '{print $3}')
- if echo "$IFACE" | grep -q "^lo"; then TYPE="loopback"
- elif echo "$IFACE" | grep -qE "^efa|^rdma"; then TYPE="EFA device"
- elif echo "$IFACE" | grep -qE "^ib[0-9]"; then TYPE="InfiniBand"
- elif echo "$IFACE" | grep -qE "^eth|^ens|^enp|^en[0-9]"; then TYPE="VPC ENI"
- elif echo "$IFACE" | grep -qE "^docker|^br-|^veth"; then TYPE="container bridge"
- else TYPE="other"; fi
- printf " %-18s %-8s %-20s (%s)\n" "$IFACE" "$STATE" "${ADDR:--}" "$TYPE"
- done
- echo ""
- echo "--- MTU ---"
- ip -br link show 2>/dev/null | grep -v "^lo" | while IFS= read -r line; do
- IFACE=$(echo "$line" | awk '{print $1}')
- MTU=$(ip link show "$IFACE" 2>/dev/null | grep -o "mtu [0-9]*" | awk '{print $2}')
- [ -z "$MTU" ] && continue
- if echo "$IFACE" | grep -qE "docker|br-|veth"; then echo " [INFO] $IFACE: MTU=$MTU (container bridge — OK)"
- elif [ "$MTU" -ge 9000 ] 2>/dev/null; then echo " [PASS] $IFACE: MTU=$MTU (jumbo frames — optimal for EFA)"
- else echo " [WARN] $IFACE: MTU=$MTU — expected 9001 for EFA/RDMA (fragmentation risk for large tensors)"; fi
- done
- echo ""
- echo "--- Memory & Limits ---"
- free -h
- echo ""
- SHM_SIZE=$(df -BG /dev/shm 2>/dev/null | tail -1 | awk '{print $2}' | tr -d 'G')
- SHM_FS=$(df -T /dev/shm 2>/dev/null | tail -1 | awk '{print $2}' || echo "unknown")
- if [ -n "$SHM_SIZE" ] && [ "$SHM_SIZE" -ge 1 ] 2>/dev/null; then
- echo "[PASS] /dev/shm: ${SHM_SIZE}GB (fs: ${SHM_FS})"
- [ "$SHM_SIZE" -lt 4 ] 2>/dev/null && \
- echo "[WARN] /dev/shm ${SHM_SIZE}GB < 4GB — consider 4GB+ for large model training"
- else
- echo "[FAIL] /dev/shm: ${SHM_SIZE:-0}GB — NCCL needs ≥1GB (K8s default=64MB)"
- echo " Symptom: 'failed to extend /dev/shm/nccl-*' or Bus error"
- fi
- [ "$SHM_FS" != "tmpfs" ] && [ "$SHM_FS" != "unknown" ] && \
- echo "[WARN] /dev/shm fs type: $SHM_FS (expected tmpfs)"
- MEMLOCK=$(ulimit -l 2>/dev/null || echo "unknown")
- if [ "$MEMLOCK" = "0" ]; then
- echo "[FAIL] memlock=0 — InfiniBand/EFA RDMA memory registration WILL FAIL"
- echo " Symptom: 'NCCL WARN Call to ibv_reg_mr failed'"
- elif [ -n "$MEMLOCK" ] && [ "$MEMLOCK" != "unlimited" ] && [ "$MEMLOCK" -ge 8388608 ] 2>/dev/null; then
- echo "[PASS] memlock=${MEMLOCK}KB (≥8GB — OK)"
- elif [ "$MEMLOCK" = "unlimited" ]; then
- echo "[INFO] memlock=unlimited (OK for RDMA; see stack check below for libc quirk)"
- else
- echo "[INFO] memlock=${MEMLOCK}KB"
- fi
- # Stack size — GNU libc quirk: when memlock=unlimited, thread stack is reduced to 2MB.
- # NCCL topology graph search (especially MNNVL on 256+ nodes) needs 8MB+ stack.
- STACK=$(ulimit -s 2>/dev/null || echo "unknown")
- if [ "$MEMLOCK" = "unlimited" ] && [ "$STACK" = "unlimited" ]; then
- echo "[WARN] memlock=unlimited + stack=unlimited — GNU libc reduces NCCL thread stack to 2MB"
- echo " NCCL MNNVL/large topology graph search needs 8MB+ and will fail"
- elif [ "$STACK" = "unlimited" ]; then
- echo "[PASS] stack=unlimited (memlock is bounded, so libc quirk does not apply)"
- elif [ "$STACK" != "unknown" ] && [ "$STACK" -lt 4096 ] 2>/dev/null; then
- echo "[FAIL] stack=${STACK}KB — too small for NCCL topology search (need ≥4096KB)"
- else
- echo "[PASS] stack=${STACK:-unknown}KB"
- fi
- # systemd RemoveIPC — deletes NCCL shm files when session ends (Slurm nodes)
- # Strip comment lines first; many distros ship logind.conf with `#RemoveIPC=yes`
- # as the documented default, which would false-WARN on a substring match.
- if [ -f /etc/systemd/logind.conf ]; then
- REMOVEIPC=$(grep -v '^[[:space:]]*#' /etc/systemd/logind.conf 2>/dev/null \
- | grep -i "RemoveIPC" | tail -1 || echo "")
- if [ -z "$REMOVEIPC" ]; then
- echo "[WARN] RemoveIPC unset in /etc/systemd/logind.conf — defaults to 'yes' on RHEL/Amazon Linux"
- echo " Symptom: 'unlink shared memory /dev/shm/nccl-* failed: No such file'"
- elif echo "$REMOVEIPC" | grep -qi "yes\|true\|1"; then
- echo "[WARN] systemd RemoveIPC=yes — NCCL shm files will be deleted at session end"
- echo " Symptom: 'unlink shared memory /dev/shm/nccl-* failed: No such file'"
- else
- echo "[PASS] systemd RemoveIPC=no — NCCL shm files will not be deleted"
- fi
- fi
- # cuMem NUMA (NCCL 2.23+)
- NUMA_NODES=$(ls /sys/devices/system/node/ 2>/dev/null | { grep -c "^node[0-9]" 2>/dev/null; true; })
- if [ "$NUMA_NODES" -gt 0 ] 2>/dev/null; then
- echo "[PASS] NUMA topology: $NUMA_NODES node(s) visible (cuMem host alloc OK)"
- else
- echo "[WARN] NUMA topology not visible — cuMem host allocations may fail"
- fi
- echo ""
- echo "--- NCCL RAS ---"
- # RAS port is configurable via NCCL_RAS_ADDR. Probe whatever is in the
- # training process's environment; skip if no candidate is found rather than
- # hard-coding a port that may not match every NCCL build.
- NC_CMD=$(command -v nc 2>/dev/null || command -v ncat 2>/dev/null || echo "")
- if [ -n "$NC_CMD" ]; then
- RAS_PID=$(pgrep -f "python|torchrun|mpirun" 2>/dev/null | head -1)
- RAS_ADDR=""
- if [ -n "$RAS_PID" ] && [ -r "/proc/$RAS_PID/environ" ]; then
- RAS_ADDR=$(tr '\0' '\n' < "/proc/$RAS_PID/environ" 2>/dev/null \
- | awk -F= '/^NCCL_RAS_ADDR=/{print $2}' | head -1)
- fi
- RAS_HOST="${RAS_ADDR%:*}"; RAS_PORT="${RAS_ADDR##*:}"
- if [ -n "$RAS_PORT" ] && [ "$RAS_PORT" != "$RAS_ADDR" ]; then
- RAS=$(echo "status" | timeout 3 $NC_CMD -w 2 "${RAS_HOST:-localhost}" "$RAS_PORT" 2>/dev/null || echo "")
- if [ -n "$RAS" ]; then
- echo "[PASS] NCCL RAS responding at ${RAS_HOST:-localhost}:${RAS_PORT}:"
- echo "$RAS" | head -10
- else
- echo "[INFO] NCCL RAS port ${RAS_PORT} not responding — training job may not be using RAS, or RAS is disabled (NCCL_RAS_ENABLE=0)"
- fi
- else
- echo "[INFO] NCCL_RAS_ADDR not set in any training process — skipping RAS probe (set NCCL_RAS_ADDR=<host>:<port> and re-run during training to enable)"
- fi
- else
- echo "[INFO] nc/ncat not found — cannot probe NCCL RAS"
- fi
- echo ""
- echo "--- Active Training Processes ---"
- PROCS=$(ps aux 2>/dev/null | grep -E "python|torchrun|mpirun|nccl_test" | grep -v grep | head -10)
- if [ -n "$PROCS" ]; then
- echo "$PROCS"
- else
- echo "[INFO] No active training processes"
- fi
- echo ""
- echo "--- Recent Hardware Errors (dmesg) ---"
- DMESG=$(dmesg 2>/dev/null | grep -iE "xid|nvrm|efa|ib_core|rdma|correctable|uncorrectable|acs|iommu" \
- | tail -20 || echo "")
- if [ -n "$DMESG" ]; then
- echo "$DMESG"
- else
- echo "[PASS] No hardware errors in dmesg"
- fi
- # iptables / nftables — host-level firewall rules that block NCCL
- echo "--- Host Firewall (iptables/nftables) ---"
- IPT_DROP=0
- if command -v iptables &>/dev/null; then
- IPT_DROP=$(iptables -L -n 2>/dev/null | grep -cE "DROP|REJECT" || echo 0)
- if [ "$IPT_DROP" -gt 0 ] 2>/dev/null; then
- echo "[WARN] iptables has $IPT_DROP DROP/REJECT rules — may block NCCL traffic"
- iptables -L -n 2>/dev/null | grep -E "DROP|REJECT" | head -5
- echo " Verify NCCL ports (29400-29500, RDMA) are not blocked"
- else
- echo "[PASS] iptables: no DROP/REJECT rules"
- fi
- elif command -v nft &>/dev/null; then
- NFT_DROP=$(nft list ruleset 2>/dev/null | grep -cE "drop|reject" || echo 0)
- if [ "$NFT_DROP" -gt 0 ] 2>/dev/null; then
- echo "[WARN] nftables has $NFT_DROP drop/reject rules — may block NCCL traffic"
- else
- echo "[PASS] nftables: no drop/reject rules"
- fi
- else
- echo "[INFO] iptables/nftables not found"
- fi
- echo ""
- echo "--- Stale NCCL Shared Memory ---"
- STALE_SHM=$(ls /dev/shm/nccl-* 2>/dev/null || echo "")
- if [ -n "$STALE_SHM" ]; then
- STALE_COUNT=$(echo "$STALE_SHM" | wc -l)
- echo "[WARN] $STALE_COUNT stale NCCL shared memory file(s) found:"
- echo "$STALE_SHM" | head -5
- echo " From a previous training run — may cause 'file exists' errors"
- else
- echo "[PASS] No stale NCCL shared memory files"
- fi
- echo ""
- # EFA Latency Check (fi_ping) — catches degraded EFA ports (straggler #1 cause)
- echo "--- EFA Latency (fi_ping self-test) ---"
- FI_PING_CMD=""
- command -v fi_ping &>/dev/null && FI_PING_CMD="fi_ping"
- [ -z "$FI_PING_CMD" ] && [ -f /opt/amazon/efa/bin/fi_ping ] && FI_PING_CMD="/opt/amazon/efa/bin/fi_ping"
- if [ -n "$FI_PING_CMD" ]; then
- # Self-ping on loopback — tests EFA stack without needing a second node
- # A degraded EFA port shows high latency (>20us) even on self-ping
- # Validate FI_PING_CMD is a known safe EFA binary path (not user-controlled)
- if [[ ! "$FI_PING_CMD" =~ ^(/opt/amazon/efa/bin/fi_ping|fi_ping)$ ]]; then
- echo "[SKIP] fi_ping path not recognised: $FI_PING_CMD"
- else
- # Try EFA provider first; if it succeeds, the result reflects EFA. If
- # EFA isn't reachable on loopback (some kernels), fall back to TCP — but
- # label it explicitly so a TCP latency isn't reported as if it were EFA.
- # nosemgrep: ai.ai-best-practices.hooks-dns-exfiltration.hooks-dns-exfiltration.hooks-dns-exfiltration-generic -- FI_PING_CMD validated to known EFA binary path above; targets loopback 127.0.0.1
- PING_OUT=$($FI_PING_CMD -p efa -I 10 127.0.0.1 2>/dev/null || echo "")
- PROVIDER="efa"
- if [ -z "$PING_OUT" ]; then
- # nosemgrep: ai.ai-best-practices.hooks-dns-exfiltration.hooks-dns-exfiltration.hooks-dns-exfiltration-generic -- FI_PING_CMD validated above; loopback only
- PING_OUT=$($FI_PING_CMD -p tcp -I 10 127.0.0.1 2>/dev/null || echo "")
- PROVIDER="tcp"
- fi
- if [ -n "$PING_OUT" ]; then
- LATENCY=$(echo "$PING_OUT" | grep -oE "[0-9]+\.[0-9]+ us" | tail -1 || echo "")
- LAT_VAL=$(echo "$LATENCY" | grep -oE "[0-9]+" | head -1 || echo 0)
- if [ "$PROVIDER" = "tcp" ]; then
- # TCP loopback latency does NOT reflect EFA path health; an EFA
- # straggler will not be visible here. Surface as INFO, not PASS/WARN.
- echo "[INFO] fi_ping fell back to provider=tcp (EFA loopback unreachable) — latency=${LATENCY:-?}; this does NOT measure EFA path health"
- echo " For EFA latency, run fi_ping/fi_pingpong between two real nodes (not loopback) — see references/performance-testing.md"
- elif [ -n "$LATENCY" ]; then
- if [ "$LAT_VAL" -gt 20 ] 2>/dev/null; then
- echo "[WARN] fi_ping latency (provider=efa): $LATENCY (>20us — EFA port may be degraded; normal is 1-5us)"
- echo " Impact: straggler AllReduce, training much slower than expected"
- echo " Action: drain this node and replace via HyperPod API"
- else
- echo "[PASS] fi_ping latency (provider=efa): $LATENCY"
- fi
- else
- echo "[INFO] fi_ping (provider=efa) ran but no latency value extracted"
- echo "$PING_OUT" | tail -3
- fi
- else
- echo "[INFO] fi_ping self-test skipped (no EFA/TCP provider reachable)"
- fi
- fi
- else
- echo "[INFO] fi_ping not found (install EFA tools for latency testing)"
- fi
- echo ""
- echo "=== END NODE DIAGNOSTICS ==="
- NODE_SCRIPT
- # Strategy for 100s of nodes:
- # 1. Resolve all Running compute nodes via HyperPod API (paginated)
- # 2. Sample --sample-nodes (default 3) for SSM hardware checks
- # 3. Each SSM call has a 60s timeout
- # 4. Results show per-node summary; failures are highlighted
- # 5. This check does NOT increment ISSUES_FOUND (hardware checks are advisory)
- # unless a critical hardware fault is detected (XID errors, EFA fail on GPU instance)
- check_node_hardware_via_ssm() {
- header "Check 8: Node Hardware Checks (via SSM — runs ON cluster nodes)"
- info "Resolving cluster nodes for SSM..."
- if ! resolve_cluster_nodes_for_ssm; then
- info "Could not resolve cluster nodes via HyperPod API"
- info " (DescribeCluster needs sagemaker:DescribeCluster on this cluster)"
- info " To check a specific node: --node <INSTANCE_ID>"
- return
- fi
- if [[ ${#SSM_NODES[@]} -eq 0 ]]; then
- info "No Running compute nodes found in cluster"
- return
- fi
- local total_nodes
- total_nodes="${#SSM_NODES[@]}"
- info "Sampling $total_nodes node(s) for hardware checks (use --sample-nodes N for more)"
- info "Cluster ID: $SSM_CLUSTER_ID"
- local node_pass=0 node_warn=0 node_fail=0
- for entry in "${SSM_NODES[@]}"; do
- local instance_id group_name
- instance_id=$(echo "$entry" | awk '{print $1}')
- group_name=$(echo "$entry" | awk '{print $2}')
- local target="sagemaker-cluster:${SSM_CLUSTER_ID}_${group_name}-${instance_id}"
- section "Node: $instance_id ($group_name)"
- info " SSM target: $target"
- info " Connecting (timeout 60s)..."
- local output
- output=$(_ssm_run "$instance_id" "$group_name" "$SSM_CLUSTER_ID" "$_NODE_DIAG_SCRIPT")
- # Detect SSM transport failures. Letting error text fall through as
- # diagnostic output produces a misleading "0 [PASS]" finding.
- if [[ -z "$output" ]] || echo "$output" | grep -qiE "SessionManagerPlugin|error.*session|not authorized|AccessDenied|Could not connect|^Cannot perform start session|EOF$|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable|TargetNotConnected"; then
- warn " SSM connection failed for $instance_id → references/operations.md § 3 SSM target format (HyperPod)"
- node_warn=$((node_warn + 1))
- continue
- fi
- echo "$output"
- local passes fails
- passes=$(echo "$output" | { grep -c "\[PASS\]" 2>/dev/null; true; })
- fails=$(echo "$output" | { grep -c "\[FAIL\]" 2>/dev/null; true; })
- # Non-GPU / non-EFA nodes (controllers, logins, CPU families) sampled
- # as a fallback. Flag as SKIP rather than PASS — a PASS on a node
- # without GPU/EFA is meaningless for NCCL.
- local is_non_gpu=false
- if echo "$output" | grep -qE "^\[INFO\].*(CPU-only node|non-EFA instance|no GPU devices visible|nvidia-smi not found)"; then
- if ! echo "$output" | grep -qE "^\[PASS\] EFA provider available|^\[PASS\] GPU row-remap"; then
- is_non_gpu=true
- fi
- fi
- if [[ "$fails" -gt 0 ]]; then
- error " Node $instance_id: $fails hardware issue(s) detected — see above"
- node_fail=$((node_fail + 1))
- # XID errors or EFA fail on a GPU instance = cluster-level issue
- if echo "$output" | grep -q "\[FAIL\] GPU XID"; then
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "XID errors on GPU hardware ($instance_id) → references/operations.md § 8 NCCL-specific remediations (Node reboot / replacement); hyperpod-node-debugger skill" "P0"
- elif echo "$output" | grep -q "\[FAIL\] EFA provider NOT"; then
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "EFA provider failure on $instance_id → references/debugging-guide.md § 6 EFA Configuration / § 13 EFA TCP Fallback" "P0"
- fi
- elif $is_non_gpu; then
- info " Node $instance_id: no GPU/EFA present — skipping (NCCL checks apply only to GPU/EFA compute nodes)"
- node_warn=$((node_warn + 1))
- else
- success " Node $instance_id: hardware checks passed ($passes [PASS])"
- node_pass=$((node_pass + 1))
- fi
- done
- echo ""
- info "Hardware check summary: $node_pass PASS | $node_warn UNREACHABLE | $node_fail FAIL"
- if [[ "$node_fail" -gt 0 ]]; then
- warn " $node_fail node(s) have hardware issues — check above for details"
- warn " For ALL nodes: re-run with --sample-nodes <total> to check every node"
- fi
- if [[ "$node_warn" -gt 0 ]]; then
- warn " $node_warn node(s) unreachable via SSM — verify SSM agent and IAM permissions"
- fi
- }
- # CloudWatch covers ALL nodes at once without per-node SSM calls.
- # This runs for EKS when K8S_CONNECTED=false (can't use kubectl logs).
- check_cloudwatch_nccl_logs() {
- header "Check 6b: NCCL Pattern Analysis via CloudWatch"
- local cluster_arn cluster_id
- cluster_arn=$(aws sagemaker describe-cluster \
- --cluster-name "$CLUSTER_NAME" --region "$REGION" \
- --query 'ClusterArn' --output text 2>/dev/null || echo "")
- cluster_id=$(echo "$cluster_arn" | awk -F'/' '{print $NF}')
- if [[ -z "$cluster_id" || "$cluster_id" == "None" ]]; then
- info "Cluster ID unavailable — skipping CloudWatch log analysis"
- return
- fi
- local log_group="/aws/sagemaker/Clusters/${CLUSTER_NAME}/${cluster_id}"
- info "CloudWatch log group: $log_group"
- local lg_exists
- lg_exists=$(aws logs describe-log-groups \
- --log-group-name-prefix "$log_group" --region "$REGION" \
- --query 'logGroups[0].logGroupName' --output text 2>&1) || {
- if echo "$lg_exists" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
- warn "Permission denied: logs:DescribeLogGroups — check IAM policy"
- fi
- lg_exists=""
- }
- if [[ -z "$lg_exists" || "$lg_exists" == "None" ]]; then
- info "CloudWatch log group not found — CloudWatch agent may not be configured"
- info " Enable the CloudWatch agent in the cluster's lifecycle script (see operations.md § 4)"
- return
- fi
- local start_time=$(( ($(date +%s) - 7200) * 1000 ))
- local patterns=(
- "NCCL WARN" "Watchdog timeout" "Timeout waiting for"
- "fi_getinfo failed" "unhandled system error" "nccl error"
- "Connection refused" "NCCL_OFI_RDMA"
- )
- local found_any=false
- for pattern in "${patterns[@]}"; do
- local matches
- matches=$(aws logs filter-log-events \
- --log-group-name "$log_group" \
- --filter-pattern "\"$pattern\"" \
- --start-time "$start_time" \
- --region "$REGION" \
- --query 'events[*].{t:timestamp,s:logStreamName,m:message}' \
- --output json 2>/dev/null || echo "[]")
- local count
- count=$(echo "$matches" | python3 -c \
- "import sys,json; print(len(json.load(sys.stdin)))" 2>/dev/null || echo 0)
- if [[ "$count" -gt 0 ]]; then
- error "CloudWatch: '$pattern' found $count time(s) in last 2h:"
- echo "$matches" | python3 -c "
- import sys,json,datetime
- events=json.load(sys.stdin)[:5]
- for e in events:
- ts=datetime.datetime.utcfromtimestamp(e['t']//1000).strftime('%H:%M:%S')
- stream=e['s'][:30]
- msg=e['m'][:120].strip()
- print(f' [{ts}] {stream}: {msg}')
- " 2>/dev/null
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "CloudWatch pattern '$pattern' found ${count} time(s) → references/error-patterns-quick-ref.md" "P1"
- found_any=true
- fi
- done
- $found_any || success "No NCCL error patterns in CloudWatch logs (last 2h)"
- }
- # Slurm: run command on head node via SSM (start-session, not send-command)
- run_slurm_cmd_via_ssm() {
- local cmd="$1"
- if ! resolve_cluster_nodes_for_ssm; then
- return 1
- fi
- # Paginate list-cluster-nodes so controller/head nodes in the last page
- # of a large cluster aren't missed.
- local all_nodes
- all_nodes=$(sagemaker_list_paginated list-cluster-nodes ClusterNodeSummaries)
- local head_entry
- head_entry=$(echo "$all_nodes" | python3 -c "
- import sys,json
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- for n in nodes:
- g=n.get('InstanceGroupName','').lower()
- if any(x in g for x in ['controller','head','master']):
- print(n['InstanceId'] + ' ' + n['InstanceGroupName'])
- break
- else:
- for n in nodes:
- if n.get('InstanceStatus',{}).get('Status') == 'Running':
- print(n['InstanceId'] + ' ' + n['InstanceGroupName'])
- break
- " 2>/dev/null || echo "")
- [[ -z "$head_entry" ]] && return 1
- local iid grp
- iid=$(echo "$head_entry" | awk '{print $1}')
- grp=$(echo "$head_entry" | awk '{print $2}')
- _ssm_run "$iid" "$grp" "$SSM_CLUSTER_ID" "$cmd"
- }
- check_slurm_nodes() {
- header "Check 2 [Slurm]: Node States"
- local sinfo_output=""
- if command -v sinfo &>/dev/null; then
- sinfo_output=$(sinfo -o "%N %T %30E" --noheader 2>/dev/null || echo "")
- else
- sinfo_output=$(run_slurm_cmd_via_ssm "sinfo -o '%N %T %30E' --noheader" || echo "")
- fi
- # Treat SSM transport errors as retrieval failures, not as healthy state.
- # Without this, "Cannot perform start session: EOF" is non-empty and falls
- # through the empty-check below → grep finds no "down" → misleading [PASS].
- if echo "$sinfo_output" | grep -qiE "^(Cannot perform start session|SessionManagerPlugin is not found)|EOF$|TargetNotConnected|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable|AccessDenied|UnauthorizedOperation|not authorized to perform"; then
- warn "Could not retrieve Slurm node states — SSM transient error after retries"
- info " Rerun the diagnostic; if persistent, delegate to hyperpod-ssm skill for manual probe."
- return
- fi
- if [[ -z "$sinfo_output" ]]; then
- warn "Could not retrieve Slurm node states"
- return
- fi
- local down drained
- down=$(echo "$sinfo_output" | grep -E "\bdown\b|\bdraining\b" | awk '{print $1}' || echo "")
- drained=$(echo "$sinfo_output" | grep -E "\bdrained\b" | awk '{print $1}' || echo "")
- if [[ -z "$down" && -z "$drained" ]]; then
- success "All Slurm nodes: UP/IDLE/ALLOC — no NCCL-impacting states"
- else
- if [[ -n "$down" ]]; then
- error "DOWN/DRAINING nodes: $down"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "Slurm nodes DOWN/DRAINING: $down → references/operations.md § 7 Slurm — NCCL-specific operations" "P1"
- while IFS= read -r node; do
- [[ -z "$node" ]] && continue
- done <<< "$(echo "$down" | tr ',' '\n')"
- fi
- [[ -n "$drained" ]] && warn "DRAINED nodes (not available): $drained"
- fi
- section "Slurm Job Queue"
- local q=""
- if command -v squeue &>/dev/null; then
- q=$(squeue -o "%i %j %T %R %N" --noheader 2>/dev/null || echo "")
- fi
- if [[ -z "$q" ]]; then
- q=$(run_slurm_cmd_via_ssm "squeue -o '%i %j %T %R %N' --noheader" 2>/dev/null || echo "")
- fi
- # Same SSM-error detection as above — without this, the error string is
- # parsed as a job list and produces false "stuck" rows.
- if echo "$q" | grep -qiE "^(Cannot perform start session|SessionManagerPlugin is not found)|EOF$|TargetNotConnected|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable|AccessDenied|UnauthorizedOperation|not authorized to perform"; then
- warn "Could not retrieve Slurm job queue — SSM transient error after retries"
- q=""
- fi
- if [[ -z "$q" ]]; then
- info "No jobs in queue"
- else
- local stuck
- stuck=$(echo "$q" | grep -E "PENDING|COMPLETING" | head -10 || echo "")
- if [[ -n "$stuck" ]]; then
- warn "Stuck PENDING/COMPLETING jobs:"
- echo "$stuck"
- ISSUES_FOUND=$((ISSUES_FOUND+1))
- add_issue_detail "Stuck PENDING/COMPLETING Slurm jobs → references/operations.md § 7 Slurm — NCCL-specific operations" "P1"
- else
- success "No stuck jobs in queue"
- fi
- info "Queue (top 10):"; echo "$q" | head -10
- fi
- }
- check_slurm_nccl_logs() {
- header "Check 6 [Slurm]: NCCL Log Pattern Analysis"
- check_cloudwatch_nccl_logs
- }
- check_slurm_nccl_env() {
- header "Check 7 [Slurm]: NCCL Environment Variable Audit (via SSM)"
- local env_check
- env_check=$(run_slurm_cmd_via_ssm \
- "{ cat /etc/profile.d/nccl.sh /opt/ml/config/nccl.conf /etc/slurm/prolog.d/*.sh 2>/dev/null; env; } \
- | grep -E '^(NCCL_|FI_|MASTER_)' | sort -u | head -30 || echo '(none)'" \
- 2>/dev/null || echo "")
- # If SSM returned a transport error, don't interpret it as the controller's
- # env output — that produces false "FI_PROVIDER=efa not set" warnings.
- if echo "$env_check" | grep -qiE "^(Cannot perform start session|SessionManagerPlugin is not found)|EOF$|TargetNotConnected|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable|AccessDenied|UnauthorizedOperation|not authorized to perform"; then
- warn "Could not retrieve NCCL env vars from controller — SSM transient error after retries"
- info " Rerun the diagnostic; if persistent, delegate to hyperpod-ssm skill."
- return
- fi
- if [[ -n "$env_check" && "$env_check" != "(none)" ]]; then
- info "NCCL/EFA env vars on head node:"
- echo "$env_check" | while IFS= read -r line; do info " $line"; done
- local warn_count=0
- if echo "$env_check" | grep -q "NCCL_DEBUG=INFO"; then
- warn "NCCL_DEBUG=INFO detected — verbose logging adds runtime overhead. Set NCCL_DEBUG=WARN for production."
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "NCCL_DEBUG=INFO in Slurm env (set NCCL_DEBUG=WARN in production) → references/operations.md § 5 NCCL environment variable reference" "P1"
- warn_count=$((warn_count + 1))
- fi
- if echo "$env_check" | grep -q "NCCL_DEBUG=TRACE"; then
- warn "NCCL_DEBUG=TRACE detected — TRACE prints replayable trace info on every NCCL call (large overhead and verbose logs). Set NCCL_DEBUG=WARN immediately."
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "NCCL_DEBUG=TRACE in Slurm env (set NCCL_DEBUG=WARN immediately) → references/operations.md § 5 NCCL environment variable reference" "P0"
- warn_count=$((warn_count + 1))
- fi
- if ! echo "$env_check" | grep -q "FI_PROVIDER=efa"; then
- warn "FI_PROVIDER=efa not set — EFA may not be used for NCCL transport"
- warn_count=$((warn_count + 1))
- fi
- if ! echo "$env_check" | grep -q "NCCL_SOCKET_IFNAME"; then
- warn "NCCL_SOCKET_IFNAME not set — NCCL may pick wrong interface. Recommend: ^lo,docker,efa,veth"
- warn_count=$((warn_count + 1))
- fi
- if [[ "$warn_count" -eq 0 ]]; then
- success "System-level NCCL env vars look correct"
- fi
- else
- info "No NCCL env vars found in system config on head node"
- info " (Expected — NCCL vars are typically set in job scripts, not system-wide)"
- fi
- }
- check_slurm_controller_health() {
- # Slurm controller health — retry up to 3× before declaring it down, because
- # SSM cold-start / session-service EOF errors are common on the first call.
- header "Check 0 [Slurm]: Controller Health"
- local ping_result=""
- for _ in 1 2 3; do
- ping_result=$(run_slurm_cmd_via_ssm "scontrol ping 2>/dev/null" || echo "")
- [[ -n "$ping_result" ]] && echo "$ping_result" | grep -qi "is UP\|slurmctld.*UP" && break
- sleep 3
- done
- if echo "$ping_result" | grep -qi "is UP\|slurmctld.*UP"; then
- success "slurmctld is responsive"
- elif echo "$ping_result" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized to perform"; then
- # IAM denial ≠ Slurm failure. Reporting "slurmctld down" would be wrong
- # and would send the customer down a Slurm-rescue path for an IAM issue.
- warn "Could not check slurmctld — caller lacks ssm:StartSession on this cluster"
- info " Grant ssm:StartSession on the HyperPod cluster ARN and rerun."
- elif echo "$ping_result" | grep -qiE "Cannot perform start session|SessionManager|EOF$|TargetNotConnected|ConnectTimeout|ServiceError|ThrottlingException|RequestLimitExceeded|InternalFailure|ServiceUnavailable"; then
- # Transport-level SSM errors — not a Slurm failure. Downgrade to WARN.
- warn "Could not reach controller via SSM (transient): $(echo "$ping_result" | head -1)"
- info " Rerun the diagnostic; if the error persists, delegate to hyperpod-ssm skill."
- elif [[ -n "$ping_result" ]]; then
- error "slurmctld not responding — all Slurm operations blocked"
- local _diag_line
- _diag_line="$(echo "$ping_result" | head -1)"
- info " Controller response: $_diag_line"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "slurmctld down on controller → references/operations.md § 7 Slurm — NCCL-specific operations" "P0"
- else
- info "Could not reach controller via SSM — slurmctld status unknown"
- fi
- local munge_result
- munge_result=$(run_slurm_cmd_via_ssm "systemctl is-active munge 2>/dev/null || echo munge_inactive" || echo "")
- if echo "$munge_result" | grep -q "^active"; then
- success "munge authentication service active"
- elif echo "$munge_result" | grep -q "munge_inactive"; then
- error "munge service inactive — Slurm auth will fail"
- ISSUES_FOUND=$((ISSUES_FOUND + 1))
- add_issue_detail "munge service inactive → references/operations.md § 7 Slurm — NCCL-specific operations" "P0"
- fi
- }
- run_slurm_checks() {
- check_slurm_controller_health
- check_cluster_health
- check_slurm_nodes
- check_cluster_events
- check_security_groups
- check_slurm_nccl_logs
- check_slurm_nccl_env
- check_node_hardware_via_ssm
- }
- print_summary() {
- header "NCCL Diagnostic Summary"
- echo ""
- echo -e " Cluster: ${BOLD}$CLUSTER_NAME${RESET}"
- echo -e " Region: ${BOLD}$REGION${RESET}"
- echo -e " Orchestrator: ${BOLD}$(printf '%s' "$ORCHESTRATOR" | tr '[:lower:]' '[:upper:]')${RESET}"
- [[ "$ORCHESTRATOR" == "eks" ]] && \
- echo -e " Namespace: ${BOLD}${NAMESPACE:-all}${RESET}"
- [[ -n "$JOB_NAME" ]] && echo -e " Job: ${BOLD}$JOB_NAME${RESET}"
- [[ -n "$NODE_ID" ]] && echo -e " Node: ${BOLD}$NODE_ID${RESET}"
- echo -e " Mode: ${BOLD}READ-ONLY${RESET} (no changes applied)"
- echo ""
- echo -e " ┌──────────────────────────────────┐"
- echo -e " │ Issues Found: ${RED}${BOLD}$ISSUES_FOUND${RESET} │"
- echo -e " └──────────────────────────────────┘"
- if [[ ${#ISSUE_DETAILS[@]} -gt 0 ]]; then
- echo ""
- echo " Issue Details (prioritized):"
- for priority in P0 P1 P2; do
- local has_items=false
- for detail in "${ISSUE_DETAILS[@]}"; do
- if [[ "$detail" == "${priority}|"* ]]; then
- if ! $has_items; then
- case "$priority" in
- P0) echo -e " ${RED}${BOLD}[$priority — Fix Immediately]${RESET}" ;;
- P1) echo -e " ${YELLOW}${BOLD}[$priority — Fix Soon]${RESET}" ;;
- P2) echo -e " ${BOLD}[$priority — Advisory]${RESET}" ;;
- esac
- has_items=true
- fi
- echo " → ${detail#*|}"
- fi
- done
- done
- fi
- echo ""
- if [[ "$ISSUES_FOUND" -eq 0 ]]; then
- success "No actionable NCCL issues detected — cluster looks healthy"
- echo ""
- info "If training is still hanging, check:"
- echo " 1. CloudWatch: aws logs filter-log-events --log-group-name /aws/sagemaker/Clusters/$CLUSTER_NAME/..."
- echo " 2. Version check: hyperpod-version-checker skill"
- echo " 3. Full diagnostics: hyperpod-issue-report skill"
- else
- warn "$ISSUES_FOUND issue(s) found — see the Issue Details list above."
- warn "Each issue line includes a reference pointer (→ references/<file>.md § <section>)."
- warn "The hyperpod-nccl skill will read these findings, look up the matching section,"
- warn "and guide you through remediation. This script does not modify cluster state."
- fi
- echo ""
- echo -e "${BOLD}References:${RESET}"
- echo " Debugging guide: references/debugging-guide.md"
- echo " Operations: references/operations.md"
- echo " Performance test: references/performance-testing.md"
- echo ""
- }
- main() {
- header "NCCL Diagnostic — SageMaker HyperPod (read-only)"
- detect_orchestrator
- echo -e " Cluster: ${BOLD}$CLUSTER_NAME${RESET}"
- echo -e " Region: ${BOLD}$REGION${RESET}"
- echo -e " Orchestrator: ${BOLD}$(printf '%s' "$ORCHESTRATOR" | tr '[:lower:]' '[:upper:]')${RESET}"
- [[ "$ORCHESTRATOR" == "eks" ]] && echo -e " Namespace: ${BOLD}${NAMESPACE:-all}${RESET}"
- info "READ-ONLY DIAGNOSTIC — no cluster state will be modified."
- info "This script collects signals only. The hyperpod-nccl skill interprets findings"
- info "and looks up remediation in references/*.md."
- echo ""
- check_prerequisites
- if [[ "$ORCHESTRATOR" == "slurm" ]]; then
- info "Running Slurm NCCL diagnostics..."
- run_slurm_checks
- else
- info "Running EKS NCCL diagnostics..."
- check_cluster_health
- check_cluster_events
- check_security_groups
- if $K8S_CONNECTED; then
- check_k8s_nodes
- check_efa_k8s
- check_pod_status
- check_nccl_infra_prereqs
- analyze_nccl_logs
- check_nccl_env_vars
- check_network_policies
- else
- warn "K8s checks skipped (2, 2b, 5, 5b, 6, 7, 9) — kubectl not authenticated"
- # CloudWatch analysis doesn't need kubectl.
- check_cloudwatch_nccl_logs
- fi
- check_node_hardware_via_ssm
- fi
- print_summary
- # Exit 1 only on P0/P1 findings; P2 are informational.
- local _critical=0
- for _issue in "${ISSUE_DETAILS[@]:-}"; do
- [[ -z "$_issue" ]] && continue
- case "${_issue%%|*}" in P0|P1) _critical=$((_critical+1)) ;; esac
- done
- [[ "$_critical" -eq 0 ]] && exit 0 || exit 1
- }
- main "$@"
plugins/sagemaker-ai/skills/hyperpod-node-debugger/scripts/check-node-reachability.sh · Bundled script · 389 lines · 1 finding marked
- #!/usr/bin/env bash
- # check-node-reachability.sh
- #
- # Diagnose EFA reachability and inter-node communication health on a single
- # HyperPod node. Run this on each node via the hyperpod-ssm skill.
- #
- # Usage (via ssm-exec.sh):
- # ssm-exec.sh --target <TARGET> --upload scripts/check-node-reachability.sh /tmp/check-node-reachability.sh
- # ssm-exec.sh --target <TARGET> 'bash /tmp/check-node-reachability.sh'
- #
- # Usage (direct on node):
- # bash check-node-reachability.sh [--json] [--no-color]
- #
- # Exit codes:
- # 0 — all critical checks passed
- # 1 — one or more critical checks failed
- set -euo pipefail
- # Note: this script runs ON the node (via SSM), so aws CLI may not be present.
- # Only python3 is checked here; other tools are checked individually per section.
- JSON_MODE=false
- USE_COLOR=true
- usage() {
- cat <<EOF
- Usage: bash check-node-reachability.sh [--json] [--no-color]
- Read-only on-node diagnostic for EFA reachability and inter-node communication
- health. Must be executed on a HyperPod compute node (typically via the
- hyperpod-ssm skill). Checks EFA interfaces, /dev/infiniband devices, GPU
- count and Neuron device count against the expected counts for the node's
- instance type.
- Options:
- --json Emit findings as JSON instead of human-readable output.
- --no-color Disable ANSI colors.
- -h, --help Show this message.
- Exit codes:
- 0 All critical checks passed.
- 1 One or more critical checks failed.
- EOF
- }
- while [[ $# -gt 0 ]]; do
- case "$1" in
- --json) JSON_MODE=true; shift ;;
- --no-color) USE_COLOR=false; shift ;;
- -h|--help) usage; exit 0 ;;
- *) echo "Unknown argument: $1" >&2; usage >&2; exit 1 ;;
- esac
- done
- # Colors — auto-disable when stdout isn't a TTY.
- if ! [ -t 1 ] || [ "${TERM:-}" = "dumb" ]; then
- USE_COLOR=false
- fi
- if "$USE_COLOR"; then
- RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
- BOLD='\033[1m'; NC='\033[0m'
- else
- RED=''; GREEN=''; YELLOW=''; BOLD=''; NC=''
- fi
- HOSTNAME=$(hostname 2>/dev/null || echo "unknown")
- TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
- CRITICAL_FAILURES=0
- declare -A RESULTS # associative array: check_name → pass|fail|warn|skip
- pass() { RESULTS["$1"]="pass"; [[ "$JSON_MODE" == false ]] && echo -e " ${GREEN}[PASS]${NC} $1${2:+ — $2}"; }
- fail() { RESULTS["$1"]="fail"; CRITICAL_FAILURES=$((CRITICAL_FAILURES+1)); \
- [[ "$JSON_MODE" == false ]] && echo -e " ${RED}[FAIL]${NC} $1${2:+ — $2}"; }
- warn() { RESULTS["$1"]="warn"; [[ "$JSON_MODE" == false ]] && echo -e " ${YELLOW}[WARN]${NC} $1${2:+ — $2}"; }
- skip() { RESULTS["$1"]="skip"; [[ "$JSON_MODE" == false ]] && echo -e " [SKIP] $1${2:+ — $2}"; }
- info() { [[ "$JSON_MODE" == false ]] && echo -e " $1"; }
- if [[ "$JSON_MODE" == false ]]; then
- echo ""
- echo -e "${BOLD}=== HyperPod Node EFA Reachability Check ===${NC}"
- echo -e "Host: ${BOLD}${HOSTNAME}${NC}"
- echo -e "Timestamp: ${TIMESTAMP}"
- echo ""
- fi
- if [[ "$JSON_MODE" == false ]]; then echo -e "${BOLD}--- EFA Kernel Module ---${NC}"; fi
- EFA_MODULE=$(lsmod 2>/dev/null | grep -E '^efa\b' | awk '{print $1}' || true)
- if [[ -n "$EFA_MODULE" ]]; then
- EFA_MODULE_VER=$(modinfo efa 2>/dev/null | grep -E '^version:' | awk '{print $2}' || echo "unknown")
- pass "efa_kernel_module" "loaded (version: ${EFA_MODULE_VER})"
- else
- # Read-only invariant: detect only, never `sudo modprobe efa` — loading kernel
- # modules mutates node state, which the hyperpod-ssm skill's approval flow owns.
- fail "efa_kernel_module" "not loaded — see references/node-diagnostics-detail.md § A (EFA / Security Group)"
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- EFA Devices ---${NC}"; fi
- # shellcheck disable=SC2010 # /dev/ entries are kernel-named, safe to ls|grep
- EFA_DEVICES=$(ls /dev/infiniband/ 2>/dev/null | grep -E 'rdma_cm|uverbs|efa' || true)
- if [[ -n "$EFA_DEVICES" ]]; then
- pass "efa_devices_present" "found in /dev/infiniband/: $(echo "$EFA_DEVICES" | tr '\n' ' ')"
- else
- fail "efa_devices_present" "/dev/infiniband/ is empty or missing — EFA hardware not detected"
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- libfabric EFA Provider ---${NC}"; fi
- if command -v fi_info &>/dev/null; then
- # If the previous section found no EFA hardware, fi_info failing is expected —
- # don't emit [FAIL] on top of the hardware [FAIL], which would double-count and
- # conflate "libfabric can't see EFA" with "node has no EFA at all".
- if [[ -z "$EFA_DEVICES" ]]; then
- skip "fi_info_efa_provider" "no EFA devices detected upstream — see efa_devices_present"
- else
- FI_EXIT=0
- FI_OUTPUT=$(fi_info -p efa 2>&1) || FI_EXIT=$?
- if echo "$FI_OUTPUT" | grep -q "provider: efa"; then
- EFA_PROVIDER_COUNT=$(echo "$FI_OUTPUT" | { grep -c "provider: efa" 2>/dev/null; true; })
- pass "fi_info_efa_provider" "EFA provider found (${EFA_PROVIDER_COUNT} endpoint(s))"
- info "$(echo "$FI_OUTPUT" | grep -E 'provider:|fabric:|domain:|version:' | head -8 | sed 's/^/ /')"
- else
- fail "fi_info_efa_provider" "fi_info -p efa returned no EFA provider (exit code ${FI_EXIT}) — libfabric cannot enumerate EFA devices. See references/node-diagnostics-detail.md § A (EFA / Security Group)"
- info "fi_info output: ${FI_OUTPUT:0:200}"
- fi
- fi
- else
- warn "fi_info_efa_provider" "fi_info not found — install libfabric to run this check (fi_info comes with EFA installer)"
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- EFA Network Interfaces ---${NC}"; fi
- # EFA interfaces typically appear as eth0/ens* for primary + rdmaX or efa* for EFA devices
- # EFA ifaces on p5/p5en use regular kernel names (ens*) — filter by driver via ethtool
- # rather than by name pattern (the old 'rdma|efa' name grep misses ens* on p5).
- EFA_IFACES=""
- if command -v ethtool &>/dev/null; then
- while IFS= read -r iface; do
- [[ -z "$iface" ]] && continue
- DRIVER=$(ethtool -i "$iface" 2>/dev/null | awk -F': ' '/^driver:/{print $2}')
- if [[ "$DRIVER" == "efa" ]]; then
- EFA_IFACES+="${iface}"$'\n'
- fi
- done < <(ip -o link show 2>/dev/null | awk -F': ' '{print $2}' | awk -F'@' '{print $1}' | grep -v '^lo$')
- fi
- # Fallback to name-based detection for older kernels / containers without ethtool
- if [[ -z "$EFA_IFACES" ]]; then
- EFA_IFACES=$(ip link show 2>/dev/null | grep -E 'rdma|efa' | awk -F': ' '{print $2}' | tr -d '@' || true)
- fi
- REGULAR_IFACES=$(ip link show 2>/dev/null | grep -E 'state UP' | awk -F': ' '{print $2}' | tr -d '@' || true)
- if [[ -n "$EFA_IFACES" ]]; then
- pass "efa_interfaces_up" "EFA interfaces found: $(echo "$EFA_IFACES" | tr '\n' ' ')"
- while IFS= read -r iface; do
- [[ -z "$iface" ]] && continue
- IP=$(ip addr show "$iface" 2>/dev/null | grep 'inet ' | awk '{print $2}' || true)
- if [[ -n "$IP" ]]; then
- info " $iface → $IP"
- else
- warn "efa_interface_ip_${iface}" "interface $iface has no IP address — check DHCP/subnet config"
- fi
- done <<< "$EFA_IFACES"
- else
- info "No EFA interfaces detected (by driver or name)"
- if [[ -n "$REGULAR_IFACES" ]]; then
- skip "efa_interfaces_up" "no separate EFA interface — primary interfaces: $(echo "$REGULAR_IFACES" | tr '\n' ' ' | head -c 80)"
- else
- warn "efa_interfaces_up" "no UP network interfaces found"
- fi
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- EFA Installation ---${NC}"; fi
- EFA_VER_FILE="/opt/amazon/efa_installed_packages"
- if [[ -f "$EFA_VER_FILE" ]]; then
- # Format is "EFA installer version: 1.30.0" — grab only the version token.
- EFA_VER=$(grep -iE '^EFA installer version' "$EFA_VER_FILE" 2>/dev/null \
- | head -1 \
- | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?' \
- | head -1 || echo "")
- if [[ -z "$EFA_VER" ]]; then
- warn "efa_installer_present" "EFA installer file present but version line not parsed"
- else
- pass "efa_installer_present" "EFA installer version: ${EFA_VER}"
- fi
- else
- warn "efa_installer_present" "EFA installer marker not found at ${EFA_VER_FILE} — EFA may not be installed via standard method"
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- NCCL / OFI Configuration ---${NC}"; fi
- NCCL_VARS=("FI_PROVIDER" "FI_EFA_USE_DEVICE_RDMA" "NCCL_SOCKET_IFNAME" "NCCL_ALGO" "LD_LIBRARY_PATH")
- ANY_NCCL_SET=false
- for var in "${NCCL_VARS[@]}"; do
- val="${!var:-}"
- if [[ -n "$val" ]]; then
- info " ${var}=${val}"
- ANY_NCCL_SET=true
- fi
- done
- if "$ANY_NCCL_SET"; then
- FI_PROVIDER_VAL="${FI_PROVIDER:-}"
- if [[ -n "$FI_PROVIDER_VAL" && "$FI_PROVIDER_VAL" != "efa" ]]; then
- warn "nccl_fi_provider" "FI_PROVIDER=${FI_PROVIDER_VAL} — for EFA workloads this should be 'efa'"
- elif [[ "$FI_PROVIDER_VAL" == "efa" ]]; then
- pass "nccl_fi_provider" "FI_PROVIDER=efa"
- fi
- else
- skip "nccl_env_vars" "no NCCL/OFI env vars set in current shell — may be set in job launcher environment"
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- AWS OFI NCCL Plugin ---${NC}"; fi
- OFI_LIB=$(find /opt/amazon/efa /opt/aws-ofi-nccl /usr/local/lib /usr/lib \
- -name "libnccl-net.so*" -o -name "aws-ofi-nccl.so*" 2>/dev/null | head -1 || true)
- if [[ -n "$OFI_LIB" ]]; then
- pass "aws_ofi_nccl_plugin" "found: ${OFI_LIB}"
- else
- if [[ -f "$EFA_VER_FILE" ]] && grep -q "ofi\|OFI" "$EFA_VER_FILE" 2>/dev/null; then
- pass "aws_ofi_nccl_plugin" "referenced in ${EFA_VER_FILE}"
- else
- warn "aws_ofi_nccl_plugin" "libnccl-net.so not found — required for EFA-accelerated NCCL (distributed training)"
- fi
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- Instance Metadata Reachability ---${NC}"; fi
- IMDS_TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
- Note: Quoted as data in code: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
- Line 233 on GitHub
- -H "X-aws-ec2-metadata-token-ttl-seconds: 60" --connect-timeout 3 -m 5 2>/dev/null || true)
- if [[ -n "$IMDS_TOKEN" ]]; then
- INSTANCE_TYPE=$(curl -s -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
- http://169.254.169.254/latest/meta-data/instance-type --connect-timeout 3 -m 5 2>/dev/null || echo "unknown")
- LOCAL_IP=$(curl -s -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
- http://169.254.169.254/latest/meta-data/local-ipv4 --connect-timeout 3 -m 5 2>/dev/null || echo "unknown")
- pass "imds_reachable" "instance-type=${INSTANCE_TYPE}, local-ipv4=${LOCAL_IP}"
- # Static list of EFA-capable families; unknown types fall through to the
- # EC2 API check. aws CLI may not be present on-node, so the static path
- # covers the common case.
- case "$INSTANCE_TYPE" in
- p4de*|p4d*|p5en*|p5e*|p5*|p6*|trn1*|trn2*|inf2*|g5.48xlarge|g6e.48xlarge|g6.48xlarge|hpc6a*|hpc6id*|hpc7a*|hpc7g*|dl1*|dl2q*)
- pass "efa_capable_instance" "${INSTANCE_TYPE} supports EFA" ;;
- *)
- if command -v aws &>/dev/null; then
- EFA_CHECK=$(aws ec2 describe-instance-types \
- --instance-types "${INSTANCE_TYPE}" \
- --query 'InstanceTypes[0].NetworkInfo.EfaSupported' \
- --output text 2>/dev/null || echo "unknown")
- if [[ "$EFA_CHECK" == "True" ]]; then
- pass "efa_capable_instance" "${INSTANCE_TYPE} supports EFA (verified via API)"
- elif [[ "$EFA_CHECK" == "False" ]]; then
- warn "efa_capable_instance" "${INSTANCE_TYPE} does NOT support EFA"
- else
- warn "efa_capable_instance" "${INSTANCE_TYPE} — could not verify EFA support"
- fi
- else
- warn "efa_capable_instance" "${INSTANCE_TYPE} — not in known EFA list; verify with: aws ec2 describe-instance-types --instance-types ${INSTANCE_TYPE} --query 'InstanceTypes[0].NetworkInfo.EfaSupported'"
- fi
- ;;
- esac
- # Multi-EFA validation — counts per EC2 instance-type documentation.
- # NOTE: EFA counts vary between instance families (p5en has fewer than p5/p5e).
- EXPECTED_EFA=0
- case "$INSTANCE_TYPE" in
- p5.48xlarge|p5e.48xlarge) EXPECTED_EFA=32 ;;
- p5en.48xlarge) EXPECTED_EFA=16 ;;
- p4d.24xlarge|p4de.24xlarge) EXPECTED_EFA=4 ;;
- trn1.32xlarge) EXPECTED_EFA=8 ;;
- trn2.48xlarge) EXPECTED_EFA=16 ;;
- # p6 family and newer: don't hardcode counts; discover via ethtool to avoid false FAILs.
- esac
- if [[ "$EXPECTED_EFA" -gt 0 ]]; then
- # Count actual EFA devices — avoid grep -c pattern that returns "0\n0" fallthrough.
- ACTUAL_EFA=$(find /dev/infiniband -maxdepth 1 -name 'uverbs*' 2>/dev/null | wc -l)
- [[ -z "$ACTUAL_EFA" ]] && ACTUAL_EFA=0
- if [[ "$ACTUAL_EFA" -ge "$EXPECTED_EFA" ]]; then
- pass "multi_efa_interfaces" "${ACTUAL_EFA}/${EXPECTED_EFA} EFA interfaces present for ${INSTANCE_TYPE}"
- elif [[ "$ACTUAL_EFA" -gt 0 ]]; then
- warn "multi_efa_interfaces" "only ${ACTUAL_EFA}/${EXPECTED_EFA} EFA interfaces — some may not be attached or driver issue"
- else
- fail "multi_efa_interfaces" "0/${EXPECTED_EFA} EFA interfaces on ${INSTANCE_TYPE} — EFA driver or attachment issue"
- fi
- fi
- else
- warn "imds_reachable" "IMDS not reachable. If running inside a container: check IMDSv2 HttpPutResponseHopLimit on the instance (default 1 is often too low for container networking — set to 2 or higher). Otherwise: verify the instance metadata service is enabled (HttpEndpoint != disabled) and that no local iptables / nftables rules block 169.254.169.254. Note: SGs do not filter link-local addresses."
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- Network Interface Statistics ---${NC}"; fi
- if command -v ip &>/dev/null; then
- IFACE_ERRORS=$(ip -s link show 2>/dev/null | awk '
- BEGIN { rx_err=0; tx_err=0; iface="" }
- /^[0-9]+:/ {
- if (iface != "" && (rx_err > 0 || tx_err > 0))
- print " " iface ": RX errors=" rx_err " TX errors=" tx_err
- iface=$2; gsub(/:$/, "", iface)
- rx_err=0; tx_err=0
- }
- /RX:/ { getline; rx_err=$3+0 }
- /TX:/ { getline; tx_err=$3+0 }
- END {
- if (iface != "" && (rx_err > 0 || tx_err > 0))
- print " " iface ": RX errors=" rx_err " TX errors=" tx_err
- }
- ' || true)
- if [[ -n "$IFACE_ERRORS" ]]; then
- warn "network_interface_errors" "interfaces with errors detected:"
- info "$IFACE_ERRORS"
- else
- pass "network_interface_errors" "no RX/TX errors on active interfaces"
- fi
- else
- skip "network_interface_errors" "ip command not available"
- fi
- if [[ "$JSON_MODE" == false ]]; then echo ""; echo -e "${BOLD}--- Neuron Devices (Trainium/Inferentia) ---${NC}"; fi
- if command -v neuron-ls &>/dev/null; then
- NEURON_OUTPUT=$(neuron-ls 2>&1 || true)
- NEURON_DEVICE_COUNT=$(echo "$NEURON_OUTPUT" | { grep -c "neuron_device" 2>/dev/null; true; })
- if [[ "$NEURON_DEVICE_COUNT" -gt 0 ]]; then
- pass "neuron_devices" "${NEURON_DEVICE_COUNT} Neuron device(s) detected"
- info "$(echo "$NEURON_OUTPUT" | head -10 | sed 's/^/ /')"
- else
- NEURON_MOD=$(lsmod 2>/dev/null | grep -E '^neuron' || true)
- if [[ -n "$NEURON_MOD" ]]; then
- warn "neuron_devices" "Neuron driver loaded but neuron-ls shows 0 devices → references/node-diagnostics-detail.md § G.2 (Trainium/Inferentia)"
- else
- fail "neuron_devices" "Neuron driver not loaded → references/node-diagnostics-detail.md § G.2 (Trainium/Inferentia)"
- fi
- fi
- elif ls /dev/neuron* &>/dev/null 2>&1; then
- NEURON_DEV_COUNT=$(find /dev -maxdepth 1 -name 'neuron*' 2>/dev/null | wc -l)
- NEURON_DEV_COUNT=${NEURON_DEV_COUNT:-0}
- warn "neuron_devices" "${NEURON_DEV_COUNT} /dev/neuron* device(s) found but neuron-ls not installed → references/node-diagnostics-detail.md § G.2 (Trainium/Inferentia)"
- else
- skip "neuron_devices" "not a Trainium/Inferentia instance (no Neuron devices)"
- fi
- if [[ "$JSON_MODE" == false ]]; then
- echo ""
- echo -e "${BOLD}--- Summary ---${NC}"
- TOTAL=${#RESULTS[@]}
- PASSED=$(printf '%s\n' "${RESULTS[@]}" | { grep -c "^pass$" 2>/dev/null; true; })
- WARNED=$(printf '%s\n' "${RESULTS[@]}" | { grep -c "^warn$" 2>/dev/null; true; })
- FAILED=$(printf '%s\n' "${RESULTS[@]}" | { grep -c "^fail$" 2>/dev/null; true; })
- SKIPPED=$(printf '%s\n' "${RESULTS[@]}" | { grep -c "^skip$" 2>/dev/null; true; })
- echo -e " Host: ${HOSTNAME}"
- echo -e " Checks: ${TOTAL} total | ${GREEN}${PASSED} passed${NC} | ${YELLOW}${WARNED} warnings${NC} | ${RED}${FAILED} failed${NC} | ${SKIPPED} skipped"
- if [[ $CRITICAL_FAILURES -eq 0 ]]; then
- echo -e "\n ${GREEN}${BOLD}Node EFA reachability checks PASSED.${NC}"
- echo " If inter-node communication still fails, verify security group rules with check-efa-sg.sh"
- echo " and compare EFA versions across nodes with the hyperpod-version-checker skill."
- else
- echo -e "\n ${RED}${BOLD}Node EFA reachability checks FAILED (${CRITICAL_FAILURES} critical issue(s)).${NC}"
- echo " See [FAIL] items above. Each finding ends with a pointer of the form"
- echo " '→ references/node-diagnostics-detail.md § <section>' — open that section"
- echo " for root cause and remediation. Remediation lives in references, not in scripts."
- fi
- echo ""
- else
- CHECKS_JSON=""
- for key in "${!RESULTS[@]}"; do
- val="${RESULTS[$key]}"
- CHECKS_JSON+="\"${key}\": \"${val}\","
- done
- CHECKS_JSON="${CHECKS_JSON%,}" # remove trailing comma
- cat <<EOF
- {
- "hostname": "${HOSTNAME}",
- "timestamp": "${TIMESTAMP}",
- "critical_failures": ${CRITICAL_FAILURES},
- "overall_pass": $([ $CRITICAL_FAILURES -eq 0 ] && echo true || echo false),
- "checks": { ${CHECKS_JSON} }
- }
- EOF
- fi
- exit "$([[ $CRITICAL_FAILURES -eq 0 ]] && echo 0 || echo 1)"
plugins/sagemaker-ai/skills/hyperpod-node-debugger/scripts/triage-cluster.sh · Bundled script · 1,258 lines · 1 finding marked
- #!/usr/bin/env bash
- # triage-cluster.sh — read-only HyperPod node triage.
- #
- # Collects signals to route node issues to the right reference section:
- # - Cluster status, orchestrator, NodeRecovery
- # - Cluster events (root-cause signal for provisioning failures)
- # - Per-node health (HyperPod + EKS labels, Slurm state)
- # - VPC / SG config
- # - SSM reachability to compute nodes (hardware checks)
- #
- # Read-only: never modifies cluster state, never prints remediation commands.
- # Each [FAIL] / added issue carries a pointer of the form
- # "... → references/node-diagnostics-detail.md § <section>"
- # which the hyperpod-node-debugger skill uses to look up remediation.
- #
- # Usage:
- # bash triage-cluster.sh --cluster <name-or-arn> --region <region>
- # bash triage-cluster.sh --cluster <name-or-arn> --region <region> --node <instance-id>
- #
- # Exit codes:
- # 0 No critical (P0/P1) issues; P2 informational findings are allowed.
- # 1 One or more critical issues, or a fatal prerequisite / cluster-not-found.
- # 2 Invalid argument.
- set -euo pipefail
- for cmd in aws python3; do
- command -v "$cmd" &>/dev/null || {
- echo "ERROR: '$cmd' is required but not found." >&2
- exit 1
- }
- done
- HAS_UNBUFFER=true
- if ! command -v unbuffer &>/dev/null; then
- HAS_UNBUFFER=false
- fi
- CLUSTER=""
- REGION="${AWS_DEFAULT_REGION:-}"
- TARGET_NODE=""
- USE_COLOR=true
- usage() {
- cat <<EOF
- Usage: $0 --cluster <name-or-arn> --region <region> [options]
- Options:
- --cluster <name-or-arn> HyperPod cluster name or ARN (required)
- --region <region> AWS region (required unless \$AWS_DEFAULT_REGION is set)
- --node <instance-id> Focus on a single instance ID
- --no-color Disable ANSI colors
- -h, --help This message
- Read-only diagnostic. Every [FAIL] line carries a pointer like
- "→ references/node-diagnostics-detail.md § <section>" which the
- hyperpod-node-debugger skill uses to look up remediation.
- EOF
- }
- while [[ $# -gt 0 ]]; do
- case "$1" in
- --cluster) [[ $# -lt 2 ]] && { echo "ERROR: --cluster needs a value"; exit 2; }
- [[ ! "$2" =~ ^(arn:aws[a-z-]*:sagemaker:[a-z0-9-]+:[0-9]{12}:cluster/[a-z0-9]{12}|[a-zA-Z0-9]([-a-zA-Z0-9]{0,62}))$ ]] && { echo "ERROR: --cluster must be a valid HyperPod cluster name or ARN (got '$2')"; exit 2; }
- CLUSTER="$2"; shift 2 ;;
- --region) [[ $# -lt 2 ]] && { echo "ERROR: --region needs a value"; exit 2; }
- [[ ! "$2" =~ ^[a-z]{2}(-[a-z]+){1,2}-[0-9]+$ ]] && { echo "ERROR: --region must be a valid AWS region (got '$2')"; exit 2; }
- REGION="$2"; shift 2 ;;
- --node) [[ $# -lt 2 ]] && { echo "ERROR: --node needs a value"; exit 2; }
- [[ ! "$2" =~ ^i-[0-9a-f]{8,17}$ ]] && { echo "ERROR: --node must be an EC2 instance ID (i-xxxxxxxx...)"; exit 2; }
- TARGET_NODE="$2"; shift 2 ;;
- --no-color) USE_COLOR=false; shift ;;
- -h|--help) usage; exit 0 ;;
- *) echo "Unknown argument: $1"; usage; exit 2 ;;
- esac
- done
- [[ -z "$CLUSTER" ]] && {
- echo "Usage: $0 --cluster <name-or-arn> --region <region> [--node <instance-id>]"
- exit 1
- }
- if [[ -z "$REGION" ]]; then
- echo "ERROR: --region is required (or set AWS_DEFAULT_REGION before running)." >&2
- exit 2
- fi
- _CREDS=$(aws sts get-caller-identity --output json 2>&1) || {
- echo "ERROR: AWS credentials not configured or expired."
- echo "$_CREDS"
- echo ""
- echo "→ references/node-diagnostics-detail.md § K (Node Access via SSM) for credential setup"
- exit 1
- }
- # Auto-disable colors when stdout is not a TTY (agent-piped / redirected).
- if ! [ -t 1 ] || [ "${TERM:-}" = "dumb" ]; then
- USE_COLOR=false
- fi
- if "$USE_COLOR"; then
- RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'
- CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
- else
- RED=''; GREEN=''; YELLOW=''; CYAN=''; BOLD=''; NC=''
- fi
- section() { echo ""; echo -e "${BOLD}${CYAN}════════════════════════════════════════════════════════${NC}"; echo -e "${BOLD}${CYAN} $1${NC}"; echo -e "${BOLD}${CYAN}════════════════════════════════════════════════════════${NC}"; }
- ok() { echo -e " ${GREEN}[PASS]${NC} $1"; }
- warn() { echo -e " ${YELLOW}[WARN]${NC} $1"; }
- bad() { echo -e " ${RED}[FAIL]${NC} $1"; }
- info() { echo -e " ${BOLD}[INFO]${NC} $1"; }
- ISSUES_FOUND=()
- add_issue() {
- local priority="${2:-P1}"
- ISSUES_FOUND+=("${priority}|$1")
- }
- aws_check_perms() {
- local result="$1" api_name="$2"
- if echo "$result" | grep -qiE "AccessDenied|UnauthorizedOperation|not authorized|AuthorizationError"; then
- warn "Permission denied: $api_name — results may be incomplete"
- add_issue "Missing IAM permission for $api_name → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P1"
- return 0
- fi
- return 1
- }
- _TEMP_FILES=()
- cleanup_temp() {
- [[ ${#_TEMP_FILES[@]} -gt 0 ]] && rm -f "${_TEMP_FILES[@]}" 2>/dev/null || true
- }
- trap cleanup_temp EXIT
- # Run a shell command on a HyperPod node via SSM.
- #
- # HyperPod uses a SageMaker-managed instance fleet, so `aws ssm send-command`
- # with a bare instance-id is not supported. The supported path is
- # `aws ssm start-session` with target `sagemaker-cluster:<cluster-id>_<group>-<iid>`
- # and document `AWS-StartNonInteractiveCommand`.
- #
- # Usage: ssm_run_on_node <instance-id> <instance-group-name> "<shell command>"
- # Returns remote stdout. start-session does not propagate the remote exit code.
- ssm_run_on_node() {
- local iid="$1" grp="$2" cmd="$3"
- [[ -z "$iid" || -z "$grp" || -z "$cmd" ]] && return 1
- [[ ! "$iid" =~ ^i-[0-9a-f]{8,17}$ ]] && return 1
- [[ -z "${CLUSTER_ID:-}" ]] && return 1
- [[ ! "$grp" =~ ^[A-Za-z0-9._-]+$ ]] && return 1
- if [[ "${HAS_UNBUFFER:-true}" != "true" ]]; then
- echo " [SKIP] on-node SSM probe skipped — install 'unbuffer' (expect package) to enable" >&2
- return 1
- fi
- local target="sagemaker-cluster:${CLUSTER_ID}_${grp}-${iid}"
- local tmp; tmp=$(mktemp 2>/dev/null) || return 1
- chmod 600 "$tmp" 2>/dev/null || true
- _TEMP_FILES+=("$tmp")
- # Embed the command as base64 because AWS-StartNonInteractiveCommand
- # collapses newlines in a single command element.
- local cmd_b64
- cmd_b64=$(printf '%s' "$cmd" | base64 | tr -d '\n') || return 1
- local remote="bash -c \"echo $cmd_b64 | base64 -d | bash\""
- Note: Quoted as data in code: Decodes base64 and runs the result in a shell
- Line 164 on GitHub
- python3 -c "import json,sys; print(json.dumps({'command':[sys.argv[1]]}))" "$remote" > "$tmp" || return 1
- local attempt=0 out rc
- while (( attempt < 5 )); do
- out=$(unbuffer timeout 180 aws ssm start-session \
- --target "$target" \
- --document-name AWS-StartNonInteractiveCommand \
- --parameters "file://$tmp" \
- --region "$REGION" 2>&1)
- rc=$?
- # SSM sometimes returns rc=0 with a transport error baked into stdout —
- # retry those (EOF, SessionManagerPlugin not found, i/o timeout).
- if (( rc == 0 )) && ! echo "$out" | grep -qiE "Cannot perform start session|EOF$|SessionManagerPlugin is not found|ERROR: Unable to|i/o timeout"; then
- # Strip SSM session banners and the echoed base64 command line.
- echo "$out" | grep -vE '^(Starting session with SessionId:|Exiting session with sessionId:|\s*$)' \
- | grep -vE "^(bash -c \"echo [A-Za-z0-9+/=]+ \| base64 -d \| bash\"|echo '[A-Za-z0-9+/=]+'|[A-Za-z0-9+/=]{40,}={0,2})[[:space:]]*\|?[[:space:]]*base64?[[:space:]]*-?d?[[:space:]]*\|?[[:space:]]*bash\"?\$" || true
- return 0
- fi
- if echo "$out" | grep -qiE "ThrottlingException|RequestLimitExceeded|InternalFailure|InternalError|ServiceUnavailable|TooManyUpdates|Cannot perform start session|EOF$|SessionManagerPlugin is not found|i/o timeout"; then
- attempt=$((attempt + 1))
- sleep $((attempt * 3))
- continue
- fi
- # Non-transient error; surface stderr so callers can diagnose.
- echo "$out" >&2
- return 1
- done
- return 1
- }
- echo ""
- echo -e "${CYAN}${BOLD}HyperPod Node Triage — READ-ONLY${NC}"
- echo -e "${CYAN} No cluster state will be modified. Each issue line below includes a${NC}"
- echo -e "${CYAN} pointer to references/node-diagnostics-detail.md for remediation.${NC}"
- section "1. Cluster Identity"
- CLUSTER_JSON=$(aws sagemaker describe-cluster \
- --cluster-name "$CLUSTER" \
- --region "$REGION" \
- --cli-read-timeout 30 \
- --output json 2>&1) || {
- echo -e "${RED}ERROR: Cannot find cluster '$CLUSTER' in region '$REGION'${NC}"
- echo ""
- echo "Available clusters in $REGION:"
- aws sagemaker list-clusters --region "$REGION" \
- --query 'ClusterSummaries[*].{Name:ClusterName,Status:ClusterStatus,ARN:ClusterArn}' \
- --output table 2>/dev/null || echo " (unable to list)"
- exit 1
- }
- CLUSTER_ARN=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ClusterArn',''))" 2>/dev/null || echo "")
- CLUSTER_STATUS=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ClusterStatus',''))" 2>/dev/null || echo "")
- ORCHESTRATOR=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); print('EKS' if 'Eks' in d.get('Orchestrator',{}) else 'Slurm')" 2>/dev/null || echo "Unknown")
- NODE_RECOVERY=$(echo "$CLUSTER_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin).get('NodeRecovery','Unknown'))" 2>/dev/null || echo "Unknown")
- CLUSTER_ID=$(echo "$CLUSTER_ARN" | cut -d/ -f2 2>/dev/null || echo "")
- echo -e " ARN: ${CLUSTER_ARN}"
- echo -e " Status: ${BOLD}${CLUSTER_STATUS}${NC}"
- echo -e " Orchestrator: ${ORCHESTRATOR}"
- echo -e " NodeRecovery: ${NODE_RECOVERY}"
- echo -e " ClusterId: ${CLUSTER_ID}"
- [[ "$NODE_RECOVERY" == "None" || "$NODE_RECOVERY" == "Unknown" ]] && \
- warn "NodeRecovery is '$NODE_RECOVERY' — auto-replacement disabled. Manual intervention required for hardware failures."
- section "2. Cluster Events (Root Cause Signals)"
- # Fetch multiple pages and merge into a single JSON blob. Cap at 500 events to
- # bound memory and runtime on long-lived clusters (each page is up to 100).
- fetch_cluster_events() {
- local merged='[]' token='' page_json i=0
- while (( i < 5 )); do
- # Only pass --next-token if the token parses as a non-empty, strictly
- # base64/URL-safe string. Sending garbage (e.g. an error message that
- # leaked into $token) would cause ValidationException / BadRequest.
- if [[ -n "$token" && "$token" =~ ^[a-zA-Z0-9/+]*={0,2}$ ]]; then
- page_json=$(aws sagemaker list-cluster-events \
- --cluster-name "$CLUSTER" --region "$REGION" \
- --max-results 100 --next-token "$token" \
- --cli-read-timeout 30 --output json 2>&1) || break
- else
- page_json=$(aws sagemaker list-cluster-events \
- --cluster-name "$CLUSTER" --region "$REGION" \
- --max-results 100 \
- --cli-read-timeout 30 --output json 2>&1) || break
- fi
- local combined
- combined=$(printf '%s\0%s' "$merged" "$page_json" | python3 -c "
- import sys, json
- blob = sys.stdin.buffer.read()
- try:
- a, b = blob.split(b'\0', 1)
- merged = json.loads(a)
- page = json.loads(b)
- except (json.JSONDecodeError, ValueError):
- sys.exit(2)
- merged.extend(page.get('Events', []))
- print(json.dumps(merged))
- print(page.get('NextToken','') or '')
- " 2>/dev/null) || break
- merged=$(printf '%s\n' "$combined" | sed -n '1p')
- token=$(printf '%s\n' "$combined" | sed -n '2p')
- i=$((i+1))
- [[ -z "$token" ]] && break
- done
- printf '%s' "$merged" | python3 -c "
- import sys, json
- try:
- print(json.dumps({'Events': json.loads(sys.stdin.read())}))
- except json.JSONDecodeError:
- print('{\"Events\":[]}')
- " 2>/dev/null || echo '{"Events":[]}'
- }
- EVENTS=$(fetch_cluster_events)
- if [[ -z "$EVENTS" ]] || echo "$EVENTS" | grep -qE "AccessDenied|not authorized"; then
- aws_check_perms "$EVENTS" "sagemaker:ListClusterEvents"
- EVENTS='{"Events":[]}'
- fi
- EVENT_COUNT=$(echo "$EVENTS" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('Events',[])))" 2>/dev/null || echo "0")
- if [[ "$EVENT_COUNT" -gt 0 ]]; then
- echo -e " Found ${BOLD}${EVENT_COUNT}${NC} cluster events. Recent events:"
- echo ""
- echo "$EVENTS" | python3 -c "
- import sys, json
- events = json.load(sys.stdin).get('Events', [])
- for e in events[:20]:
- ts = e.get('EventTime','?')
- msg = e.get('Description','') or ''
- grp = e.get('InstanceGroupName','') or ''
- rt = e.get('ResourceType','') or ''
- tag = ''
- low = msg.lower()
- if 'EFA health checks did not run' in msg:
- tag = ' ← [GO TO SECTION A: EFA/SG FIX]'
- elif 'bootstrap failed' in low and 'network' in low:
- tag = ' ← [GO TO SECTION A+B: VPC/EKS FIX]'
- elif 'Lifecycle scripts' in msg or 'lifecycle script' in low:
- tag = ' ← [GO TO SECTION D: LIFECYCLE FIX]'
- elif 'hardware failure' in low:
- tag = ' ← [GO TO SECTION F: HARDWARE]'
- elif 'insufficient capacity' in low or 'sufficient capacity' in low:
- tag = ' ← [GO TO SECTION C: CAPACITY]'
- elif 'failed to provision' in low:
- tag = ' ← [CHECK SECTION C or F]'
- elif 'successfully' in low and 'failed' not in low:
- tag = ' [OK]'
- label = (grp or rt or '?')
- print(f' [{label}] {ts}')
- print(f' {msg[:120]}{\"...\" if len(msg) > 120 else \"\"}{tag}')
- print()
- " 2>/dev/null
- FAILURE_EVENTS=$(echo "$EVENTS" | python3 -c "
- import sys,json
- events=json.load(sys.stdin).get('Events',[])
- fails=[(e.get('Description','') or '') for e in events if any(k in (e.get('Description','') or '').lower() for k in ['failed','error','timeout','fault','unhealthy'])]
- for f in fails[:5]:
- print(f)
- " 2>/dev/null || echo "")
- if echo "$FAILURE_EVENTS" | grep -qi "efa health"; then
- add_issue "EFA health check failure → references/node-diagnostics-detail.md § A (EFA / Security Group)" "P0"
- fi
- if echo "$FAILURE_EVENTS" | grep -qi "network misconfiguration\|bootstrap failed"; then
- add_issue "K8s bootstrap network error → references/node-diagnostics-detail.md § A (EFA / Security Group) + § B (VPC / Routing)" "P1"
- fi
- if echo "$FAILURE_EVENTS" | grep -qi "lifecycle script"; then
- add_issue "Lifecycle script failure → references/node-diagnostics-detail.md § D (Lifecycle Scripts)" "P1"
- fi
- if echo "$FAILURE_EVENTS" | grep -qi "hardware failure"; then
- add_issue "Hardware failure detected → references/node-diagnostics-detail.md § F (Hardware / Auto-Repair)" "P1"
- fi
- if echo "$FAILURE_EVENTS" | grep -qi "insufficient capacity"; then
- add_issue "Insufficient capacity → references/node-diagnostics-detail.md § C (Capacity / AZ)" "P1"
- fi
- else
- warn "No cluster events available (may be Slurm cluster or no events yet)"
- fi
- section "3. Node Health Status"
- # Paginate list-cluster-nodes — default page is only 10 nodes, so large clusters
- # would otherwise be diagnosed on a tiny sample.
- fetch_all_cluster_nodes() {
- local merged='[]' token='' page_json combined i=0
- local max_pages=200 # 200 × 100 = 20 000 nodes, supports 7k+ clusters
- while (( i < max_pages )); do
- # Validate token format before sending — avoid BadRequest on garbage.
- if [[ -n "$token" && "$token" =~ ^[a-zA-Z0-9/+]*={0,2}$ ]]; then
- page_json=$(aws sagemaker list-cluster-nodes \
- --cluster-name "$CLUSTER" --region "$REGION" \
- --max-results 100 --next-token "$token" \
- --cli-read-timeout 30 --output json 2>&1) || break
- else
- page_json=$(aws sagemaker list-cluster-nodes \
- --cluster-name "$CLUSTER" --region "$REGION" \
- --max-results 100 \
- --cli-read-timeout 30 --output json 2>&1) || break
- fi
- # Merge via stdin (NUL-delimited) to avoid ARG_MAX truncation at ~500 nodes.
- combined=$(printf '%s\0%s' "$merged" "$page_json" | python3 -c "
- import sys, json
- blob = sys.stdin.buffer.read()
- try:
- a, b = blob.split(b'\0', 1)
- merged = json.loads(a)
- page = json.loads(b)
- except (json.JSONDecodeError, ValueError):
- sys.exit(2)
- merged.extend(page.get('ClusterNodeSummaries', []))
- print(json.dumps(merged))
- print(page.get('NextToken','') or '')
- " 2>/dev/null) || break
- merged=$(printf '%s\n' "$combined" | sed -n '1p')
- token=$(printf '%s\n' "$combined" | sed -n '2p')
- i=$((i+1))
- [[ -z "$token" ]] && break
- done
- if (( i == max_pages )) && [[ -n "$token" ]]; then
- echo "WARN: list-cluster-nodes truncated at ${max_pages} pages (~$((max_pages*100)) nodes). Diagnostic sample is incomplete for very large clusters." >&2
- fi
- printf '%s' "$merged" | python3 -c "
- import sys, json
- try:
- print(json.dumps({'ClusterNodeSummaries': json.loads(sys.stdin.read())}))
- except json.JSONDecodeError:
- print('{\"ClusterNodeSummaries\":[]}')
- " 2>/dev/null || echo '{"ClusterNodeSummaries":[]}'
- }
- NODES_JSON=$(fetch_all_cluster_nodes)
- if [[ -z "$NODES_JSON" ]] || echo "$NODES_JSON" | grep -qE "AccessDenied|not authorized"; then
- aws_check_perms "$NODES_JSON" "sagemaker:ListClusterNodes"
- NODES_JSON='{"ClusterNodeSummaries":[]}'
- fi
- TOTAL_NODES=$(echo "$NODES_JSON" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('ClusterNodeSummaries',[])))" 2>/dev/null || echo "0")
- RUNNING_NODES=$(echo "$NODES_JSON" | python3 -c "import sys,json; print(sum(1 for n in json.load(sys.stdin).get('ClusterNodeSummaries',[]) if n.get('InstanceStatus',{}).get('Status')=='Running'))" 2>/dev/null || echo "0")
- BAD_NODES=$(echo "$NODES_JSON" | python3 -c "import sys,json; print(sum(1 for n in json.load(sys.stdin).get('ClusterNodeSummaries',[]) if n.get('InstanceStatus',{}).get('Status') not in ('Running','')))" 2>/dev/null || echo "0")
- echo -e " Total: ${TOTAL_NODES} Running: ${GREEN}${RUNNING_NODES}${NC} Problems: ${RED}${BAD_NODES}${NC}"
- if [[ "$BAD_NODES" -gt 0 ]]; then
- echo ""
- echo -e " ${RED}Non-Running nodes:${NC}"
- echo "$NODES_JSON" | python3 -c "
- import sys,json
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- for n in nodes:
- status=n.get('InstanceStatus',{})
- st=status.get('Status','?')
- if st not in ('Running',''):
- iid=n.get('InstanceId','?')
- grp=n.get('InstanceGroupName','?')
- itype=n.get('InstanceType','?')
- msg=status.get('Message','')
- print(f' [FAIL] {iid} ({grp} / {itype})')
- print(f' Status: {st}')
- if msg:
- print(f' Message: {msg[:100]}')
- print()
- " 2>/dev/null
- add_issue "$BAD_NODES node(s) not Running → references/node-diagnostics-detail.md § F (Hardware / Auto-Repair)" "P1"
- else
- ok "All $TOTAL_NODES nodes are Running"
- fi
- if [[ -n "$TARGET_NODE" ]]; then
- echo ""
- echo -e " ${BOLD}Targeted node: ${TARGET_NODE}${NC}"
- NODE_DETAIL=$(aws sagemaker describe-cluster-node \
- --cluster-name "$CLUSTER" \
- --node-id "$TARGET_NODE" \
- --region "$REGION" \
- --cli-read-timeout 30 \
- --output json 2>&1 || true)
- if echo "$NODE_DETAIL" | grep -qiE "ResourceNotFound|not found|ValidationException"; then
- bad "Node '$TARGET_NODE' not found in cluster '$CLUSTER'"
- info "Verify the instance ID belongs to this cluster:"
- info " aws sagemaker list-cluster-nodes --cluster-name $CLUSTER --region $REGION --query 'ClusterNodeSummaries[*].InstanceId' --output text"
- add_issue "Node $TARGET_NODE not found in cluster $CLUSTER → verify --cluster and --node arguments" "P0"
- TARGET_NODE="" # clear so downstream SSM probe doesn't retry on nonexistent node
- elif echo "$NODE_DETAIL" | grep -qiE "AccessDenied|UnauthorizedOperation"; then
- warn "Permission denied: sagemaker:DescribeClusterNode — check IAM policy"
- else
- echo "$NODE_DETAIL" | python3 -c "
- import sys,json
- try:
- d=json.load(sys.stdin).get('NodeDetails',{})
- st=d.get('InstanceStatus',{})
- print(f' Status: {st.get(\"Status\",\"?\")}')
- print(f' Launch: {d.get(\"LaunchTime\",\"?\")}')
- print(f' Message: {st.get(\"Message\",\"\")}')
- print(f' Type: {d.get(\"InstanceType\",\"?\")}')
- print(f' Group: {d.get(\"InstanceGroupName\",\"?\")}')
- except Exception:
- pass
- " 2>/dev/null
- fi
- fi
- if [[ "$ORCHESTRATOR" == "EKS" ]]; then
- section "4. EKS Node Health Labels"
- if command -v kubectl &>/dev/null; then
- UNHEALTHY_LABELS=$(kubectl get nodes \
- -l 'sagemaker.amazonaws.com/node-health-status notin (Schedulable)' \
- -o custom-columns='NODE:.metadata.name,HEALTH:.metadata.labels.sagemaker\.amazonaws\.com/node-health-status,FAULT:.metadata.labels.sagemaker\.amazonaws\.com/fault-types,DHC:.metadata.labels.sagemaker\.amazonaws\.com/deep-health-check-status' \
- --no-headers 2>/dev/null || echo "")
- if [[ -n "$UNHEALTHY_LABELS" ]]; then
- bad "Nodes with health issues:"
- while IFS= read -r line; do
- echo " $line"
- if echo "$line" | grep -q "PendingReplacement"; then
- add_issue "Node pending replacement (UnschedulablePendingReplacement) → references/node-diagnostics-detail.md § F (Hardware / Auto-Repair)" "P1"
- elif echo "$line" | grep -q "PendingReboot"; then
- add_issue "Node pending reboot (UnschedulablePendingReboot) → references/node-diagnostics-detail.md § F (Hardware / Auto-Repair)" "P1"
- fi
- done <<< "$UNHEALTHY_LABELS"
- else
- ok "All EKS nodes have healthy labels (Schedulable)"
- fi
- # Check deep health check status. Under `set -o pipefail`, a failed kubectl
- # with `| wc -l || echo 0` yields "0\n0". Count safely via a tmp var.
- DHC_FAILED_OUT=$(kubectl get nodes \
- -l 'sagemaker.amazonaws.com/deep-health-check-status=Failed' \
- -o name 2>/dev/null || true)
- DHC_FAILED=$(echo -n "$DHC_FAILED_OUT" | grep -c . || true)
- [[ -z "$DHC_FAILED" ]] && DHC_FAILED=0
- [[ "$DHC_FAILED" -gt 0 ]] && bad "$DHC_FAILED node(s) have deep-health-check-status=Failed → references/node-diagnostics-detail.md § G (GPU/Accelerator) + § F (Hardware / Auto-Repair)"
- else
- warn "kubectl not available — cannot check EKS node labels (install kubectl to enable this check)"
- fi
- fi
- if [[ "$ORCHESTRATOR" == "EKS" ]] && command -v kubectl &>/dev/null; then
- section "4a. EKS CNI & System Pod Health"
- CNI_ISSUES=0
- # aws-node (VPC CNI plugin) — if this crashes, no pods can get IPs
- AWS_NODE_DS=$(kubectl get ds -n kube-system aws-node -o json 2>/dev/null || echo "")
- if [[ -n "$AWS_NODE_DS" && "$AWS_NODE_DS" != "" ]]; then
- AWS_NODE_STATUS=$(echo "$AWS_NODE_DS" | python3 -c "
- import sys, json
- ds = json.load(sys.stdin)
- desired = ds.get('status',{}).get('desiredNumberScheduled', 0)
- ready = ds.get('status',{}).get('numberReady', 0)
- unavail = ds.get('status',{}).get('numberUnavailable', 0)
- if unavail > 0:
- print(f'FAIL:{unavail} of {desired} aws-node pods not ready — pod networking broken on those nodes')
- elif ready == desired and desired > 0:
- print(f'PASS:aws-node DaemonSet healthy ({ready}/{desired} ready)')
- elif desired == 0:
- print('WARN:aws-node DaemonSet has 0 desired pods')
- else:
- print(f'WARN:aws-node DaemonSet {ready}/{desired} ready')
- " 2>/dev/null || echo "")
- if [[ -n "$AWS_NODE_STATUS" ]]; then
- _level="${AWS_NODE_STATUS%%:*}"
- _msg="${AWS_NODE_STATUS#*:}"
- case "$_level" in
- PASS) ok "$_msg" ;;
- FAIL) bad "$_msg"
- add_issue "aws-node (VPC CNI) pods failing → references/node-diagnostics-detail.md § O (CNI / Pod Networking)" "P0"
- CNI_ISSUES=$((CNI_ISSUES + 1))
- ;;
- WARN) warn "$_msg" ;;
- esac
- fi
- CNI_CRASHES=$(kubectl get pods -n kube-system -l k8s-app=aws-node --no-headers 2>/dev/null \
- | grep -iE "CrashLoopBackOff|Error|ImagePullBackOff" || true)
- if [[ -n "$CNI_CRASHES" ]]; then
- bad "aws-node pods in crash state:"
- echo "$CNI_CRASHES" | while IFS= read -r line; do echo " $line"; done
- add_issue "aws-node CrashLoopBackOff — pod networking broken → references/node-diagnostics-detail.md § O (CNI / Pod Networking)" "P0"
- CNI_ISSUES=$((CNI_ISSUES + 1))
- CNI_LOGS=$(kubectl logs -n kube-system -l k8s-app=aws-node --tail=20 2>/dev/null | \
- grep -iE "error|failed|refused|timeout|fatal|gRPC|ipamd|eni" | tail -5 || true)
- if [[ -n "$CNI_LOGS" ]]; then
- info "Recent aws-node error logs:"
- echo "$CNI_LOGS" | while IFS= read -r line; do info " $line"; done
- fi
- fi
- else
- info "aws-node DaemonSet not found in kube-system (may use alternate CNI)"
- fi
- # kube-proxy — if down, service networking breaks
- KP_CRASHES=$(kubectl get pods -n kube-system -l k8s-app=kube-proxy --no-headers 2>/dev/null \
- | grep -iE "CrashLoopBackOff|Error|ImagePullBackOff" || true)
- if [[ -n "$KP_CRASHES" ]]; then
- bad "kube-proxy pods in crash state:"
- echo "$KP_CRASHES" | while IFS= read -r line; do echo " $line"; done
- add_issue "kube-proxy crash — service networking broken → references/node-diagnostics-detail.md § O (CNI / Pod Networking)" "P0"
- CNI_ISSUES=$((CNI_ISSUES + 1))
- fi
- # CoreDNS — if down, DNS resolution fails (NCCL MASTER_ADDR, service discovery)
- COREDNS_CRASHES=$(kubectl get pods -n kube-system -l k8s-app=kube-dns --no-headers 2>/dev/null \
- | grep -iE "CrashLoopBackOff|Error|ImagePullBackOff" || true)
- if [[ -n "$COREDNS_CRASHES" ]]; then
- bad "CoreDNS pods in crash state — DNS resolution will fail:"
- echo "$COREDNS_CRASHES" | while IFS= read -r line; do echo " $line"; done
- add_issue "CoreDNS crash — DNS broken → references/node-diagnostics-detail.md § O (CNI / Pod Networking)" "P0"
- CNI_ISSUES=$((CNI_ISSUES + 1))
- fi
- [[ "$CNI_ISSUES" -eq 0 ]] && ok "kube-system networking pods healthy (aws-node, kube-proxy, CoreDNS)"
- fi
- if [[ "$ORCHESTRATOR" == "Slurm" ]]; then
- section "4b. Slurm Node States"
- if command -v sinfo &>/dev/null; then
- SLURM_DOWN=$(sinfo -o "%N %T %30E" --noheader 2>/dev/null | grep -iE "down|drain|fail" || true)
- if [[ -n "$SLURM_DOWN" ]]; then
- bad "Slurm nodes with issues:"
- echo "$SLURM_DOWN" | while IFS= read -r line; do
- echo " $line"
- done
- DOWN_COUNT=$(echo "$SLURM_DOWN" | grep -c .)
- add_issue "$DOWN_COUNT Slurm node(s) down/drained → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P1"
- else
- ok "All Slurm nodes show idle/alloc/mixed state"
- fi
- STUCK_JOBS=$(squeue -o "%i %j %T %R %N" --noheader 2>/dev/null | grep -iE "COMPLETING|CONFIGURING" || true)
- if [[ -n "$STUCK_JOBS" ]]; then
- warn "Stuck jobs detected (COMPLETING/CONFIGURING):"
- echo "$STUCK_JOBS" | head -5 | while IFS= read -r line; do echo " $line"; done
- add_issue "Stuck Slurm jobs → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P1"
- fi
- else
- info "Slurm CLI not available locally — to check Slurm node states, SSM into the head node:"
- info " sinfo -o '%N %T %30E'"
- info " squeue -o '%i %j %T %R %N'"
- info ""
- info "Or use SSM to run remotely:"
- if [[ -n "$CLUSTER_ID" ]]; then
- HEAD_NODE=$(echo "$NODES_JSON" | python3 -c "
- import sys,json
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- for n in nodes:
- g=n.get('InstanceGroupName','').lower()
- if any(x in g for x in ['controller','head','master']):
- print(n.get('InstanceId','') + ' ' + n.get('InstanceGroupName',''))
- break
- else:
- for n in nodes:
- if n.get('InstanceStatus',{}).get('Status')=='Running':
- print(n.get('InstanceId','') + ' ' + n.get('InstanceGroupName',''))
- break
- " 2>/dev/null || echo "")
- if [[ -n "$HEAD_NODE" ]]; then
- H_IID=$(echo "$HEAD_NODE" | awk '{print $1}')
- H_GRP=$(echo "$HEAD_NODE" | awk '{print $2}')
- info " aws ssm start-session --target sagemaker-cluster:${CLUSTER_ID}_${H_GRP}-${H_IID} --region $REGION"
- fi
- fi
- if command -v session-manager-plugin &>/dev/null && [[ -n "$HEAD_NODE" ]]; then
- H_IID=$(echo "$HEAD_NODE" | awk '{print $1}')
- # Validate instance ID format — defense-in-depth against unexpected input.
- if [[ "$H_IID" =~ ^i-[0-9a-f]{8,17}$ ]]; then
- info ""
- info "Running Slurm checks via SSM on controller ${H_IID}..."
- # Unique delimiter prevents false matches if check output happens to contain marker text.
- local_nonce=$(date +%s%N 2>/dev/null || echo "$RANDOM")
- SLURM_CHECK_SH=$(cat <<EOF
- echo SLURM_CHECK_START_${local_nonce}
- scontrol show config >/dev/null 2>&1 || echo SLURMCTLD_DOWN_${local_nonce}
- echo DOWN_NODES_${local_nonce}
- sinfo -o '%20N %10T %30E' --noheader 2>/dev/null | grep -iE 'down|drain|fail' | head -10
- echo END_DOWN_${local_nonce}
- echo STUCK_COUNT_${local_nonce}
- squeue -o '%i %T' --noheader 2>/dev/null | grep -cE 'COMPLETING|CONFIGURING' || echo 0
- echo MUNGE_${local_nonce}
- systemctl is-active munge 2>/dev/null || echo inactive
- echo SLURM_CHECK_END_${local_nonce}
- EOF
- )
- SSM_STDOUT=$(ssm_run_on_node "$H_IID" "$H_GRP" "$SLURM_CHECK_SH" || echo "")
- if [[ -z "$SSM_STDOUT" ]] || ! echo "$SSM_STDOUT" | grep -q "SLURM_CHECK_START_${local_nonce}"; then
- warn "Slurm SSM probe returned no usable output — controller may be unreachable or SSM agent not responding"
- add_issue "Slurm controller SSM probe failed → references/node-diagnostics-detail.md § K (Node Access via SSM) + § H (Slurm Node Management)" "P1"
- fi
- if echo "$SSM_STDOUT" | grep -q "SLURM_CHECK_START_${local_nonce}"; then
- if echo "$SSM_STDOUT" | grep -q "SLURMCTLD_DOWN_${local_nonce}"; then
- bad "slurmctld not responding on controller — all Slurm operations blocked"
- add_issue "slurmctld down → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P0"
- else
- ok "slurmctld responding"
- fi
- SSM_DOWN_LINES=$(echo "$SSM_STDOUT" | sed -n "/^DOWN_NODES_${local_nonce}\$/,/^END_DOWN_${local_nonce}\$/p" | grep -v "^DOWN_NODES_\|^END_DOWN_" | grep -v "^$" || true)
- if [[ -n "$SSM_DOWN_LINES" ]]; then
- bad "Slurm nodes with issues (via SSM):"
- echo "$SSM_DOWN_LINES" | while IFS= read -r line; do info " $line"; done
- SSM_DOWN_COUNT=$(echo "$SSM_DOWN_LINES" | grep -c .)
- add_issue "$SSM_DOWN_COUNT Slurm node(s) down/drained → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P1"
- else
- ok "All Slurm nodes healthy (via SSM)"
- fi
- STUCK_COUNT=$(echo "$SSM_STDOUT" | sed -n "/^STUCK_COUNT_${local_nonce}\$/{n;p;}" | tr -d '[:space:]')
- [[ "${STUCK_COUNT:-0}" =~ ^[0-9]+$ ]] && [[ "${STUCK_COUNT:-0}" -gt 0 ]] && \
- add_issue "$STUCK_COUNT stuck Slurm jobs → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P1"
- if echo "$SSM_STDOUT" | sed -n "/^MUNGE_${local_nonce}\$/{n;p;}" | grep -q inactive; then
- bad "munge authentication service inactive on controller"
- add_issue "munge service inactive → references/node-diagnostics-detail.md § H (Slurm Node Management)" "P0"
- fi
- fi
- fi
- fi
- fi
- fi
- section "5. Cluster VPC Resources"
- RESOURCES=$(echo "$CLUSTER_JSON" | python3 -c "
- import sys,json
- d=json.load(sys.stdin)
- vpc=d.get('VpcConfig',{})
- sgs=vpc.get('SecurityGroupIds',[])
- subnets=vpc.get('Subnets',[])
- print('SGs=' + ','.join(sgs))
- print('Subnets=' + ','.join(subnets))
- " 2>/dev/null || echo "")
- CLUSTER_SGS=$(echo "$RESOURCES" | grep "^SGs=" | cut -d= -f2)
- CLUSTER_SUBNETS=$(echo "$RESOURCES" | grep "^Subnets=" | cut -d= -f2)
- if [[ -n "$CLUSTER_SGS" ]]; then
- echo -e " Security Groups: ${BOLD}${CLUSTER_SGS}${NC}"
- echo -e " Subnets: ${BOLD}${CLUSTER_SUBNETS}${NC}"
- for SG in $(echo "$CLUSTER_SGS" | tr ',' ' '); do
- # Nested JMESPath filter `UserIdGroupPairs[?GroupId=='...']` inside an
- # already-filtered projection returns empty under AWS CLI even when the
- # rule is present — false-flags healthy SGs as a P0. Flatten the array
- # and match in bash instead.
- _SG_RESULT=$(aws ec2 describe-security-groups \
- --group-ids "$SG" --region "$REGION" \
- --cli-read-timeout 15 \
- --query "SecurityGroups[0].IpPermissionsEgress[?IpProtocol=='-1'].UserIdGroupPairs[].GroupId" \
- --output text 2>&1)
- if aws_check_perms "$_SG_RESULT" "ec2:DescribeSecurityGroups"; then
- info "SG check skipped for $SG (permission denied)"
- continue
- fi
- if echo "$_SG_RESULT" | tr '\t' '\n' | grep -qxF "$SG"; then
- ok "SG ${SG} has outbound self-referencing rule (EFA ready)"
- else
- bad "SG ${SG} missing outbound self-referencing rule → EFA will fail"
- add_issue "Missing SG outbound self-ref rule on ${SG} → references/node-diagnostics-detail.md § A (EFA / Security Group)" "P0"
- fi
- done
- if [[ -n "$CLUSTER_SUBNETS" ]]; then
- echo ""
- # shellcheck disable=SC2046 # intentional word splitting for multiple subnet IDs
- IFS=',' read -ra _subnet_arr <<< "$CLUSTER_SUBNETS"
- _SUB_RESULT=$(aws ec2 describe-subnets \
- --subnet-ids "${_subnet_arr[@]}" \
- --region "$REGION" \
- --cli-read-timeout 15 \
- --query 'Subnets[*].{SubnetId:SubnetId,AZ:AvailabilityZone,FreeIPs:AvailableIpAddressCount}' \
- --output table 2>&1)
- if ! aws_check_perms "$_SUB_RESULT" "ec2:DescribeSubnets"; then
- echo "$_SUB_RESULT"
- fi
- fi
- else
- warn "No VpcConfig found in cluster — cluster may not have customer VPC"
- fi
- section "6. CloudWatch Logs"
- if [[ -n "$CLUSTER_ID" ]]; then
- CLUSTER_NAME_ONLY=$(echo "$CLUSTER" | awk -F/ '{print $NF}')
- LOG_GROUP="/aws/sagemaker/Clusters/${CLUSTER_NAME_ONLY}/${CLUSTER_ID}"
- echo -e " Log group: ${LOG_GROUP}"
- _LOG_RESULT=$(aws logs describe-log-groups \
- --log-group-name-prefix "$LOG_GROUP" \
- --region "$REGION" \
- --cli-read-timeout 15 \
- --query 'logGroups[0].logGroupName' \
- --output text 2>&1)
- if aws_check_perms "$_LOG_RESULT" "logs:DescribeLogGroups"; then
- LOG_EXISTS="None"
- else
- LOG_EXISTS="$_LOG_RESULT"
- fi
- if [[ "$LOG_EXISTS" == "None" || -z "$LOG_EXISTS" ]]; then
- warn "No CloudWatch log group found — logs may not be configured or cluster is new"
- info "Expected: $LOG_GROUP"
- else
- ok "Log group exists: $LOG_EXISTS"
- # Count recent log streams — paginate so the count reflects all streams,
- # not just the first 50 (default CloudWatch page size).
- STREAM_COUNT=0
- _LS_TOKEN=""; _LS_I=0
- while (( _LS_I < 20 )); do
- # Validate token format before sending — avoid BadRequest on garbage.
- if [[ -n "$_LS_TOKEN" && "$_LS_TOKEN" =~ ^[a-zA-Z0-9/+]*={0,2}$ ]]; then
- _LS_PAGE=$(aws logs describe-log-streams --log-group-name "$LOG_GROUP" \
- --region "$REGION" --cli-read-timeout 15 --limit 50 --next-token "$_LS_TOKEN" \
- --output json 2>/dev/null) || break
- else
- _LS_PAGE=$(aws logs describe-log-streams --log-group-name "$LOG_GROUP" \
- --region "$REGION" --cli-read-timeout 15 --limit 50 \
- --output json 2>/dev/null) || break
- fi
- _LS_INC=$(echo "$_LS_PAGE" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('logStreams',[])))" 2>/dev/null || echo 0)
- STREAM_COUNT=$((STREAM_COUNT + _LS_INC))
- _LS_TOKEN=$(echo "$_LS_PAGE" | python3 -c "import sys,json; print(json.load(sys.stdin).get('nextToken',''))" 2>/dev/null || echo "")
- _LS_I=$((_LS_I + 1))
- [[ -z "$_LS_TOKEN" ]] && break
- done
- info "$STREAM_COUNT log stream(s) available"
- info "To view: aws logs describe-log-streams --log-group-name \"$LOG_GROUP\" --region $REGION --output table"
- fi
- fi
- section "7. SSM Connectivity"
- if command -v session-manager-plugin &>/dev/null; then
- # `command -v` only verifies the binary exists — run --version to confirm it
- # actually works (permissions, broken install, etc.).
- if SSM_VER=$(session-manager-plugin --version 2>/dev/null); then
- ok "SSM Session Manager plugin installed (${SSM_VER})"
- else
- warn "SSM Session Manager plugin installed but --version failed — plugin may be corrupt or missing libs"
- add_issue "SSM plugin installed but broken → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P1"
- fi
- else
- warn "SSM Session Manager plugin NOT found"
- info "Install session-manager-plugin (see AWS Systems Manager documentation)"
- add_issue "SSM plugin missing → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P2"
- fi
- RUNNING_IDS=$(echo "$NODES_JSON" | python3 -c "
- import sys,json
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- ids=[n.get('InstanceId') for n in nodes if n.get('InstanceStatus',{}).get('Status')=='Running']
- print(','.join(ids[:3]))
- " 2>/dev/null || echo "")
- if [[ -n "$RUNNING_IDS" ]]; then
- ok "Running nodes available for SSM (examples: ${RUNNING_IDS})"
- info "Use hyperpod-ssm skill with cluster ID: ${CLUSTER_ID}"
- else
- warn "No Running nodes found — SSM access not possible until nodes are healthy"
- fi
- # 8: On-Node Resource Checks (Memory / Storage / Utilities)
- # Runs via SSM on the target node (or first running node) to detect resource
- # exhaustion issues that only show up on-node: disk full, /dev/shm too small,
- # huge pages misconfigured, OOM signals.
- NODE_TO_PROBE="${TARGET_NODE}"
- NODE_TO_PROBE_GROUP=""
- if [[ -z "$NODE_TO_PROBE" ]]; then
- # Prefer GPU / accelerator nodes: a node probe on a CPU-only utility node
- # produces empty GPU / EFA sections and the user can't tell whether the
- # result is "no hardware" or "hardware is broken." Three-tier fallback
- # so the script still returns something on a CPU-only cluster.
- NODE_TO_PROBE=$(echo "$NODES_JSON" | python3 -c "
- import sys, json
- nodes = json.load(sys.stdin).get('ClusterNodeSummaries', [])
- GPU_PREFIXES = ('ml.p3', 'ml.p3dn', 'ml.p4d', 'ml.p4de', 'ml.p5', 'ml.p5e',
- 'ml.p5en', 'ml.p6', 'ml.g4dn', 'ml.g5', 'ml.g6', 'ml.g6e', 'ml.g7e')
- NEURON_PREFIXES = ('ml.trn1', 'ml.trn2', 'ml.inf2')
- ACCEL_PREFIXES = GPU_PREFIXES + NEURON_PREFIXES
- def is_utility(n):
- g = (n.get('InstanceGroupName','') or '').lower()
- return any(x in g for x in ('controller', 'head', 'master'))
- running = [n for n in nodes if n.get('InstanceStatus', {}).get('Status','') == 'Running']
- tier1 = [n for n in running if (n.get('InstanceType','') or '').startswith(ACCEL_PREFIXES) and not is_utility(n)]
- tier2 = [n for n in running if n not in tier1 and not is_utility(n)]
- tier3 = [n for n in running if n not in tier1 and n not in tier2]
- for n in tier1 + tier2 + tier3:
- print(n.get('InstanceId', ''))
- break
- " 2>/dev/null || echo "")
- fi
- if [[ -n "$NODE_TO_PROBE" ]]; then
- NODE_TO_PROBE_GROUP=$(echo "$NODES_JSON" | NODE_ID_ENV="$NODE_TO_PROBE" python3 -c "
- import sys,json,os
- target=os.environ['NODE_ID_ENV']
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- for n in nodes:
- if n.get('InstanceId','')==target:
- print(n.get('InstanceGroupName',''))
- break
- " 2>/dev/null || echo "")
- fi
- if [[ -n "$NODE_TO_PROBE" ]] \
- && [[ "$NODE_TO_PROBE" =~ ^i-[0-9a-f]{8,17}$ ]] \
- && [[ -n "$NODE_TO_PROBE_GROUP" ]] \
- && command -v session-manager-plugin &>/dev/null; then
- section "8. On-Node Resource Checks (via SSM)"
- info "Probing node: $NODE_TO_PROBE (group: ${NODE_TO_PROBE_GROUP})"
- resource_nonce=$(date +%s%N 2>/dev/null || echo "$RANDOM")
- RESOURCE_SH=$(cat <<EOF
- echo RESOURCE_CHECK_START_${resource_nonce}
- echo DISK_ROOT_${resource_nonce}
- df -h / 2>/dev/null | tail -1
- echo DISK_OPT_${resource_nonce}
- df -h /opt/sagemaker 2>/dev/null | tail -1 || echo NOT_MOUNTED
- echo DISK_NVME_${resource_nonce}
- df -h /opt/dlami/nvme 2>/dev/null | tail -1 || echo NOT_MOUNTED
- echo SHM_SIZE_${resource_nonce}
- df -h /dev/shm 2>/dev/null | tail -1
- echo MEMORY_INFO_${resource_nonce}
- free -h | grep Mem
- echo HUGEPAGES_${resource_nonce}
- cat /proc/meminfo 2>/dev/null | grep -i huge | head -5
- echo EFA_HUGE_PAGE_${resource_nonce}
- env 2>/dev/null | grep FI_EFA_USE_HUGE_PAGE || echo NOT_SET
- echo OOM_RECENT_${resource_nonce}
- dmesg 2>/dev/null | grep -iE 'oom|out of memory|cannot allocate' | tail -5 || echo NONE
- echo INODE_CHECK_${resource_nonce}
- df -i / 2>/dev/null | tail -1
- echo TIME_SYNC_${resource_nonce}
- chronyc tracking 2>/dev/null | grep -E 'System time|Leap status' || timedatectl status 2>/dev/null | grep -E 'synchronized|NTP service' || echo UNKNOWN
- echo SSM_AGENT_${resource_nonce}
- systemctl is-active amazon-ssm-agent 2>/dev/null || echo inactive
- echo NVME_MOUNTS_${resource_nonce}
- lsblk -nr -o NAME,MOUNTPOINT 2>/dev/null | grep -E 'nvme[0-9]+n[0-9]+\$' | head -10 || echo NONE
- echo GPU_XID_${resource_nonce}
- if command -v nvidia-smi >/dev/null 2>&1; then
- _gpu_xid_out=\$(
- dmesg 2>/dev/null | grep -E 'NVRM: Xid' | tail -10
- nvidia-smi -q 2>/dev/null | awk '
- /Uncorrectable/ { if (\$NF ~ /^[0-9]+\$/ && \$NF+0 > 0) print; next }
- /Pending Page (Blacklist|Blocklist|Retirement)/ { if (\$NF ~ /^[0-9]+\$/ && \$NF+0 > 0) print; next }
- '
- )
- if [[ -z "\$_gpu_xid_out" ]]; then echo NONE; else echo "\$_gpu_xid_out" | head -20; fi
- else
- echo NO_NVIDIA_SMI
- fi
- echo GPU_REMAP_${resource_nonce}
- # Row-remap state: 'Pending' rows indicate marginal GPU memory that needs a reset
- # to finalize the remap. If remap is reported Failed, the GPU is bad.
- # A stuck 'Pending' state across reboots is a known firmware edge case that can
- # silently degrade training without NCCL/DCGM flagging it — capture explicitly.
- if command -v nvidia-smi >/dev/null 2>&1; then
- nvidia-smi --query-remapped-rows=gpu_bus_id,remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure \
- --format=csv,noheader 2>/dev/null | head -16 || echo UNSUPPORTED
- else
- echo NO_NVIDIA_SMI
- fi
- echo GPU_DCGM_${resource_nonce}
- # DCGM health summary. Presence of 'Health Monitor Report' + 'PASS'/'Warn'/'Fail'
- # tells us DCGM has run recently. Absence is informational, not an error.
- # Row-remap errors surface here on drivers where nvidia-smi lags the firmware.
- if command -v dcgmi >/dev/null 2>&1; then
- dcgmi health --check -j 2>/dev/null | head -40 || dcgmi health --check 2>/dev/null | head -20 || echo DCGM_UNAVAILABLE
- else
- echo NO_DCGMI
- fi
- echo GPU_DCGM_LOGS_${resource_nonce}
- # DCGM nvvs log presence — SageMaker HyperPod runs DCGM medium/memtest as part
- # of deep-health-check. If this log is present the node has been health-checked
- # recently; tail captures last run result.
- if [ -d /var/log/nvidia-dcgm ] 2>/dev/null; then
- find /var/log/nvidia-dcgm -maxdepth 1 -type f -printf '%f\n' 2>/dev/null | head -5
- # \$ escapes are required: this heredoc is <<EOF (not <<'EOF'), so unescaped
- # shell variables would expand locally. Keep \$ to defer to the remote shell.
- NVVS_LATEST=\$(find /var/log/nvidia-dcgm -maxdepth 1 -name 'nvvs*.log' -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -1 | awk '{print \$2}')
- if [ -n "\$NVVS_LATEST" ]; then
- echo "--- tail of \$NVVS_LATEST ---"
- tail -n 5 "\$NVVS_LATEST" 2>/dev/null || true
- fi
- else
- echo NO_DCGM_LOG_DIR
- fi
- echo KERNEL_PANIC_${resource_nonce}
- dmesg 2>/dev/null | grep -iE 'Kernel panic - not syncing|watchdog: BUG|soft lockup|hard lockup|hung_task: blocked|BUG: unable to handle|BUG: kernel NULL|NMI watchdog' | tail -10 || echo NONE
- echo CONTAINERD_${resource_nonce}
- if command -v systemctl >/dev/null 2>&1; then
- systemctl is-active containerd 2>/dev/null || echo inactive
- else
- echo UNKNOWN
- fi
- echo RESOURCE_CHECK_END_${resource_nonce}
- EOF
- )
- RES_STDOUT=$(ssm_run_on_node "$NODE_TO_PROBE" "$NODE_TO_PROBE_GROUP" "$RESOURCE_SH" || echo "")
- extract_section() {
- local start="$1" end="$2"
- # grep -v returns 1 when every line is filtered out; under pipefail this
- # kills the pipeline even though the EMPTY output is legitimate. Force 0.
- { echo "$RES_STDOUT" | sed -n "/^${start}_${resource_nonce}\$/,/^${end}_${resource_nonce}\$/p" \
- | grep -v "^${start}_${resource_nonce}\$\|^${end}_${resource_nonce}\$" || true; }
- }
- if echo "$RES_STDOUT" | grep -q "RESOURCE_CHECK_START_${resource_nonce}"; then
- echo ""
- echo -e " ${BOLD}Storage:${NC}"
- ROOT_LINE=$(extract_section DISK_ROOT DISK_OPT | head -1)
- if [[ -n "$ROOT_LINE" ]]; then
- ROOT_USE_PCT=$(echo "$ROOT_LINE" | awk '{print $5}' | tr -d '%')
- if [[ "$ROOT_USE_PCT" =~ ^[0-9]+$ ]] && [[ "$ROOT_USE_PCT" -gt 90 ]]; then
- bad "Root volume: ${ROOT_USE_PCT}% used — CRITICALLY FULL (100GB fixed, cannot expand)"
- add_issue "Root volume ${ROOT_USE_PCT}% full → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P0"
- elif [[ "$ROOT_USE_PCT" =~ ^[0-9]+$ ]] && [[ "$ROOT_USE_PCT" -gt 80 ]]; then
- warn "Root volume: ${ROOT_USE_PCT}% used — approaching full"
- add_issue "Root volume ${ROOT_USE_PCT}% used → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
- else
- ok "Root volume: ${ROOT_USE_PCT:-?}% used"
- fi
- fi
- OPT_LINE=$(extract_section DISK_OPT DISK_NVME | head -1)
- if [[ "$OPT_LINE" != "NOT_MOUNTED" && -n "$OPT_LINE" ]]; then
- OPT_USE=$(echo "$OPT_LINE" | awk '{print $5}' | tr -d '%')
- if [[ "$OPT_USE" =~ ^[0-9]+$ ]] && [[ "$OPT_USE" -gt 90 ]]; then
- warn "/opt/sagemaker: ${OPT_USE}% used — secondary EBS nearing full"
- add_issue "/opt/sagemaker ${OPT_USE}% full → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
- else
- ok "/opt/sagemaker: ${OPT_USE:-?}% used"
- fi
- fi
- NVME_LINE=$(extract_section DISK_NVME SHM_SIZE | head -1)
- if [[ "$NVME_LINE" != "NOT_MOUNTED" && -n "$NVME_LINE" ]]; then
- ok "NVMe instance store: mounted at /opt/dlami/nvme"
- else
- # On GPU training instances NVMe is expected — flag if not mounted
- INSTANCE_TYPE_LOC=$(echo "$NODES_JSON" | NODE_ID_ENV="$NODE_TO_PROBE" python3 -c "
- import sys,json,os
- target=os.environ['NODE_ID_ENV']
- for n in json.load(sys.stdin).get('ClusterNodeSummaries',[]):
- if n.get('InstanceId','')==target:
- print(n.get('InstanceType',''))
- break
- " 2>/dev/null || echo "")
- if [[ "$INSTANCE_TYPE_LOC" =~ ^ml\.(p5|p5e|p5en|p4d|p4de|p6|trn1|trn2)\. ]]; then
- warn "/opt/dlami/nvme not mounted on $INSTANCE_TYPE_LOC — instance store expected"
- add_issue "NVMe instance store not mounted on $NODE_TO_PROBE ($INSTANCE_TYPE_LOC) → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
- fi
- fi
- INODE_LINE=$(extract_section INODE_CHECK TIME_SYNC | head -1)
- if [[ -n "$INODE_LINE" ]]; then
- INODE_PCT=$(echo "$INODE_LINE" | awk '{print $5}' | tr -d '%')
- if [[ "$INODE_PCT" =~ ^[0-9]+$ ]] && [[ "$INODE_PCT" -gt 90 ]]; then
- bad "Inode usage: ${INODE_PCT}% — filesystem running out of inodes"
- add_issue "Inode exhaustion ${INODE_PCT}% → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
- fi
- fi
- echo ""
- echo -e " ${BOLD}Memory:${NC}"
- MEM_LINE=$(extract_section MEMORY_INFO HUGEPAGES | head -1)
- [[ -n "$MEM_LINE" ]] && info "RAM: $MEM_LINE"
- SHM_LINE=$(extract_section SHM_SIZE MEMORY_INFO | head -1)
- if [[ -n "$SHM_LINE" ]]; then
- SHM_SIZE=$(echo "$SHM_LINE" | awk '{print $2}')
- SHM_USE_PCT=$(echo "$SHM_LINE" | awk '{print $5}' | tr -d '%')
- if [[ "$SHM_USE_PCT" =~ ^[0-9]+$ ]] && [[ "$SHM_USE_PCT" -gt 80 ]]; then
- warn "/dev/shm: ${SHM_USE_PCT}% used (size: $SHM_SIZE) — NCCL may fail with 'Bus error'"
- add_issue "/dev/shm ${SHM_USE_PCT}% full → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
- else
- ok "/dev/shm: ${SHM_USE_PCT:-?}% used (size: ${SHM_SIZE:-?})"
- fi
- fi
- EFA_HP=$(extract_section EFA_HUGE_PAGE OOM_RECENT | head -1)
- if [[ "$EFA_HP" == "NOT_SET" ]]; then
- HUGEPAGES_TOTAL=$(extract_section HUGEPAGES EFA_HUGE_PAGE | { grep "HugePages_Total" || true; } | awk '{print $2}')
- if [[ "${HUGEPAGES_TOTAL:-0}" == "0" ]]; then
- warn "FI_EFA_USE_HUGE_PAGE not set and HugePages_Total=0"
- add_issue "FI_EFA_USE_HUGE_PAGE not configured → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P2"
- fi
- elif echo "$EFA_HP" | grep -q "=0"; then
- ok "FI_EFA_USE_HUGE_PAGE=0 (huge pages disabled for EFA — os.fork() safe)"
- fi
- OOM_LINES=$(extract_section OOM_RECENT INODE_CHECK | { grep -v "^NONE$" || true; } | head -3)
- if [[ -n "$OOM_LINES" ]]; then
- echo ""
- bad "Recent OOM events detected on node:"
- echo "$OOM_LINES" | while IFS= read -r line; do info " $line"; done
- add_issue "OOM events on node $NODE_TO_PROBE → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
- else
- echo ""
- ok "No recent OOM events"
- fi
- # Time sync health — clock drift breaks TLS/SigV4 and Slurm accounting.
- TIME_STATUS=$(extract_section TIME_SYNC SSM_AGENT | head -3)
- if echo "$TIME_STATUS" | grep -qiE "synchronized: no|Not synchronised|UNKNOWN"; then
- warn "Time sync unhealthy — chronyc/timedatectl reports not synchronised"
- info "Clock drift breaks TLS/IAM (SigV4) and Slurm accounting"
- add_issue "Node $NODE_TO_PROBE time sync not healthy → references/node-diagnostics-detail.md § I (Resource Exhaustion)" "P1"
- elif [[ -n "$TIME_STATUS" ]]; then
- ok "Time sync healthy"
- fi
- # SSM agent health — if we got here it's mostly working, but flag if systemd says otherwise.
- SSM_AGENT_STATUS=$(extract_section SSM_AGENT NVME_MOUNTS | head -1)
- if [[ "$SSM_AGENT_STATUS" == "inactive" ]]; then
- warn "amazon-ssm-agent reported inactive — may be restarting or broken"
- add_issue "amazon-ssm-agent inactive on $NODE_TO_PROBE → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P1"
- fi
- # GPU XID / ECC / page-retirement — hardware faults visible via nvidia-smi query.
- GPU_XID_LINES=$(extract_section GPU_XID GPU_REMAP | { grep -v "^NONE$" || true; } | { grep -v "^NO_NVIDIA_SMI$" || true; } | head -5)
- if [[ -n "$GPU_XID_LINES" ]]; then
- echo ""
- bad "GPU XID / ECC / page-retirement signals on node $NODE_TO_PROBE:"
- echo "$GPU_XID_LINES" | while IFS= read -r line; do info " $line"; done
- add_issue "GPU XID / ECC / page-retirement on $NODE_TO_PROBE → references/node-diagnostics-detail.md § G (GPU/Accelerator) + § F (Hardware / Auto-Repair)" "P0"
- fi
- # GPU row-remapping — marginal GPU memory. Pending rows that never clear
- # indicate a firmware edge case where the remap is stuck; Failed rows mean
- # the GPU is bad and must be replaced. Silent degrader — NCCL and DCGM's
- # default checks can miss this.
- GPU_REMAP_LINES=$(extract_section GPU_REMAP GPU_DCGM | { grep -v "^NO_NVIDIA_SMI$" || true; } | { grep -v "^UNSUPPORTED$" || true; })
- if [[ -n "$GPU_REMAP_LINES" ]]; then
- # Columns (csv,noheader): gpu_bus_id, correctable, uncorrectable, pending, failure
- REMAP_PENDING_TOTAL=0
- REMAP_FAILED_TOTAL=0
- REMAP_UNCORRECT_TOTAL=0
- while IFS= read -r line; do
- [[ -z "$line" ]] && continue
- _p=$(echo "$line" | awk -F, '{gsub(/ /,""); print $4}')
- _f=$(echo "$line" | awk -F, '{gsub(/ /,""); print $5}')
- _u=$(echo "$line" | awk -F, '{gsub(/ /,""); print $3}')
- [[ "$_p" =~ ^[0-9]+$ ]] && REMAP_PENDING_TOTAL=$((REMAP_PENDING_TOTAL + _p))
- [[ "$_u" =~ ^[0-9]+$ ]] && REMAP_UNCORRECT_TOTAL=$((REMAP_UNCORRECT_TOTAL + _u))
- [[ "$_f" == "Yes" || "$_f" == "1" ]] && REMAP_FAILED_TOTAL=$((REMAP_FAILED_TOTAL + 1))
- done <<< "$GPU_REMAP_LINES"
- if [[ "$REMAP_FAILED_TOTAL" -gt 0 ]]; then
- bad "GPU row-remap FAILED on $REMAP_FAILED_TOTAL device(s) — GPU has exceeded remap capacity"
- add_issue "GPU row-remap failure on $NODE_TO_PROBE (bad memory, replace GPU) → references/node-diagnostics-detail.md § G (GPU/Accelerator) + § F (Hardware / Auto-Repair)" "P0"
- elif [[ "$REMAP_PENDING_TOTAL" -gt 0 ]]; then
- bad "GPU row-remap PENDING — $REMAP_PENDING_TOTAL row(s) awaiting reset"
- info " Pending remaps indicate marginal memory that a GPU reset/reboot should finalize."
- info " If pending persists across reboots, the firmware may be stuck (known edge case) — escalate."
- add_issue "GPU row-remap pending on $NODE_TO_PROBE (reset/reboot to finalize; if stuck, marginal memory) → references/node-diagnostics-detail.md § G (GPU/Accelerator) + § F (Hardware / Auto-Repair)" "P1"
- elif [[ "$REMAP_UNCORRECT_TOTAL" -gt 0 ]]; then
- warn "GPU has $REMAP_UNCORRECT_TOTAL uncorrectable remapped rows (healthy now, but history of faults)"
- fi
- fi
- GPU_DCGM_LINES=$(extract_section GPU_DCGM GPU_DCGM_LOGS | { grep -v "^NO_DCGMI$" || true; } | { grep -v "^DCGM_UNAVAILABLE$" || true; })
- if [[ -n "$GPU_DCGM_LINES" ]]; then
- if echo "$GPU_DCGM_LINES" | grep -qiE '"overall_health"\s*:\s*"(Fail|Warn)"|HEALTH_RESULT_FAIL|HEALTH_RESULT_WARN|Health Monitor Report.*(Fail|Warn)'; then
- bad "DCGM health check reported Fail/Warn on $NODE_TO_PROBE"
- add_issue "DCGM health Fail/Warn on $NODE_TO_PROBE → references/node-diagnostics-detail.md § G (GPU/Accelerator)" "P0"
- fi
- fi
- # DCGM log presence — informational. Confirms deep-health-check history.
- GPU_DCGM_LOG_LINES=$(extract_section GPU_DCGM_LOGS KERNEL_PANIC)
- if echo "$GPU_DCGM_LOG_LINES" | grep -qi "nvvs"; then
- ok "DCGM nvvs logs present on $NODE_TO_PROBE (/var/log/nvidia-dcgm/)"
- if echo "$GPU_DCGM_LOG_LINES" | grep -qE "^--- tail"; then
- DCGM_TAIL=$(echo "$GPU_DCGM_LOG_LINES" | sed -n '/^--- tail/,$p' | head -20)
- if echo "$DCGM_TAIL" | grep -qiE 'FAIL|Error:|row ?remap.*(pending|fail)'; then
- warn "DCGM nvvs log tail contains failure/row-remap signals — inspect on node:"
- echo "$DCGM_TAIL" | while IFS= read -r line; do info " $line"; done
- add_issue "DCGM nvvs log shows failure/row-remap signals on $NODE_TO_PROBE → references/node-diagnostics-detail.md § G (GPU/Accelerator)" "P0"
- fi
- fi
- fi
- # Kernel panic / watchdog / hung task signals — indicate node-level instability.
- KERNEL_PANIC_LINES=$(extract_section KERNEL_PANIC CONTAINERD | { grep -v "^NONE$" || true; } | head -5)
- if [[ -n "$KERNEL_PANIC_LINES" ]]; then
- echo ""
- bad "Kernel panic / watchdog / hung_task signals on node $NODE_TO_PROBE:"
- echo "$KERNEL_PANIC_LINES" | while IFS= read -r line; do info " $line"; done
- add_issue "Kernel panic / watchdog on $NODE_TO_PROBE → references/node-diagnostics-detail.md § N (Kernel & System)" "P0"
- fi
- # containerd health — if the runtime is inactive, every pod on this node fails.
- CONTAINERD_STATUS=$(extract_section CONTAINERD RESOURCE_CHECK_END | head -1)
- if [[ "$CONTAINERD_STATUS" == "inactive" ]]; then
- warn "containerd is inactive on $NODE_TO_PROBE — all pods on this node will fail"
- add_issue "containerd inactive on $NODE_TO_PROBE → references/node-diagnostics-detail.md § M (Container Runtime)" "P0"
- fi
- else
- warn "SSM command returned no output — node may not be reachable"
- add_issue "Cannot reach node $NODE_TO_PROBE via SSM → references/node-diagnostics-detail.md § K (Node Access via SSM)" "P1"
- fi
- else
- if [[ -z "$NODE_TO_PROBE" ]]; then
- info "No running nodes to probe for resource checks"
- else
- info "SSM plugin not installed — skipping on-node resource checks → references/node-diagnostics-detail.md § K (Node Access via SSM)"
- fi
- fi
- if [[ "$ORCHESTRATOR" == "Slurm" && "$TOTAL_NODES" -gt 0 ]]; then
- section "8b. Slurm Node Mapping"
- info "Slurm node name → HyperPod instance ID mapping:"
- info "(PrivateDnsHostname is not in list-cluster-nodes; use 'describe-cluster-node --node-id <i-...>' to retrieve it for a specific instance.)"
- echo ""
- echo "$NODES_JSON" | python3 -c "
- import sys,json
- nodes=json.load(sys.stdin).get('ClusterNodeSummaries',[])
- print(f' {\"Instance ID\":<22} {\"Group\":<20} {\"Type\":<22} {\"Status\"}')
- print(f' {\"─\"*22} {\"─\"*20} {\"─\"*22} {\"─\"*10}')
- for n in nodes[:20]:
- iid=n.get('InstanceId','?')
- grp=n.get('InstanceGroupName','?')
- itype=n.get('InstanceType','?')
- st=n.get('InstanceStatus',{}).get('Status','?')
- print(f' {iid:<22} {grp:<20} {itype:<22} {st}')
- if len(nodes) > 20:
- print(f' ... and {len(nodes)-20} more nodes')
- " 2>/dev/null
- echo ""
- info "To get PrivateDnsHostname for a specific instance: aws sagemaker describe-cluster-node --cluster-name $CLUSTER --region $REGION --node-id <i-...> --query 'NodeDetails.PrivateDnsHostname' --output text"
- fi
- section "9. Triage Summary"
- echo ""
- if [[ ${#ISSUES_FOUND[@]} -eq 0 ]]; then
- echo -e " ${GREEN}${BOLD}No critical issues detected from available signals.${NC}"
- echo ""
- echo " Next steps:"
- echo " • If cluster is still failing: check cluster events above for error details"
- echo " • For node-level issues: use hyperpod-ssm skill to inspect nodes directly"
- echo " • For EFA issues: bash scripts/check-efa-sg.sh --cluster ${CLUSTER} --region ${REGION}"
- else
- echo -e " ${RED}${BOLD}Issues found (${#ISSUES_FOUND[@]}):${NC}"
- echo ""
- for priority in P0 P1 P2; do
- has_items=false
- for issue in "${ISSUES_FOUND[@]}"; do
- if [[ "$issue" == "${priority}|"* ]]; then
- if ! "$has_items"; then
- case "$priority" in
- P0) echo -e " ${RED}${BOLD}[$priority — Fix Immediately]${NC}" ;;
- P1) echo -e " ${YELLOW}${BOLD}[$priority — Fix Soon]${NC}" ;;
- P2) echo -e " ${BOLD}[$priority — Informational]${NC}" ;;
- esac
- has_items=true
- fi
- echo -e " → ${issue#*|}"
- fi
- done
- done
- echo ""
- echo -e " ${BOLD}Recommended next steps:${NC}"
- echo " 1. Address P0 issues first, then P1. Each issue above includes a"
- echo " pointer of the form '→ references/node-diagnostics-detail.md § X'."
- echo " 2. The hyperpod-node-debugger skill will open the referenced section"
- echo " and guide you through the fix with explicit approval."
- echo " 3. After fixing, re-run: bash scripts/triage-cluster.sh --cluster ${CLUSTER} --region ${REGION}"
- echo " 4. For shell access on nodes, use the hyperpod-ssm skill."
- fi
- echo ""
- echo -e "${BOLD}Cluster: ${CLUSTER} | Region: ${REGION} | Orchestrator: ${ORCHESTRATOR}${NC}"
- echo ""
- # Exit 1 only on critical (P0/P1) issues so CI / retry loops don't fail on
- # P2 informational findings. Fatal prerequisite failures exit 1 earlier at
- # argument-validation time.
- _critical=0
- for _issue in "${ISSUES_FOUND[@]}"; do
- case "${_issue%%|*}" in P0|P1) _critical=$((_critical+1)) ;; esac
- done
- [[ "$_critical" -eq 0 ]] && exit 0 || exit 1
plugins/sagemaker-ai/skills/hyperpod-slurm-debugger/scripts/slurm-diagnose.sh · Bundled script · 802 lines · 2 findings marked
- #!/usr/bin/env bash
- # slurm-diagnose.sh
- #
- # Read-only diagnostic for Slurm node-management issues on Amazon SageMaker HyperPod
- # Slurm clusters. Covers the scenarios documented in the HyperPod troubleshooting guide:
- #
- # A. Node DOWN / not responding
- # B. Node DOWN with reason "Node unexpectedly rebooted"
- # C. Controller state — slurmctld desync, plus the two folded triggers:
- # C (slurmdbd): accounting daemon connectivity
- # C (config): pending slurm.conf reconfiguration
- # D. Auto-recovery reason-string mismatches (Action:Reboot / Action:Replace)
- # E. HyperPod --auto-resume support and recent missed-resume detection
- #
- # Security model:
- # - All CLI inputs are validated against strict regexes at parse time.
- # - All AWS-derived values (instance IDs, group names, node names) are validated before
- # they reach any shell context — invalid values cause an immediate exit.
- # - Remote SSM payloads are base64-encoded literals; server-derived values are
- # prepended to the remote script as `export VAR='<jq @sh-quoted VALUE>'` lines so
- # they are never string-interpolated into shell commands.
- # - Local printf calls use `%s` with the data as a separate argument; format-string
- # attacks via server values are not possible.
- #
- # The script never mutates cluster state.
- #
- # Usage:
- # bash slurm-diagnose.sh --cluster <NAME-or-ARN> --region <REGION>
- # bash slurm-diagnose.sh --cluster <N> --region <R> --node <SLURM_NODE>
- # bash slurm-diagnose.sh --cluster <N> --region <R> --controller-group <NAME>
- #
- # Optional flags:
- # --node <SLURM_NODE> Scope inspection to a single Slurm node.
- # --controller-group <N> Override controller-group discovery (for self-managed
- # Slurm clusters where SlurmConfig is not set).
- # --no-color Plain output (no ANSI colors).
- set -euo pipefail
- CLUSTER=""
- REGION="${AWS_DEFAULT_REGION:-us-east-1}"
- TARGET_NODE=""
- CONTROLLER_GROUP_OVERRIDE=""
- USE_COLOR=true
- # --- Input-validation helpers -------------------------------------------------
- # Each validator prints the value if valid, exits non-zero if not. All callsites
- # capture into a local variable; failure aborts the script via `set -e`.
- # AWS region: lowercase letters, digits, dashes only.
- validate_region() {
- local v="${1-}"
- [[ "$v" =~ ^[a-z]{2,3}-[a-z]+-[0-9]+$ ]] || { echo "Error: invalid region: $v" >&2; exit 2; }
- printf '%s' "$v"
- }
- # HyperPod cluster name OR ARN. Names are 1-63 chars of [a-zA-Z0-9_-]; ARNs match the
- # documented SageMaker cluster ARN shape.
- validate_cluster() {
- local v="${1-}"
- if [[ "$v" =~ ^arn:aws[a-zA-Z-]*:sagemaker:[a-z0-9-]+:[0-9]{12}:cluster/[a-zA-Z0-9-]+$ ]]; then
- printf '%s' "$v"
- elif [[ "$v" =~ ^[a-zA-Z0-9_-]{1,63}$ ]]; then
- printf '%s' "$v"
- else
- echo "Error: invalid cluster name/ARN: $v" >&2
- exit 2
- fi
- }
- # Slurm node names on HyperPod follow the `ip-x-x-x-x` form, but admins may rename.
- # Allow [a-zA-Z0-9._-]+ with length 1..253; reject anything that could escape a shell.
- validate_node_name() {
- local v="${1-}"
- [[ "$v" =~ ^[a-zA-Z0-9._-]{1,253}$ ]] || { echo "Error: invalid node name: $v" >&2; exit 2; }
- printf '%s' "$v"
- }
- # EC2 instance IDs: i- followed by 8 or 17 hex characters. Documented and stable.
- validate_instance_id() {
- local v="${1-}"
- [[ "$v" =~ ^i-[a-f0-9]{8}([a-f0-9]{9})?$ ]] || { echo "Error: invalid instance ID: $v" >&2; exit 2; }
- printf '%s' "$v"
- }
- # Cluster ID (from ARN): lowercase alphanumeric, currently 12 chars (e.g.
- # qrmv6xhralg4). Allow 4..32 to be future-tolerant.
- validate_cluster_id() {
- local v="${1-}"
- [[ "$v" =~ ^[a-z0-9]{4,32}$ ]] || { echo "Error: invalid cluster ID: $v" >&2; exit 2; }
- printf '%s' "$v"
- }
- # Instance group name: SageMaker allows 1..63 chars [a-zA-Z0-9_-] per the
- # CreateCluster API.
- validate_group_name() {
- local v="${1-}"
- [[ "$v" =~ ^[a-zA-Z0-9_-]{1,63}$ ]] || { echo "Error: invalid instance group name: $v" >&2; exit 2; }
- printf '%s' "$v"
- }
- # --- Argument parsing ---------------------------------------------------------
- while [[ $# -gt 0 ]]; do
- case "$1" in
- --cluster)
- [[ $# -lt 2 ]] && { echo "Error: --cluster requires a value" >&2; exit 2; }
- CLUSTER=$(validate_cluster "$2"); shift 2 ;;
- --region)
- [[ $# -lt 2 ]] && { echo "Error: --region requires a value" >&2; exit 2; }
- REGION=$(validate_region "$2"); shift 2 ;;
- --node)
- [[ $# -lt 2 ]] && { echo "Error: --node requires a value" >&2; exit 2; }
- TARGET_NODE=$(validate_node_name "$2"); shift 2 ;;
- --controller-group)
- [[ $# -lt 2 ]] && { echo "Error: --controller-group requires a value" >&2; exit 2; }
- CONTROLLER_GROUP_OVERRIDE=$(validate_group_name "$2"); shift 2 ;;
- --no-color) USE_COLOR=false; shift ;;
- -h|--help)
- # Print every leading-comment line at the top of this file (lines 2..N until the
- # first non-comment line). Robust against future header edits.
- awk 'NR==1{next} /^#/{sub(/^# ?/,""); print; next} {exit}' "$0"
- exit 0 ;;
- --*) echo "Error: unknown flag: $1" >&2; exit 2 ;;
- *) echo "Error: unexpected positional argument: $1" >&2; exit 2 ;;
- esac
- done
- [[ -z "$CLUSTER" ]] && { echo "Error: --cluster is required" >&2; exit 2; }
- REGION=$(validate_region "$REGION") # validate even when sourced from env default
- # --- Prerequisite checks ------------------------------------------------------
- command -v aws >/dev/null 2>&1 || { echo "Error: aws CLI is required (v2 recommended)." >&2; exit 1; }
- command -v jq >/dev/null 2>&1 || { echo "Error: jq is required. Install with your package manager." >&2; exit 1; }
- # `unbuffer` (from the `expect` package) attaches a PTY to aws ssm start-session, which
- # avoids a known race where session-manager-plugin closes stdout before flushing and the
- # caller sees "Cannot perform start session: EOF" with empty output. Without it, every
- # SSM command silently returns empty, causing every downstream check to misreport.
- command -v unbuffer >/dev/null 2>&1 || {
- echo "Error: unbuffer (from the 'expect' package) is required." >&2
- echo " Install: sudo yum install expect | sudo apt install expect | brew install expect" >&2
- exit 1
- }
- # --- Output formatting --------------------------------------------------------
- if "$USE_COLOR"; then
- RED=$'\033[0;31m'; GREEN=$'\033[0;32m'; YELLOW=$'\033[1;33m'
- CYAN=$'\033[0;36m'; BOLD=$'\033[1m'; NC=$'\033[0m'
- else
- RED=''; GREEN=''; YELLOW=''; CYAN=''; BOLD=''; NC=''
- fi
- # All status helpers use %s with the message as a separate arg — never embed message
- # text into the format string. Strip ANSI escape sequences from incoming server data
- # so a malicious or buggy upstream cannot rewrite the operator's terminal.
- _sanitize() {
- # Drop ANSI CSI sequences and bell, but leave printable UTF-8 alone.
- sed -e 's/\x1b\[[0-9;?]*[a-zA-Z]//g' -e 's/\x07//g' -e 's/\r$//' <<< "${1-}"
- }
- section() { printf '\n%s%s=== %s ===%s\n' "$BOLD" "$CYAN" "$(_sanitize "$1")" "$NC"; }
- ok() { printf ' %s[PASS]%s %s\n' "$GREEN" "$NC" "$(_sanitize "$1")"; }
- warn() { printf ' %s[WARN]%s %s\n' "$YELLOW" "$NC" "$(_sanitize "$1")"; }
- bad() { printf ' %s[FAIL]%s %s\n' "$RED" "$NC" "$(_sanitize "$1")"; }
- info() { printf ' %s\n' "$(_sanitize "$1")"; }
- hint() { printf ' %s[NEXT]%s %s\n' "$CYAN" "$NC" "$(_sanitize "$1")"; }
- ISSUES=()
- NEXT_STEPS=()
- # --- Verify cluster + orchestrator --------------------------------------------
- section "1. Cluster identity"
- DESC=$(aws sagemaker describe-cluster --cluster-name "$CLUSTER" --region "$REGION" \
- --output json 2>&1) || { bad "cannot describe cluster: $DESC"; exit 1; }
- ORCH=$(jq -r '.Orchestrator // {} | keys[0] // "Slurm"' <<< "$DESC")
- if [[ "$ORCH" == "Eks" ]]; then
- bad "cluster uses EKS orchestrator - this skill is for Slurm only"
- info "use hyperpod-node-debugger or hyperpod-nccl instead"
- exit 1
- fi
- # Managed Slurm vs self-managed Slurm:
- # - Managed: DescribeCluster.Orchestrator.Slurm is present AND the cluster was created
- # with the SlurmConfig API parameter — InstanceGroups[].SlurmConfig.NodeType identifies
- # controllers, login nodes, workers. AWS docs treat this as the authoritative source.
- # - Self-managed: anything else. The customer brought their own Slurm setup via the
- # lifecycle scripts and InstanceGroups[].SlurmConfig is empty. The controller-group
- # name lives in /opt/ml/config/provisioning_parameters.json on every node, or the
- # customer can pass --controller-group <NAME>.
- HAS_SLURM_CONFIG=$(jq -r '
- any(.InstanceGroups[]?; (.SlurmConfig // {}) != {})
- ' <<< "$DESC")
- CLUSTER_NAME=$(jq -r '.ClusterName // "unknown"' <<< "$DESC")
- CLUSTER_STATUS=$(jq -r '.ClusterStatus // "unknown"' <<< "$DESC")
- if [[ "$HAS_SLURM_CONFIG" == "true" ]]; then
- ok "Managed Slurm cluster: $CLUSTER_NAME status=$CLUSTER_STATUS"
- else
- ok "Self-managed Slurm cluster: $CLUSTER_NAME status=$CLUSTER_STATUS"
- fi
- # Cluster ID from ARN. Validate before it gets embedded into SSM target strings.
- CLUSTER_ID=$(jq -r '.ClusterArn // "" | split("/") | last' <<< "$DESC")
- [[ -n "$CLUSTER_ID" ]] || { bad "cannot extract cluster ID from ARN"; exit 1; }
- CLUSTER_ID=$(validate_cluster_id "$CLUSTER_ID")
- # --- SSM remote-execution helper ----------------------------------------------
- #
- # `ssm_run` runs a command on a HyperPod node via SSM (read-only).
- #
- # Design notes:
- # 1. The remote script is base64-encoded locally and decoded remotely. The agent's
- # command parameter is a fixed `sh -c "echo <BASE64> | base64 -d | bash"`; the
- Note: Quoted as data in code: Mentions decoding base64 into a shell (no payload shown)
- Line 212 on GitHub
- # base64 string contains only [A-Za-z0-9+/=] and is safe inside double quotes.
- # Nothing from the script's caller appears unescaped in the SSM-agent's argv.
- # 2. Server-derived values that need to be visible to the remote script are passed
- # as named environment variables (`VAR=VALUE` trailing args). Each value is run
- # through `jq @sh` (single-quoted shell-safe encoding with `'\''` escapes) and
- # prepended to the remote script as `export VAR='<safely-quoted>'; ...`. The remote
- # shell reads them as `$NODE`, `$NODELIST`, etc. — values never reach a remote
- # shell-eval context as raw interpolated text.
- # 3. `unbuffer` is required to defeat the SSM "Cannot perform start session: EOF"
- # race; the prerequisite check above guarantees it's present.
- # 4. Returns the underlying aws-cli exit code so callers can distinguish transport
- # failures from successful empty output.
- #
- # Usage:
- # ssm_run TARGET REMOTE_SCRIPT [VAR=VALUE ...]
- ssm_run() {
- local target="$1"; shift
- local script="$1"; shift
- local export_block="" raw_kv key val safe_val
- for raw_kv in "$@"; do
- [[ "$raw_kv" =~ ^([A-Za-z_][A-Za-z0-9_]*)=(.*)$ ]] || {
- echo "ssm_run: invalid VAR=VALUE: $raw_kv" >&2
- return 2
- }
- key="${BASH_REMATCH[1]}"
- val="${BASH_REMATCH[2]}"
- # jq's @sh produces single-quoted shell-safe text with embedded `'\''` escapes.
- safe_val=$(jq -nr --arg v "$val" '$v | @sh')
- export_block+="export ${key}=${safe_val}; "
- done
- local full_script="${export_block}${script}"
- local b64
- if base64 --help 2>&1 | grep -q '\-w'; then
- b64=$(printf '%s' "$full_script" | base64 -w0)
- else
- b64=$(printf '%s' "$full_script" | base64 -b0)
- fi
- local wrapper="sh -c \"echo $b64 | base64 -d | bash\""
- Note: Quoted as data in code: Decodes base64 and runs the result in a shell
- Line 250 on GitHub
- local params
- params=$(jq -nc --arg c "$wrapper" '{command: [$c]}')
- local out rc=0
- out=$(unbuffer aws ssm start-session --region "$REGION" --target "$target" \
- --document-name AWS-StartNonInteractiveCommand \
- --parameters "$params" 2>&1) || rc=$?
- # NOTE: do NOT strip 'Cannot perform start session' here — that line is the
- # SSM transport-failure signal that ssm_transport_failed() detects. Only filter
- # benign session chrome ('Starting session' / 'Exiting session') and ANSI escapes.
- printf '%s' "$out" \
- | sed -e 's/\x1b\[[0-9;?]*[a-zA-Z]//g' \
- -e '/^Starting session/d' \
- -e '/^Exiting session/d'
- return "$rc"
- }
- # Returns 0 if the SSM raw output indicates a transport-layer failure (no command
- # output, session refused, EOF before flush) — distinct from "command ran and returned
- # nothing." Used to bail out early rather than misreport every downstream check.
- ssm_transport_failed() {
- local raw="${1-}"
- grep -qiE 'Cannot perform start session|TargetNotConnected|InvalidTarget|AccessDeniedException|UnauthorizedOperation' <<< "$raw"
- }
- # --- Find controller node -----------------------------------------------------
- NODES_JSON=$(aws sagemaker list-cluster-nodes --cluster-name "$CLUSTER" --region "$REGION" \
- --output json 2>&1) || { bad "list-cluster-nodes failed: $NODES_JSON"; exit 1; }
- # Discovery priority:
- # 1. --controller-group <NAME> (operator override — always wins)
- # 2. InstanceGroups[].SlurmConfig.NodeType == "Controller" (managed-Slurm authoritative)
- # 3. /opt/ml/config/provisioning_parameters.json on a probe node (self-managed fallback)
- # 4. Refuse to guess — print available groups and exit.
- # We never guess based on instance-group naming — that's a lifecycle-script convention,
- # not a guarantee, and getting it wrong sends every command to a non-controller.
- CONTROLLER_GROUP=""
- CONTROLLER_DISCOVERY_METHOD=""
- # (1) Operator override — always wins.
- if [[ -n "$CONTROLLER_GROUP_OVERRIDE" ]]; then
- CONTROLLER_GROUP="$CONTROLLER_GROUP_OVERRIDE"
- CONTROLLER_DISCOVERY_METHOD="--controller-group flag"
- fi
- # (2) Managed-Slurm authoritative source.
- if [[ -z "$CONTROLLER_GROUP" && "$HAS_SLURM_CONFIG" == "true" ]]; then
- CONTROLLER_GROUP=$(jq -r '
- .InstanceGroups[]?
- | select((.SlurmConfig.NodeType // "") == "Controller")
- | .InstanceGroupName' <<< "$DESC" | head -1)
- if [[ -n "$CONTROLLER_GROUP" ]]; then
- CONTROLLER_DISCOVERY_METHOD="DescribeCluster.SlurmConfig"
- fi
- fi
- # (3) Self-managed: read provisioning_parameters.json from any node.
- # The lifecycle-script convention is that this file is dropped at the same path on every
- # node, so we pick any node arbitrarily, SSM in, and read the controller_group field.
- if [[ -z "$CONTROLLER_GROUP" ]]; then
- PROBE_ID=$(jq -r '.ClusterNodeSummaries[0].InstanceId // ""' <<< "$NODES_JSON")
- PROBE_GROUP=$(jq -r '.ClusterNodeSummaries[0].InstanceGroupName // ""' <<< "$NODES_JSON")
- if [[ -n "$PROBE_ID" && -n "$PROBE_GROUP" ]]; then
- PROBE_ID_V=$(validate_instance_id "$PROBE_ID")
- PROBE_GROUP_V=$(validate_group_name "$PROBE_GROUP")
- PROBE_TARGET="sagemaker-cluster:${CLUSTER_ID}_${PROBE_GROUP_V}-${PROBE_ID_V}"
- # Field name varies between lifecycle-script generations — try both.
- PROV_GROUP=$(ssm_run "$PROBE_TARGET" \
- 'jq -r ".controller_group // .ControllerGroup // empty" /opt/ml/config/provisioning_parameters.json 2>/dev/null' \
- 2>/dev/null | tr -d '\r\n' || true)
- if [[ -n "$PROV_GROUP" ]]; then
- MATCHED=$(jq -r --arg g "$PROV_GROUP" \
- '[.ClusterNodeSummaries[]? | select(.InstanceGroupName == $g)] | length' <<< "$NODES_JSON")
- if [[ "$MATCHED" -gt 0 ]]; then
- CONTROLLER_GROUP="$PROV_GROUP"
- CONTROLLER_DISCOVERY_METHOD="provisioning_parameters.json on $PROBE_ID_V"
- fi
- fi
- fi
- fi
- # (4) Out of options — refuse to guess. Tell the operator how to unblock.
- if [[ -z "$CONTROLLER_GROUP" ]]; then
- bad "cannot identify the Slurm controller instance group"
- if [[ "$HAS_SLURM_CONFIG" == "true" ]]; then
- info "no InstanceGroup has SlurmConfig.NodeType=Controller in DescribeCluster output"
- info "this is unexpected for a managed-Slurm cluster — verify the cluster was"
- info "created with the SlurmConfig parameter, or pass --controller-group <NAME>."
- else
- info "self-managed Slurm cluster — provisioning_parameters.json was not readable"
- info "from a probe node, and no --controller-group flag was provided."
- info ""
- info "Resolve by either:"
- info " 1. inspecting the head node manually:"
- info " aws ssm start-session --target $PROBE_TARGET --region $REGION"
- info " cat /opt/ml/config/provisioning_parameters.json | jq ."
- info " 2. re-running with the controller group's name:"
- info " --controller-group <INSTANCE_GROUP_NAME>"
- info ""
- info "Available instance groups in this cluster:"
- jq -r '.ClusterNodeSummaries[] | " - " + .InstanceGroupName + " (" + .InstanceId + ")"' \
- <<< "$NODES_JSON" | sort -u
- fi
- exit 1
- fi
- CONTROLLER_GROUP=$(validate_group_name "$CONTROLLER_GROUP")
- # Pick the first node from the controller group.
- CONTROLLER_ID=$(jq -r --arg g "$CONTROLLER_GROUP" \
- '.ClusterNodeSummaries[]? | select(.InstanceGroupName == $g) | .InstanceId' <<< "$NODES_JSON" | head -1)
- [[ -n "$CONTROLLER_ID" ]] || { bad "controller group $CONTROLLER_GROUP has no nodes"; exit 1; }
- CONTROLLER_ID=$(validate_instance_id "$CONTROLLER_ID")
- ok "controller node: $CONTROLLER_ID (group=$CONTROLLER_GROUP, source=$CONTROLLER_DISCOVERY_METHOD)"
- SSM_HEAD="sagemaker-cluster:${CLUSTER_ID}_${CONTROLLER_GROUP}-${CONTROLLER_ID}"
- # --- Collect Slurm state from head node ---------------------------------------
- section "2. Slurm cluster state (from head node)"
- SSM_PROBE=$(ssm_run "$SSM_HEAD" 'echo SSM_OK' || true)
- if ! grep -q '^SSM_OK$' <<< "$SSM_PROBE"; then
- bad "cannot reach head node via SSM — every downstream check would be unreliable"
- if ssm_transport_failed "$SSM_PROBE"; then
- info " transport error detected (TargetNotConnected, AccessDenied, or EOF race)"
- fi
- info " reproduce manually with the same target and region:"
- info " aws ssm start-session --target $SSM_HEAD --region $REGION"
- info " if that fails, route to the hyperpod-ssm skill before retrying."
- exit 1
- fi
- ok "SSM transport to head node working"
- SINFO_OUT=$(ssm_run "$SSM_HEAD" 'sinfo -h -o "%N|%T|%E" 2>&1 | head -200' || true)
- if [[ $(printf '%s\n' "$SINFO_OUT" | wc -l) -ge 200 ]]; then
- warn "sinfo output reached the 200-line cap — node-state results may be truncated on this large cluster"
- fi
- if grep -qi 'command not found' <<< "$SINFO_OUT"; then
- bad "sinfo not installed on head node — Slurm lifecycle script may not have run"
- info "verify on the node: systemctl status slurmctld; ls /opt/slurm*/etc /etc/slurm 2>/dev/null"
- exit 1
- fi
- if [[ -z "$SINFO_OUT" ]]; then
- warn "sinfo returned no rows — empty cluster, or controller not yet responding"
- fi
- # Parse sinfo lines. Node names from sinfo are server-controlled; validate before they
- # can be embedded into any later command. Values that fail validation are dropped, not
- # trusted; we report the count of skipped entries so the operator notices.
- DOWN_NODES=()
- REBOOT_NODES=()
- FAIL_NODES=()
- BAD_REASON_NODES=()
- SKIPPED_INVALID=0
- while IFS='|' read -r node state reason; do
- [[ -z "$node" ]] && continue
- if ! [[ "$node" =~ ^[a-zA-Z0-9._-]{1,253}$ ]]; then
- SKIPPED_INVALID=$((SKIPPED_INVALID+1))
- continue
- fi
- # Reasons can contain spaces and punctuation; allow them but strip ANSI/control chars.
- reason="$(_sanitize "$reason")"
- if grep -qi 'fail' <<< "$state"; then
- if [[ "$reason" =~ ^Action:(Reboot|Replace)$ ]]; then
- FAIL_NODES+=("$node|$reason")
- elif grep -qiE 'action[ :_-]*re(boot|place)|reboot|replace' <<< "$reason"; then
- BAD_REASON_NODES+=("$node|$reason")
- fi
- fi
- if grep -qiE 'down|drain' <<< "$state"; then
- if grep -qi 'unexpectedly rebooted' <<< "$reason"; then
- REBOOT_NODES+=("$node")
- else
- DOWN_NODES+=("$node|$reason")
- fi
- fi
- done <<< "$SINFO_OUT"
- [[ "$SKIPPED_INVALID" -gt 0 ]] && warn "$SKIPPED_INVALID sinfo row(s) had invalid node names and were ignored"
- if [[ ${#DOWN_NODES[@]} -eq 0 && ${#REBOOT_NODES[@]} -eq 0 && ${#FAIL_NODES[@]} -eq 0 && ${#BAD_REASON_NODES[@]} -eq 0 ]]; then
- ok "all nodes in healthy Slurm states"
- else
- [[ ${#DOWN_NODES[@]} -gt 0 ]] && bad "${#DOWN_NODES[@]} node(s) DOWN/DRAIN (Section A)"
- [[ ${#REBOOT_NODES[@]} -gt 0 ]] && bad "${#REBOOT_NODES[@]} node(s) with 'unexpectedly rebooted' (Section B)"
- [[ ${#FAIL_NODES[@]} -gt 0 ]] && warn "${#FAIL_NODES[@]} node(s) in fail state with valid Action:* reason (HyperPod recovery in progress)"
- [[ ${#BAD_REASON_NODES[@]} -gt 0 ]] && bad "${#BAD_REASON_NODES[@]} node(s) in fail state with non-matching reason (Section D)"
- fi
- # --- Section D: Action:* reason-string validation -----------------------------
- if [[ ${#BAD_REASON_NODES[@]} -gt 0 ]]; then
- section "D. Reason-string mismatch — HyperPod auto-recovery will NOT trigger"
- for entry in "${BAD_REASON_NODES[@]}"; do
- n="${entry%%|*}"; r="${entry#*|}"
- bad "$n: reason='$r'"
- done
- info "the reason field must match exactly: Action:Reboot or Action:Replace"
- info "(case-sensitive, no spaces, no trailing punctuation)"
- hint "for re-issue procedure, see:"
- info " https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-replace-faulty-instance.html"
- info " references/slurm-details.md#action-reason-string-validation"
- ISSUES+=("bad-action-reason")
- NEXT_STEPS+=("see AWS replace-faulty-instance docs (link above)")
- fi
- # --- Detect in-progress HyperPod replacements (informational) -----------------
- if [[ ${#FAIL_NODES[@]} -gt 0 ]]; then
- section " HyperPod recovery in progress (do not interfere)"
- for entry in "${FAIL_NODES[@]}"; do
- n="${entry%%|*}"; r="${entry#*|}"
- info "$n ($r)"
- done
- info "AWS docs: do NOT change node state or restart slurmctld until this completes."
- info "If a replacement seems stuck > 30 min, see:"
- info " https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-replace-faulty-instance.html"
- fi
- # --- Check controller health --------------------------------------------------
- section "3. slurmctld health"
- PING_OUT=$(ssm_run "$SSM_HEAD" 'scontrol ping 2>&1' || true)
- PING_FIRST_LINE=$(head -1 <<< "$PING_OUT" | tr -d '\r')
- if grep -qi 'UP' <<< "$PING_OUT"; then
- ok "slurmctld responding: $(tr '\n' ' ' <<< "$PING_OUT")"
- elif [[ -z "$PING_OUT" ]] || ssm_transport_failed "$PING_OUT"; then
- warn "could not get a response from scontrol ping — cannot determine controller health"
- info "this is most likely an SSM transport problem, not a hung controller"
- info "do NOT restart slurmctld based on this finding alone"
- elif grep -qi 'DOWN' <<< "$PING_OUT"; then
- bad "slurmctld reports DOWN: $PING_FIRST_LINE"
- ISSUES+=("controller-hung")
- NEXT_STEPS+=("controller restart — see references/slurm-details.md#-c-controller-state--diagnostic-context")
- else
- bad "slurmctld responded with an unrecognized status: $PING_FIRST_LINE"
- ISSUES+=("controller-hung")
- NEXT_STEPS+=("inspect logs first; controller restart only if logs confirm a hang")
- fi
- # --- Section C-1: slurmdbd connectivity (controller-state restart trigger) ---
- section "C (slurmdbd): accounting daemon connectivity"
- DBD_OUT=$(ssm_run "$SSM_HEAD" 'sacctmgr -i show stats 2>&1 | head -20' || true)
- if grep -qiE 'unable to contact|connection refused|cannot connect|no slurmdbd' <<< "$DBD_OUT"; then
- bad "slurmctld cannot reach slurmdbd"
- info "$(head -3 <<< "$DBD_OUT")"
- hint "diagnostic and recovery procedure:"
- info " https://slurm.schedmd.com/accounting.html"
- info " references/slurm-details.md#slurmdbd-connectivity"
- ISSUES+=("slurmdbd-disconnected")
- NEXT_STEPS+=("restore slurmdbd connectivity (see AWS / Slurm docs linked above)")
- elif grep -qiE 'rollup|rpc' <<< "$DBD_OUT"; then
- ok "slurmdbd reachable"
- else
- warn "could not determine slurmdbd state from sacctmgr output"
- info "if accounting is configured, run on the head node: sacctmgr show stats"
- fi
- # --- Section C-2: pending slurm.conf reconfiguration (controller-state restart trigger) ---
- # HyperPod's slurm.conf lives at /opt/slurm-<version>/etc/slurm.conf rather than the
- # upstream /etc/slurm/slurm.conf, so the remote script asks scontrol where the live
- # config is. The output is a `<conf-mtime>|<ctld-start>|<conf-path>` line that we
- # match strictly with a regex before parsing.
- section "C (config): slurm.conf freshness"
- read -r -d '' F_REMOTE <<'REMOTE_F' || true
- set -e
- # nosemgrep: bash.lang.correctness.unquoted-expansion.unquoted-variable-expansion-in-command
- _CONF="$(scontrol show config 2>/dev/null | awk -F= '/^SLURM_CONF/ {gsub(/ /,"",$2); print $2; exit}')"
- CONF_MTIME=0
- if [ -n "$_CONF" ] && [ -r "$_CONF" ]; then
- CONF_MTIME=$(stat -c %Y "$_CONF" 2>/dev/null || echo 0)
- fi
- CTLD_TS=$(systemctl show slurmctld -p ActiveEnterTimestamp --value 2>/dev/null || true)
- CTLD_START=0
- if [ -n "$CTLD_TS" ]; then
- CTLD_START=$(date -d "$CTLD_TS" +%s 2>/dev/null || echo 0)
- fi
- printf 'F_RESULT|%s|%s|%s\n' "${CONF_MTIME}" "${CTLD_START}" "${_CONF}"
- REMOTE_F
- F_LINE=$(ssm_run "$SSM_HEAD" "$F_REMOTE" 2>/dev/null | grep -E '^F_RESULT\|[0-9]+\|[0-9]+\|' | head -1 || true)
- if [[ "$F_LINE" =~ ^F_RESULT\|([0-9]+)\|([0-9]+)\|(.*)$ ]]; then
- CONF_MTIME="${BASH_REMATCH[1]}"
- CTLD_START="${BASH_REMATCH[2]}"
- CONF_PATH="${BASH_REMATCH[3]}"
- # CONF_PATH must be a real-looking absolute path before we put it into operator-
- # facing recommendations. Reject anything that has shell-active characters.
- if ! [[ "$CONF_PATH" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
- warn "slurm.conf path returned by remote did not validate; skipping freshness check"
- elif [[ "$CONF_MTIME" -gt "$CTLD_START" && "$CTLD_START" -gt 0 ]]; then
- DELTA=$((CONF_MTIME - CTLD_START))
- warn "$CONF_PATH modified ${DELTA}s after slurmctld last started — config may be stale in memory"
- hint "for the reload-vs-restart decision and procedure, see:"
- info " https://slurm.schedmd.com/scontrol.html"
- info " https://slurm.schedmd.com/slurm.conf.html"
- info " references/slurm-details.md#scontrol-reconfigure-vs-restart"
- ISSUES+=("stale-conf")
- NEXT_STEPS+=("review reload procedure in linked docs")
- else
- ok "slurm.conf older than slurmctld start time — no pending reconfigure"
- fi
- else
- warn "could not determine slurm.conf vs slurmctld timestamps"
- fi
- # --- Check for stuck jobs -----------------------------------------------------
- section "4. Job queue health"
- SQUEUE_OUT=$(ssm_run "$SSM_HEAD" 'squeue -h -o "%i|%T|%r" 2>&1 | head -200' || true)
- if [[ $(printf '%s\n' "$SQUEUE_OUT" | wc -l) -ge 200 ]]; then
- warn "squeue output reached the 200-line cap — stuck-job counts below may underreport on this large cluster"
- fi
- STUCK_PENDING=0
- STUCK_COMPLETING=0
- while IFS='|' read -r jobid state reason; do
- [[ -z "$jobid" ]] && continue
- [[ "$state" == "PENDING" && "$reason" == "Resources" ]] && STUCK_PENDING=$((STUCK_PENDING+1))
- [[ "$state" == "COMPLETING" ]] && STUCK_COMPLETING=$((STUCK_COMPLETING+1))
- done <<< "$SQUEUE_OUT"
- if [[ $STUCK_PENDING -gt 0 ]]; then
- warn "$STUCK_PENDING job(s) PENDING with Reason=Resources"
- if [[ ${#DOWN_NODES[@]} -eq 0 ]]; then
- ISSUES+=("stuck-pending-with-idle-nodes")
- NEXT_STEPS+=("controller restart — Section C")
- fi
- fi
- if [[ $STUCK_COMPLETING -gt 0 ]]; then
- bad "$STUCK_COMPLETING job(s) stuck in COMPLETING"
- ISSUES+=("stuck-completing")
- NEXT_STEPS+=("controller restart — Section C")
- fi
- [[ $STUCK_PENDING -eq 0 && $STUCK_COMPLETING -eq 0 ]] && ok "no stuck jobs"
- # --- Per-node inspection (read-only) ------------------------------------------
- inspect_node() {
- local slurm_node="$1"
- # Defense-in-depth: validate again at the boundary even though all upstream paths
- # validate. Cheap, and catches future refactors that miss a callsite.
- slurm_node=$(validate_node_name "$slurm_node")
- local instance_id group ssm_target
- # PrivateDnsName looks like `ip-10-1-2-3.us-west-2.compute.internal`. The strict
- # `<name>.` match handles the default `ip-x-x-x-x` form and rejects the false
- # positive where node `ip-10-1-2-3` would otherwise also match
- # `ip-10-1-2-30.<region>.compute.internal`.
- instance_id=$(jq -r --arg dns "$slurm_node" '
- .ClusterNodeSummaries[]?
- | select((.PrivateDnsName // "") | startswith($dns + "."))
- | .InstanceId' <<< "$NODES_JSON" | head -1)
- if [[ -z "$instance_id" ]]; then
- if [[ ! "$slurm_node" =~ ^ip-[0-9]+-[0-9]+-[0-9]+-[0-9]+$ ]]; then
- warn "$slurm_node: not in the default ip-X-X-X-X form — Slurm-node-name → instance-ID auto-mapping needs DNS lookup or scontrol show node, neither cheap from here. Pass --target-instance-id <i-xxx> if you have it, or look up via 'scontrol show node $slurm_node | grep NodeAddr' on the controller."
- else
- warn "$slurm_node: cannot map to instance ID (PrivateDnsName mismatch — verify node is in this cluster)"
- fi
- return
- fi
- instance_id=$(validate_instance_id "$instance_id")
- group=$(jq -r --arg id "$instance_id" \
- '.ClusterNodeSummaries[] | select(.InstanceId==$id) | .InstanceGroupName // ""' <<< "$NODES_JSON")
- group=$(validate_group_name "$group")
- ssm_target="sagemaker-cluster:${CLUSTER_ID}_${group}-${instance_id}"
- local slurmd_status disk mem rpc_check
- slurmd_status=$(ssm_run "$ssm_target" 'systemctl is-active slurmd 2>&1' | tr -d '\r\n' || true)
- disk=$(ssm_run "$ssm_target" 'df -h / | awk "NR==2 {print \$5}"' | tr -d '\r\n' || true)
- mem=$(ssm_run "$ssm_target" 'free -h | awk "/Mem:/ {print \$3\"/\"\$2}"' | tr -d '\r\n' || true)
- # Slurm-RPC reachability: srun -w "$NODE" hostname. The remote script reads $NODE
- # from the environment, so the slurm node name is never string-interpolated into
- # the remote shell — it lives in env-var space the whole way.
- rpc_check=$(ssm_run "$SSM_HEAD" 'timeout 10 srun --immediate=5 -w "$NODE" hostname 2>&1 | tail -1' \
- "NODE=$slurm_node" | tr -d '\r\n' || true)
- info "$slurm_node ($instance_id): slurmd=$slurmd_status disk=$disk mem=$mem"
- info " srun RPC: ${rpc_check:-<no output>}"
- local disk_num="${disk%\%}"
- if [[ "$disk_num" =~ ^[0-9]+$ && "$disk_num" -ge 95 ]]; then
- bad " $slurm_node: root volume ${disk} — clean up before any restart"
- info " HyperPod storage layout: https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
- ISSUES+=("disk-full-$slurm_node")
- NEXT_STEPS+=("clean disk on $slurm_node before recovery")
- fi
- if [[ "$slurmd_status" != "active" ]]; then
- bad " $slurm_node: slurmd is '$slurmd_status'"
- info " for recovery procedure, see:"
- info " https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
- fi
- if [[ -n "$rpc_check" ]] && grep -qiE 'auth|munge|invalid' <<< "$rpc_check"; then
- bad " $slurm_node: srun reports auth/munge error — slurmd-controller trust broken"
- info " for munge troubleshooting, see Slurm authentication docs:"
- info " https://slurm.schedmd.com/authentication.html"
- fi
- }
- if [[ -n "$TARGET_NODE" ]]; then
- section "5. Inspecting node: $TARGET_NODE"
- inspect_node "$TARGET_NODE"
- elif [[ ${#DOWN_NODES[@]} -gt 0 || ${#REBOOT_NODES[@]} -gt 0 ]]; then
- section "5. Inspecting affected nodes"
- for entry in "${DOWN_NODES[@]-}"; do
- [[ -z "$entry" ]] && continue
- inspect_node "${entry%%|*}"
- done
- for n in "${REBOOT_NODES[@]-}"; do
- [[ -z "$n" ]] && continue
- inspect_node "$n"
- done
- fi
- # --- Section E: HyperPod auto-resume support + recent missed-resume detection ---
- section "E. Auto-resume support"
- AR_HELP=$(ssm_run "$SSM_HEAD" 'srun --help 2>&1 | grep -i auto-resume | head -3' || true)
- if [[ -n "$AR_HELP" ]]; then
- ok "srun --auto-resume is available on this cluster"
- else
- warn "srun --auto-resume not found in srun --help output"
- info "this AMI / Slurm build may predate HyperPod auto-resume support"
- info "see: references/slurm-details.md#hyperpod-auto-resume"
- ISSUES+=("auto-resume-unsupported")
- NEXT_STEPS+=("upgrade the cluster AMI / Slurm package to enable --auto-resume")
- fi
- read -r -d '' G_FAILS <<'REMOTE_G' || true
- sacct -X -n --starttime=now-6hours \
- -o JobID,State,ExitCode,NodeList \
- --state=NODE_FAIL,FAILED 2>/dev/null \
- | awk 'NF>=4 && $4!~/None/ {print $1"|"$2"|"$4}' | head -50
- REMOTE_G
- RECENT_FAILS=$(ssm_run "$SSM_HEAD" "$G_FAILS" 2>/dev/null || true)
- MISSED_AR=()
- NOW_EPOCH=$(date +%s)
- while IFS='|' read -r jobid state nodelist; do
- [[ -z "$jobid" ]] && continue
- # Only single-node failures — multi-node lists need a real range expander.
- [[ "$nodelist" == *,* || "$nodelist" == *\[* ]] && continue
- # Validate before passing to remote.
- if ! [[ "$nodelist" =~ ^[a-zA-Z0-9._-]{1,253}$ ]]; then
- continue
- fi
- # A successful HyperPod replace clears the node's Reason field once the new instance
- # registers, so grepping for "Action:Replace" is unreliable. Detect a recent replace
- # by comparing scontrol show node's BootTime to wall-clock: a fresh BootTime within
- # the last 6h that's later than the failed-job's End time strongly suggests the node
- # was replaced (or rebooted) after the job died.
- BOOT_LINE=$(ssm_run "$SSM_HEAD" 'scontrol show node "$NODE" 2>/dev/null | tr " " "\n" | grep "^BootTime="' \
- "NODE=$nodelist" | head -1 | tr -d '\r\n' || true)
- BOOT_STR="${BOOT_LINE#BootTime=}"
- [[ -z "$BOOT_STR" || "$BOOT_STR" == "Unknown" ]] && continue
- BOOT_EPOCH=$(date -d "$BOOT_STR" +%s 2>/dev/null || echo 0)
- [[ "$BOOT_EPOCH" =~ ^[0-9]+$ && "$BOOT_EPOCH" -gt 0 ]] || continue
- AGE=$((NOW_EPOCH - BOOT_EPOCH))
- if [[ $AGE -ge 0 && $AGE -le 21600 ]]; then # 6h window
- MISSED_AR+=("$jobid|$state|$nodelist|$BOOT_STR")
- fi
- done <<< "$RECENT_FAILS"
- if [[ ${#MISSED_AR[@]} -gt 0 ]]; then
- bad "${#MISSED_AR[@]} recent job(s) failed on a node that was rebooted/replaced shortly after — possible missed auto-resume:"
- for entry in "${MISSED_AR[@]}"; do
- IFS='|' read -r jobid state nodelist boot <<< "$entry"
- info " job $jobid ($state) on $nodelist (node BootTime=$boot)"
- done
- info "(heuristic: node BootTime is within the last 6h, suggesting a replace or reboot)"
- hint "verify the launch command used srun --auto-resume=1 (NOT just sbatch):"
- info " sacct -j <JOBID> -o JobID,JobName,Submit,Start,End,State,ExitCode,NodeList -X"
- info " scontrol show job <JOBID> # only if still in the controller's recent history"
- info "see: references/slurm-details.md#hyperpod-auto-resume"
- ISSUES+=("missed-auto-resume")
- NEXT_STEPS+=("verify --auto-resume=1 is on the srun line, not just sbatch")
- elif [[ -n "$RECENT_FAILS" ]]; then
- ok "recent failed jobs do not match the missed-auto-resume pattern"
- else
- ok "no recent NODE_FAIL / FAILED jobs in the last 6h"
- fi
- # --- Findings → documentation links ------------------------------------------
- # This skill is diagnostic-only. It never prints a remediation command. For each
- # finding, point the user at the authoritative doc and let them act.
- section "Where to read next"
- if [[ ${#REBOOT_NODES[@]} -gt 0 ]]; then
- hint "Section B — nodes flagged 'unexpectedly rebooted':"
- for n in "${REBOOT_NODES[@]}"; do
- info " $n"
- done
- info " HyperPod Slurm troubleshooting:"
- info " https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
- info " diagnostic context: references/slurm-details.md#-b-unexpected-reboot--diagnostic-context"
- fi
- if [[ ${#DOWN_NODES[@]} -gt 0 ]]; then
- hint "Section A — nodes DOWN/DRAIN:"
- for entry in "${DOWN_NODES[@]}"; do
- n="${entry%%|*}"; r="${entry#*|}"
- info " $n (reason: $r)"
- done
- info " HyperPod Slurm troubleshooting:"
- info " https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
- info " if the node flaps after a manual recovery → route to hyperpod-node-debugger"
- fi
- CTRL_RESTART_REASON=""
- ISSUES_STR=" ${ISSUES[*]-} "
- [[ "$ISSUES_STR" == *" controller-hung "* ]] && CTRL_RESTART_REASON="scontrol ping failed"
- [[ "$ISSUES_STR" == *" stuck-completing "* ]] && CTRL_RESTART_REASON="${CTRL_RESTART_REASON:+$CTRL_RESTART_REASON, }jobs stuck COMPLETING"
- [[ "$ISSUES_STR" == *" stuck-pending-with-idle-nodes "* ]] && CTRL_RESTART_REASON="${CTRL_RESTART_REASON:+$CTRL_RESTART_REASON, }jobs PENDING with idle nodes"
- if [[ -n "$CTRL_RESTART_REASON" ]]; then
- hint "Section C — controller-state issue ($CTRL_RESTART_REASON):"
- info " Slurm slurmctld(8) — for what is preserved across a controller restart:"
- info " https://slurm.schedmd.com/slurmctld.html"
- info " HyperPod Slurm troubleshooting:"
- info " https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md"
- if [[ ${#FAIL_NODES[@]} -gt 0 ]]; then
- warn "HyperPod recovery is in progress on:"
- for entry in "${FAIL_NODES[@]}"; do
- n="${entry%%|*}"
- info " $n"
- done
- info "AWS docs warn against changing node state or restarting slurmctld during a"
- info "replacement; wait for it to complete, then re-run this script."
- fi
- info " diagnostic context: references/slurm-details.md#-c-controller-state--diagnostic-context"
- fi
- if [[ "$ISSUES_STR" == *" missed-auto-resume "* ]]; then
- hint "Section E — recent job failed on a node that was later replaced:"
- info " the most common cause is --auto-resume on sbatch instead of srun."
- info " Use SageMaker HyperPod auto-resume:"
- info " https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-auto-resume.html"
- info " diagnostic context: references/slurm-details.md#hyperpod-auto-resume"
- fi
- # --- Summary ------------------------------------------------------------------
- section "Summary"
- printf ' Issues detected: %d\n' "${#ISSUES[@]-0}"
- if [[ ${#ISSUES[@]-0} -eq 0 ]]; then
- ok "cluster Slurm state is healthy"
- else
- echo ""
- echo " Findings:"
- for i in "${ISSUES[@]}"; do
- info "- $i"
- done
- fi
- if [[ ${#NEXT_STEPS[@]-0} -gt 0 ]]; then
- echo ""
- echo " Where to read next:"
- for s in "${NEXT_STEPS[@]}"; do
- info "- $s"
- done
- fi
plugins/sagemaker-ai/skills/hyperpod-version-checker/scripts/hyperpod_check_versions.sh · Bundled script · 556 lines · 2 findings marked
- #!/usr/bin/env bash
- # HyperPod Version Checker - Detect software component versions on HyperPod cluster nodes
- #
- # Checks: NVIDIA driver, CUDA, cuDNN, NCCL, EFA, AWS OFI NCCL, GDRCopy, MPI,
- # Neuron SDK, Python, PyTorch, container runtime
- # Works on both EKS and Slurm HyperPod clusters.
- #
- # Usage: bash hyperpod_check_versions.sh [--json] [--no-color] [--output FILE]
- command -v jq >/dev/null 2>&1 || { echo "Error: jq is required but not installed" >&2; exit 1; }
- # --- Defaults ---
- JSON_OUTPUT=false
- USE_COLOR=true
- OUTPUT_FILE=""
- # --- Parse args ---
- while [[ $# -gt 0 ]]; do
- case "$1" in
- --json) JSON_OUTPUT=true; shift ;;
- --no-color) USE_COLOR=false; shift ;;
- --output|-o) OUTPUT_FILE="$2"; shift 2 ;;
- -h|--help)
- echo "Usage: bash hyperpod_check_versions.sh [--json] [--no-color] [--output FILE]"
- echo " --json Output ONLY JSON to stdout (text report still saved to file)"
- echo " --no-color Disable color output"
- echo " --output/-o Write report to FILE (default: component_versions_<host>_<time>.txt)"
- exit 0 ;;
- *) echo "Unknown option: $1"; exit 1 ;;
- esac
- done
- # --- Color setup ---
- if [[ "$USE_COLOR" == "true" ]] && [ -t 1 ] && [[ "$JSON_OUTPUT" != "true" ]]; then
- GREEN='\033[0;32m'; YELLOW='\033[1;33m'; BLUE='\033[0;34m'; NC='\033[0m'
- else
- GREEN=''; YELLOW=''; BLUE=''; NC=''
- fi
- # --- Output file ---
- if [ -z "$OUTPUT_FILE" ]; then
- OUTPUT_FILE="component_versions_$(hostname)_$(date +%Y%m%d_%H%M%S).txt"
- fi
- # --- Helpers ---
- # In JSON mode: text goes only to file. Otherwise: both console and file.
- log() {
- local stripped
- stripped=$(printf '%b\n' "$@" | sed 's/\x1b\[[0-9;]*m//g')
- echo "$stripped" >> "$OUTPUT_FILE"
- if [[ "$JSON_OUTPUT" != "true" ]]; then
- echo -e "$@"
- fi
- }
- section() {
- log "${BLUE}========================================${NC}"
- log "${BLUE}$1${NC}"
- log "${BLUE}========================================${NC}"
- }
- cmd_exists() { command -v "$1" >/dev/null 2>&1; }
- cmd_or_path() { command -v "$1" 2>/dev/null || echo "$2"; }
- # Detect instance type via IMDS
- IMDS_TOKEN=$(curl -s -m 2 -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60" 2>/dev/null) || true
- Note: Quoted as data in code: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
- Line 67 on GitHub
- if [[ -z "$IMDS_TOKEN" ]]; then
- echo "Error: Failed to retrieve IMDS token (IMDSv2 endpoint unreachable)" >&2
- INSTANCE_TYPE=""
- else
- INSTANCE_TYPE=$(curl -s -m 2 -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" http://169.254.169.254/latest/meta-data/instance-type 2>/dev/null) || true
- Dismissed after review: Fetches a raw IP-literal URL — ambiguous (could be legitimate internal tooling) but worth a second look
- Line 72 on GitHub
- if [[ -z "$INSTANCE_TYPE" ]]; then
- echo "Error: Failed to retrieve instance type from IMDS" >&2
- fi
- fi
- IS_NEURON=false
- [[ "$INSTANCE_TYPE" =~ (^|\.)(trn|inf) ]] && IS_NEURON=true
- # GPU detection is driven by `cmd_exists nvidia-smi` at each GPU section below —
- # no explicit IS_GPU flag needed. Keeps GPU checks working on instances where
- # the driver is present but the regex would miss (e.g. new p-family SKUs).
- # JSON-safe string escape via jq (handles all special/unicode characters correctly)
- json_escape() { jq -rn --arg v "$1" '$v | @json | .[1:-1]'; }
- declare -A VERSIONS
- # --- System Information ---
- : > "$OUTPUT_FILE"
- section "System Information"
- log "Host: $(hostname)"
- log "Date: $(date)"
- log "OS: $(grep PRETTY_NAME /etc/os-release 2>/dev/null | cut -d'"' -f2)"
- log "Kernel: $(uname -r)"
- log "Architecture: $(uname -m)"
- log "Instance Type: ${INSTANCE_TYPE:-unknown}"
- log ""
- # --- NVIDIA Driver & CUDA ---
- section "CUDA Information"
- if cmd_exists nvidia-smi; then
- DRIVER_VER=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | head -1)
- if [ $? -ne 0 ] || [ -z "$DRIVER_VER" ] || [[ "$DRIVER_VER" == *"failed"* ]] || [[ "$DRIVER_VER" == *"NVIDIA-SMI"* ]]; then
- DRIVER_VER=""
- if [[ "$IS_NEURON" == "true" ]]; then
- log "${YELLOW}NVIDIA driver: N/A (Trainium/Inferentia instance)${NC}"
- else
- log "${YELLOW}nvidia-smi found but driver not responding${NC}"
- fi
- else
- VERSIONS[NVIDIA_DRIVER]="$DRIVER_VER"
- log "NVIDIA Driver: $DRIVER_VER"
- fi
- MAX_CUDA=$(nvidia-smi 2>/dev/null | grep "CUDA Version" | sed -n 's/.*CUDA Version: \([0-9.]*\).*/\1/p' | head -1)
- if [ -n "$MAX_CUDA" ]; then
- VERSIONS[MAX_CUDA]="$MAX_CUDA"
- log "Max Supported CUDA: $MAX_CUDA (driver capability)"
- fi
- log ""
- log "GPUs:"
- nvidia-smi -L 2>/dev/null | while read -r line; do log " $line"; done
- log ""
- else
- log "${YELLOW}nvidia-smi not found - no NVIDIA GPU or driver not installed${NC}"
- log ""
- fi
- if cmd_exists nvcc; then
- CUDA_VER=$(nvcc --version 2>/dev/null | grep "release" | sed -n 's/.*release \([0-9.]*\).*/\1/p')
- VERSIONS[CUDA_TOOLKIT]="$CUDA_VER"
- log "CUDA Toolkit (nvcc): $CUDA_VER"
- elif [ -L /usr/local/cuda ]; then
- CUDA_LINK=$(readlink /usr/local/cuda)
- CUDA_VER=$(echo "$CUDA_LINK" | sed -n 's/.*cuda-\([0-9.]*\).*/\1/p')
- VERSIONS[CUDA_TOOLKIT]="${CUDA_VER} (symlink)"
- log "CUDA Toolkit (symlink): $CUDA_VER"
- fi
- CUDA_DIRS=$(ls -d /usr/local/cuda-* 2>/dev/null)
- if [ -n "$CUDA_DIRS" ]; then
- log "Installed CUDA dirs: $CUDA_DIRS"
- [ -L /usr/local/cuda ] && log "Active symlink: /usr/local/cuda -> $(readlink /usr/local/cuda)"
- fi
- log ""
- # --- cuDNN ---
- section "cuDNN Information"
- CUDNN_VER=""
- # Check header file
- CUDNN_HEADER=$(find /usr/local/cuda/include /usr/include -maxdepth 2 -name "cudnn_version.h" 2>/dev/null | head -1)
- if [ -z "$CUDNN_HEADER" ]; then
- CUDNN_HEADER=$(find /usr/local/cuda/include /usr/include -maxdepth 2 -name "cudnn.h" 2>/dev/null | head -1)
- fi
- if [ -n "$CUDNN_HEADER" ]; then
- MAJOR=$(grep "#define CUDNN_MAJOR" "$CUDNN_HEADER" 2>/dev/null | awk '{print $3}')
- MINOR=$(grep "#define CUDNN_MINOR" "$CUDNN_HEADER" 2>/dev/null | awk '{print $3}')
- PATCH=$(grep "#define CUDNN_PATCHLEVEL" "$CUDNN_HEADER" 2>/dev/null | awk '{print $3}')
- [ -n "$MAJOR" ] && [ -n "$MINOR" ] && CUDNN_VER="${MAJOR}.${MINOR}.${PATCH}"
- fi
- # Package fallback
- if [ -z "$CUDNN_VER" ]; then
- if cmd_exists dpkg; then
- CUDNN_VER=$(dpkg -l 2>/dev/null | grep -i "libcudnn[0-9]" | head -1 | awk '{print $3}' | sed 's/-.*//')
- fi
- if [ -z "$CUDNN_VER" ] && cmd_exists rpm; then
- CUDNN_VER=$(rpm -qa 2>/dev/null | grep -i "libcudnn" | head -1 | sed -n 's/.*-\([0-9][0-9.]*\)-.*/\1/p')
- fi
- fi
- if [ -n "$CUDNN_VER" ]; then
- VERSIONS[CUDNN]="$CUDNN_VER"
- log "cuDNN: v${CUDNN_VER}"
- else
- # Check if library exists at all
- CUDNN_LIB=$(find /usr/local/cuda/lib64 /usr/lib -maxdepth 2 -name "libcudnn.so*" 2>/dev/null | head -1)
- if [ -n "$CUDNN_LIB" ]; then
- log "cuDNN library found: $CUDNN_LIB (version unknown)"
- else
- log "${YELLOW}cuDNN not found${NC}"
- fi
- fi
- log ""
- # --- NCCL ---
- section "NCCL Information"
- NCCL_VER=""
- NCCL_LIBS=$(find /usr/local/cuda*/lib* /usr/lib* /usr/local/lib* /opt/nccl/lib -maxdepth 2 -name "libnccl.so*" 2>/dev/null | head -10)
- if [ -n "$NCCL_LIBS" ]; then
- log "Libraries found:"
- echo "$NCCL_LIBS" | while read -r lib; do log " $lib"; done
- while IFS= read -r lib; do
- if [[ $lib =~ libnccl\.so\.([0-9]+\.[0-9]+\.[0-9]+) ]]; then
- NCCL_VER="${BASH_REMATCH[1]}"
- break
- fi
- done <<< "$NCCL_LIBS"
- fi
- # Fallback to header
- if [ -z "$NCCL_VER" ]; then
- NCCL_HEADER=$(find /usr/local/cuda*/include /usr/include /usr/local/include /opt/nccl/include -maxdepth 2 -name "nccl.h" 2>/dev/null | head -1)
- if [ -n "$NCCL_HEADER" ]; then
- MAJOR=$(grep "NCCL_MAJOR" "$NCCL_HEADER" 2>/dev/null | head -1 | awk '{print $3}')
- MINOR=$(grep "NCCL_MINOR" "$NCCL_HEADER" 2>/dev/null | head -1 | awk '{print $3}')
- PATCH=$(grep "NCCL_PATCH" "$NCCL_HEADER" 2>/dev/null | head -1 | awk '{print $3}')
- [ -n "$MAJOR" ] && [ -n "$MINOR" ] && [ -n "$PATCH" ] && NCCL_VER="${MAJOR}.${MINOR}.${PATCH}"
- [ -n "$NCCL_VER" ] && log "Version from header ($NCCL_HEADER): $NCCL_VER"
- fi
- fi
- if [ -n "$NCCL_VER" ]; then
- VERSIONS[NCCL]="$NCCL_VER"
- log "NCCL version: v${NCCL_VER}"
- else
- log "${YELLOW}NCCL not found${NC}"
- fi
- # Package info
- if cmd_exists dpkg; then
- NCCL_PKGS=$(dpkg -l 2>/dev/null | grep -i nccl)
- [ -n "$NCCL_PKGS" ] && { log ""; log "Packages (dpkg):"; echo "$NCCL_PKGS" | while read -r p; do log " $p"; done; }
- fi
- if cmd_exists rpm; then
- NCCL_RPMS=$(rpm -qa 2>/dev/null | grep -i nccl)
- [ -n "$NCCL_RPMS" ] && { log ""; log "Packages (rpm):"; echo "$NCCL_RPMS" | while read -r p; do log " $p"; done; }
- fi
- # nccl-tests
- NCCL_TESTS=$(find /opt /usr/local -maxdepth 4 -name "all_reduce_perf" 2>/dev/null | head -1)
- [ -n "$NCCL_TESTS" ] && log "nccl-tests found: $(dirname "$NCCL_TESTS")"
- log ""
- # --- EFA ---
- section "EFA Information"
- EFA_VER=""
- LIBFABRIC_VER=""
- if [ -f /opt/amazon/efa_installed_packages ]; then
- EFA_VER=$(grep "# EFA installer version:" /opt/amazon/efa_installed_packages | sed -n 's/.*version: \([0-9.]*\).*/\1/p')
- LIBFABRIC_VER=$(grep "libfabric-aws-" /opt/amazon/efa_installed_packages | sed -n 's/.*libfabric-aws-\([0-9.]*\)amzn.*/\1/p' | head -1)
- log "EFA installed packages:"
- while read -r line; do log " $line"; done < /opt/amazon/efa_installed_packages
- log ""
- fi
- if [ -z "$LIBFABRIC_VER" ]; then
- FI_INFO=""
- cmd_exists fi_info && FI_INFO="fi_info"
- [ -z "$FI_INFO" ] && [ -f /opt/amazon/efa/bin/fi_info ] && FI_INFO="/opt/amazon/efa/bin/fi_info"
- if [ -n "$FI_INFO" ]; then
- LIBFABRIC_VER=$("$FI_INFO" --version 2>&1 | grep "libfabric" | sed -n 's/.*libfabric: \([0-9.]*\).*/\1/p' | head -1)
- log "Libfabric ($FI_INFO): $LIBFABRIC_VER"
- fi
- fi
- [ -n "$EFA_VER" ] && VERSIONS[EFA_INSTALLER]="$EFA_VER" && log "EFA Installer: $EFA_VER"
- [ -n "$LIBFABRIC_VER" ] && VERSIONS[LIBFABRIC]="$LIBFABRIC_VER" && log "Libfabric: $LIBFABRIC_VER"
- # EFA provider check
- FI_CMD=""
- cmd_exists fi_info && FI_CMD="fi_info"
- [ -z "$FI_CMD" ] && [ -f /opt/amazon/efa/bin/fi_info ] && FI_CMD="/opt/amazon/efa/bin/fi_info"
- if [ -n "$FI_CMD" ]; then
- if "$FI_CMD" -p efa 2>&1 | grep -q "provider: efa"; then
- log "${GREEN}EFA provider available${NC}"
- else
- log "${YELLOW}EFA provider not detected${NC}"
- fi
- fi
- [ -d /sys/class/infiniband ] && log "InfiniBand devices: $(ls /sys/class/infiniband/ 2>/dev/null | tr '\n' ' ')" || log "${YELLOW}No InfiniBand devices found${NC}"
- log ""
- # --- AWS OFI NCCL ---
- section "AWS OFI NCCL Plugin"
- OFI_NCCL_VER=""
- if [ -f /opt/amazon/efa_installed_packages ]; then
- OFI_NCCL_VER=$(grep "libnccl-ofi-" /opt/amazon/efa_installed_packages | sed -n 's/.*libnccl-ofi-\([0-9.]*\)-.*/\1/p' | head -1)
- fi
- if [ -n "$OFI_NCCL_VER" ]; then
- VERSIONS[AWS_OFI_NCCL]="$OFI_NCCL_VER"
- log "AWS OFI NCCL: v${OFI_NCCL_VER}"
- else
- OFI_LIB=$(find /opt/amazon/ofi-nccl /usr/lib* -maxdepth 3 -name "libnccl-net.so" 2>/dev/null | head -1)
- if [ -n "$OFI_LIB" ]; then
- log "AWS OFI NCCL library found: $OFI_LIB (version unknown)"
- else
- log "${YELLOW}AWS OFI NCCL not found${NC}"
- fi
- fi
- log ""
- # --- GDRCopy ---
- section "GDRCopy Information"
- GDRCOPY_VER=""
- if cmd_exists rpm; then
- GDRCOPY_VER=$(rpm -qa 2>/dev/null | grep "^gdrcopy-[0-9]" | head -1 | sed -n 's/gdrcopy-\([0-9.]*\)-.*/\1/p')
- fi
- if [ -z "$GDRCOPY_VER" ] && cmd_exists dpkg; then
- GDRCOPY_VER=$(dpkg -l 2>/dev/null | grep "^ii.*gdrcopy" | head -1 | awk '{print $3}' | sed -n 's/\([0-9.]*\)-.*/\1/p')
- fi
- if [ -n "$GDRCOPY_VER" ]; then
- VERSIONS[GDRCOPY]="$GDRCOPY_VER"
- log "GDRCopy: v${GDRCOPY_VER}"
- else
- GDRCOPY_LIB=$(find /usr /opt -maxdepth 4 -name "libgdrapi.so*" 2>/dev/null | head -1)
- [ -n "$GDRCOPY_LIB" ] && log "GDRCopy library found: $GDRCOPY_LIB (version unknown)" || log "${YELLOW}GDRCopy not found${NC}"
- fi
- if lsmod 2>/dev/null | grep -q gdrdrv; then
- log "Kernel module: ${GREEN}gdrdrv loaded${NC}"
- else
- log "Kernel module: ${YELLOW}gdrdrv not loaded${NC}"
- fi
- log ""
- # --- MPI ---
- section "MPI Information"
- MPI_VER=""
- if cmd_exists mpirun; then
- MPI_VER=$(mpirun --version 2>&1 | head -1)
- elif [ -f /opt/amazon/openmpi/bin/mpirun ]; then
- MPI_VER=$(/opt/amazon/openmpi/bin/mpirun --version 2>&1 | head -1)
- fi
- if [ -n "$MPI_VER" ]; then
- VERSIONS[MPI]="$MPI_VER"
- log "MPI: $MPI_VER"
- else
- log "${YELLOW}MPI not found${NC}"
- fi
- log ""
- # --- Neuron SDK (Trainium/Inferentia) ---
- section "Neuron SDK Information"
- NEURON_DETECTED=false
- NEURON_BIN="/opt/aws/neuron/bin"
- # Neuron driver (kernel module)
- NEURON_DRV_VER=$(modinfo neuron 2>/dev/null | grep "^version:" | awk '{print $2}')
- if [ -n "$NEURON_DRV_VER" ]; then
- VERSIONS[NEURON_DRIVER]="$NEURON_DRV_VER"
- log "Neuron Driver: $NEURON_DRV_VER"
- NEURON_DETECTED=true
- fi
- # Neuron devices
- NEURON_DEV_COUNT=$(ls /dev/neuron* 2>/dev/null | wc -l)
- if [ "$NEURON_DEV_COUNT" -gt 0 ]; then
- VERSIONS[NEURON_DEVICES]="$NEURON_DEV_COUNT"
- log "Neuron Devices: $NEURON_DEV_COUNT"
- NEURON_DETECTED=true
- fi
- # Neuron devices listing
- NEURON_LS=$(cmd_or_path neuron-ls "$NEURON_BIN/neuron-ls")
- if [ -x "$NEURON_LS" ]; then
- NEURON_DETECTED=true
- log "Neuron devices:"
- "$NEURON_LS" 2>/dev/null | while read -r line; do log " $line"; done
- log ""
- fi
- # Neuron compiler
- NEURON_CC=$(cmd_or_path neuronx-cc "$NEURON_BIN/neuronx-cc")
- if [ -x "$NEURON_CC" ]; then
- NEURON_CC_VER=$("$NEURON_CC" --version 2>&1 | head -1)
- VERSIONS[NEURON_COMPILER]="$NEURON_CC_VER"
- log "Neuron Compiler: $NEURON_CC_VER"
- NEURON_DETECTED=true
- fi
- # Neuron runtime
- NEURON_RT_VER=""
- if cmd_exists dpkg; then
- NEURON_RT_VER=$(dpkg -l 2>/dev/null | grep "aws-neuronx-runtime-lib" | head -1 | awk '{print $3}')
- fi
- if [ -z "$NEURON_RT_VER" ] && cmd_exists rpm; then
- NEURON_RT_VER=$(rpm -qa 2>/dev/null | grep "aws-neuronx-runtime" | head -1 | sed -n 's/.*-\([0-9][0-9.]*\)-.*/\1/p')
- fi
- if [ -n "$NEURON_RT_VER" ]; then
- VERSIONS[NEURON_RUNTIME]="$NEURON_RT_VER"
- log "Neuron Runtime: $NEURON_RT_VER"
- NEURON_DETECTED=true
- fi
- # torch-neuronx
- TORCH_NEURON_VER=$(python3 -c "import torch_neuronx; print(torch_neuronx.__version__)" 2>/dev/null)
- if [ -n "$TORCH_NEURON_VER" ]; then
- VERSIONS[TORCH_NEURONX]="$TORCH_NEURON_VER"
- log "torch-neuronx: $TORCH_NEURON_VER"
- NEURON_DETECTED=true
- fi
- # Neuron tools
- NEURON_TOP=$(cmd_or_path neuron-top "$NEURON_BIN/neuron-top")
- if [ -x "$NEURON_TOP" ]; then
- NEURON_TOOLS_VER=""
- if cmd_exists dpkg; then
- NEURON_TOOLS_VER=$(dpkg -l 2>/dev/null | grep "aws-neuronx-tools" | head -1 | awk '{print $3}')
- fi
- if [ -z "$NEURON_TOOLS_VER" ] && cmd_exists rpm; then
- NEURON_TOOLS_VER=$(rpm -qa 2>/dev/null | grep "aws-neuronx-tools" | head -1 | sed -n 's/.*-\([0-9][0-9.]*\)-.*/\1/p')
- fi
- [ -n "$NEURON_TOOLS_VER" ] && log "Neuron Tools: $NEURON_TOOLS_VER"
- NEURON_DETECTED=true
- fi
- if [[ "$NEURON_DETECTED" != "true" ]]; then
- log "${YELLOW}Neuron SDK not found (expected on non-Trainium/Inferentia instances)${NC}"
- fi
- log ""
- # --- Python & PyTorch ---
- section "Python / ML Frameworks"
- if cmd_exists python3; then
- PY_VER=$(python3 --version 2>&1 | awk '{print $2}')
- VERSIONS[PYTHON]="$PY_VER"
- log "Python: $PY_VER"
- PT_INFO=$(python3 -c "
- import torch
- print(f'{torch.__version__}')
- print(f'cuda_available={torch.cuda.is_available()}')
- print(f'cuda_version={torch.version.cuda or \"N/A\"}')
- if hasattr(torch, 'xpu') and hasattr(torch.xpu, 'is_available'):
- print(f'xpu_available={torch.xpu.is_available()}')
- " 2>/dev/null)
- if [ -n "$PT_INFO" ]; then
- PT_VER=$(echo "$PT_INFO" | head -1)
- VERSIONS[PYTORCH]="$PT_VER"
- log "PyTorch: $PT_VER"
- echo "$PT_INFO" | tail -n +2 | while read -r line; do log " $line"; done
- fi
- else
- log "${YELLOW}python3 not found${NC}"
- fi
- log ""
- # --- Container Runtime ---
- section "Container Runtime"
- cmd_exists docker && log "Docker: $(docker --version 2>&1)"
- cmd_exists containerd && log "Containerd: $(containerd --version 2>&1)"
- cmd_exists kubectl && log "kubectl: $(kubectl version --client 2>&1 | head -1)"
- # NVIDIA Container Toolkit
- if cmd_exists nvidia-ctk; then
- NCTK_VER=$(nvidia-ctk --version 2>&1 | head -1)
- VERSIONS[NVIDIA_CTK]="$NCTK_VER"
- log "NVIDIA Container Toolkit: $NCTK_VER"
- elif cmd_exists dpkg && dpkg -l 2>/dev/null | grep -q nvidia-container-toolkit; then
- NCTK_VER=$(dpkg -l 2>/dev/null | grep "nvidia-container-toolkit " | head -1 | awk '{print $3}')
- VERSIONS[NVIDIA_CTK]="$NCTK_VER"
- log "NVIDIA Container Toolkit: $NCTK_VER"
- fi
- log ""
- # --- Environment Variables ---
- section "Relevant Environment Variables"
- log "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH:-<not set>}"
- log "NCCL vars: $(env | grep -i "^NCCL" 2>/dev/null | tr '\n' ' ')"
- log "EFA vars: $(env | grep -i "^FI_\|^EFA_\|^RDMAV" 2>/dev/null | tr '\n' ' ')"
- log "NEURON vars: $(env | grep -i "^NEURON" 2>/dev/null | tr '\n' ' ')"
- log ""
- # --- CUDA/Driver Compatibility Analysis ---
- section "CUDA/Driver Compatibility Analysis"
- if [ -n "${VERSIONS[NVIDIA_DRIVER]}" ] && [ -n "${VERSIONS[MAX_CUDA]}" ]; then
- DRIVER_MAJOR=$(echo "${VERSIONS[NVIDIA_DRIVER]}" | cut -d'.' -f1)
- log "Driver ${VERSIONS[NVIDIA_DRIVER]} (series $DRIVER_MAJOR):"
- if [ "$DRIVER_MAJOR" -ge 580 ] 2>/dev/null; then
- log " ${GREEN}✓ Supports CUDA 13.x, 12.x, 11.x${NC}"
- elif [ "$DRIVER_MAJOR" -ge 570 ] 2>/dev/null; then
- log " ${GREEN}✓ Supports CUDA 12.8+ (Blackwell), 12.x, 11.x${NC}"
- elif [ "$DRIVER_MAJOR" -ge 545 ] 2>/dev/null; then
- log " ${GREEN}✓ Supports CUDA 12.3-12.7, 11.x${NC}"
- log " ${YELLOW}⚠ NOT compatible with CUDA 12.8+ (needs driver 570+)${NC}"
- elif [ "$DRIVER_MAJOR" -ge 525 ] 2>/dev/null; then
- log " ${GREEN}✓ Supports CUDA 12.0-12.2, 11.x${NC}"
- log " ${YELLOW}⚠ NOT compatible with CUDA 12.3+ (needs driver 545+)${NC}"
- elif [ "$DRIVER_MAJOR" -ge 450 ] 2>/dev/null; then
- log " ${GREEN}✓ Supports CUDA 11.x${NC}"
- log " ${YELLOW}⚠ NOT compatible with CUDA 12.x (needs driver 525+)${NC}"
- else
- log " ${YELLOW}⚠ Driver older than CUDA 11.x baseline${NC}"
- fi
- fi
- log ""
- # --- Version Summary ---
- section "Version Summary"
- log "NVIDIA_DRIVER: ${VERSIONS[NVIDIA_DRIVER]:-not found}"
- log "MAX_CUDA: ${VERSIONS[MAX_CUDA]:-not found}"
- log "CUDA_TOOLKIT: ${VERSIONS[CUDA_TOOLKIT]:-not found}"
- log "CUDNN: ${VERSIONS[CUDNN]:+v${VERSIONS[CUDNN]}}${VERSIONS[CUDNN]:-not found}"
- log "NCCL: ${VERSIONS[NCCL]:+v${VERSIONS[NCCL]}}${VERSIONS[NCCL]:-not found}"
- log "EFA_INSTALLER: ${VERSIONS[EFA_INSTALLER]:-not found}"
- log "LIBFABRIC: ${VERSIONS[LIBFABRIC]:-not found}"
- log "AWS_OFI_NCCL: ${VERSIONS[AWS_OFI_NCCL]:+v${VERSIONS[AWS_OFI_NCCL]}}${VERSIONS[AWS_OFI_NCCL]:-not found}"
- log "GDRCOPY: ${VERSIONS[GDRCOPY]:+v${VERSIONS[GDRCOPY]}}${VERSIONS[GDRCOPY]:-not found}"
- log "MPI: ${VERSIONS[MPI]:-not found}"
- log "NEURON_DRIVER: ${VERSIONS[NEURON_DRIVER]:-not found}"
- log "NEURON_DEVICES: ${VERSIONS[NEURON_DEVICES]:-0}"
- log "NEURON_COMPILER: ${VERSIONS[NEURON_COMPILER]:-not found}"
- log "NEURON_RUNTIME: ${VERSIONS[NEURON_RUNTIME]:-not found}"
- log "TORCH_NEURONX: ${VERSIONS[TORCH_NEURONX]:-not found}"
- log "PYTHON: ${VERSIONS[PYTHON]:-not found}"
- log "PYTORCH: ${VERSIONS[PYTORCH]:-not found}"
- log ""
- log "Report saved to: $OUTPUT_FILE"
- # --- JSON output (stdout only) ---
- if [[ "$JSON_OUTPUT" == "true" ]]; then
- cat <<EOF
- {
- "hostname": "$(hostname)",
- "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
- "instance_type": "$(json_escape "${INSTANCE_TYPE:-unknown}")",
- "versions": {
- "nvidia_driver": "$(json_escape "${VERSIONS[NVIDIA_DRIVER]:-}")",
- "max_cuda": "$(json_escape "${VERSIONS[MAX_CUDA]:-}")",
- "cuda_toolkit": "$(json_escape "${VERSIONS[CUDA_TOOLKIT]:-}")",
- "cudnn": "$(json_escape "${VERSIONS[CUDNN]:-}")",
- "nccl": "$(json_escape "${VERSIONS[NCCL]:-}")",
- "efa_installer": "$(json_escape "${VERSIONS[EFA_INSTALLER]:-}")",
- "libfabric": "$(json_escape "${VERSIONS[LIBFABRIC]:-}")",
- "aws_ofi_nccl": "$(json_escape "${VERSIONS[AWS_OFI_NCCL]:-}")",
- "gdrcopy": "$(json_escape "${VERSIONS[GDRCOPY]:-}")",
- "mpi": "$(json_escape "${VERSIONS[MPI]:-}")",
- "neuron_driver": "$(json_escape "${VERSIONS[NEURON_DRIVER]:-}")",
- "neuron_devices": "$(json_escape "${VERSIONS[NEURON_DEVICES]:-}")",
- "neuron_compiler": "$(json_escape "${VERSIONS[NEURON_COMPILER]:-}")",
- "neuron_runtime": "$(json_escape "${VERSIONS[NEURON_RUNTIME]:-}")",
- "torch_neuronx": "$(json_escape "${VERSIONS[TORCH_NEURONX]:-}")",
- "nvidia_container_toolkit": "$(json_escape "${VERSIONS[NVIDIA_CTK]:-}")",
- "python": "$(json_escape "${VERSIONS[PYTHON]:-}")",
- "pytorch": "$(json_escape "${VERSIONS[PYTORCH]:-}")"
- }
- }
- EOF
- fi
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.