本文档介绍了如何在已配置的集群上运行使用 GPUDirect RDMA 的 NCCL/gIB 测试。根据您的使用情形，请使用以下选项之一：
- 如果您有 2 个节点要测试，请运行基本测试。
- 如果您要测试的节点超过两个，请使用 JobSet 运行测试。
在两个节点上进行测试
以下测试会在两个节点上运行 NCCL 工作负载。请了解有关此测试的以下信息：
- 默认情况下，GKE 会将这两个 Pod 调度到不同的节点池（如果可用）。如果节点池是使用不同的 NVLink 网域创建的，则此测试表示跨网域 RDMA 吞吐量。如需在同一网域中调度 Pod，请修改 Pod 亲和性以在同一节点池中调度。
运行双节点测试：
连接到集群：
gcloud container clusters get-credentials CLUSTER_NAME \ --location=COMPUTE_REGION执行以下变量替换操作：
CLUSTER_NAME：集群的名称，对于使用 Cluster Toolkit 创建的集群，该名称基于DEPLOYMENT_NAME。COMPUTE_REGION：计算区域的名称。
部署一个 NCCL 测试工作负载，该工作负载包含两个在两个 A4X Max 节点上运行的测试 Pod：
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/gpudirect-rdma/nccl-test-a4x-max.yaml检查这两个 Pod 是否都在某些节点上运行：
kubectl get pods nccl-test-host-1 nccl-test-host-2如果这两个 Pod 显示
Running状态，您可以继续执行下一步。为 A4X Max 节点触发 all-gather 测试：
HOSTS="nccl-host-1 nccl-host-2" kubectl exec nccl-test-host-1 -it -- bash -c "/usr/local/gib/scripts/run_nccl_tests.sh -t all_gather -b 1M -e 16G ${HOSTS}"输出类似于以下内容：
# nccl-tests version 2.17.6 nccl-headers=22809 nccl-library=22809 # Collective test starting: all_gather_perf # nThread 1 nGpus 1 minBytes 4096 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 # # Using devices # Rank 0 Group 0 Pid 299 on gke-psch-ug-cluster-a4xmax-2-1391c2ef-m9bm device 0 [0008:06:00] NVIDIA GB300 # Rank 1 Group 0 Pid 300 on gke-psch-ug-cluster-a4xmax-2-1391c2ef-m9bm device 1 [0009:06:00] NVIDIA GB300 # Rank 2 Group 0 Pid 301 on gke-psch-ug-cluster-a4xmax-2-1391c2ef-m9bm device 2 [0018:06:00] NVIDIA GB300 # Rank 3 Group 0 Pid 302 on gke-psch-ug-cluster-a4xmax-2-1391c2ef-m9bm device 3 [0019:06:00] NVIDIA GB300 # Rank 4 Group 0 Pid 237 on gke-psch-ug-cluster-a4xmax-03878a77-d3lp device 0 [0008:06:00] NVIDIA GB300 # Rank 5 Group 0 Pid 238 on gke-psch-ug-cluster-a4xmax-03878a77-d3lp device 1 [0009:06:00] NVIDIA GB300 # Rank 6 Group 0 Pid 239 on gke-psch-ug-cluster-a4xmax-03878a77-d3lp device 2 [0018:06:00] NVIDIA GB300 # Rank 7 Group 0 Pid 240 on gke-psch-ug-cluster-a4xmax-03878a77-d3lp device 3 [0019:06:00] NVIDIA GB300 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 4096 128 float none -1 28.51 0.14 0.13 0 27.71 0.15 0.13 0 8192 256 float none -1 28.10 0.29 0.26 0 28.40 0.29 0.25 0 16384 512 float none -1 28.55 0.57 0.50 0 28.19 0.58 0.51 0 32768 1024 float none -1 30.56 1.07 0.94 0 29.65 1.11 0.97 0 65536 2048 float none -1 33.30 1.97 1.72 0 33.14 1.98 1.73 0 131072 4096 float none -1 36.18 3.62 3.17 0 36.14 3.63 3.17 0 262144 8192 float none -1 38.50 6.81 5.96 0 94.91 2.76 2.42 0 524288 16384 float none -1 152.25 3.44 3.01 0 54.79 9.57 8.37 0 1048576 32768 float none -1 63.82 16.43 14.38 0 64.06 16.37 14.32 0 2097152 65536 float none -1 65.10 32.21 28.19 0 66.13 31.71 27.75 0 4194304 131072 float none -1 67.73 61.92 54.18 0 67.16 62.45 54.65 0 8388608 262144 float none -1 79.65 105.31 92.15 0 80.02 104.83 91.73 0 16777216 524288 float none -1 189.74 88.42 77.37 0 187.57 89.44 78.26 0 33554432 1048576 float none -1 252.85 132.70 116.11 0 202.31 165.86 145.13 0 67108864 2097152 float none -1 250.55 267.85 234.37 0 276.11 243.06 212.67 0 134217728 4194304 float none -1 394.38 340.33 297.79 0 487.60 275.26 240.85 0 268435456 8388608 float none -1 717.97 373.88 327.15 0 799.98 335.55 293.61 0 536870912 16777216 float none -1 1421.29 377.73 330.52 0 1392.81 385.46 337.28 0 1073741824 33554432 float none -1 2783.37 385.77 337.55 0 2596.97 413.46 361.78 0 2147483648 67108864 float none -1 5396.10 397.97 348.22 0 5059.01 424.49 371.43 0 4294967296 134217728 float none -1 10579.7 405.96 355.22 0 9918.44 433.03 378.90 0 8589934592 268435456 float none -1 21012.9 408.79 357.69 0 20043.4 428.57 375.00 0 17179869184 536870912 float none -1 42091.7 408.15 357.13 0 40243.2 426.90 373.54 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 146.047 # # Collective test concluded: all_gather_perf如果 Pod 调度在不同 NVLink 网域中的节点上，则此测试表示跨网域 RDMA 吞吐量，如提供的输出所示。如需将 Pod 扩展到在不同的 NVLink 网域中创建的节点池，请使用以下内容修改
nccl-test-a4x-max.yaml中的 Pod 规范亲和性：spec: ... affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app: nccl-test topologyKey: cloud.google.com/gke-nodepool
使用 JobSet 进行测试
安装 JobSet：
VERSION=v0.10.1 kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/$VERSION/manifests.yaml确保非 GPU 节点池有足够的资源来调度 JobSet 控制器。按照相应步骤定义您自己的资源调整。
如需详细了解 JobSet 安装，请参阅安装。
运行以下命令，并将
NUM_NODES替换为要运行 NCCL 测试的节点数量：wget https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/refs/heads/master/gpudirect-rdma/nccl-test-a4x-max-jobset.yaml NUM_NODES=2 sed "s|__NUM_NODES__|NUM_NODES|" nccl-test-a4x-max-jobset.yaml | kubectl apply -f -检查所有 Pod 是否都处于
Completed状态：kubectl get pods | grep allgather-worker输出类似于以下内容：
allgather-worker-0-0-g45d2 0/1 Completed 0 13m allgather-worker-0-1-prpvw 0/1 Completed 0 13m allgather-worker-0-2-qbwt5 0/1 Completed 0 13m查看启动测试的头 Pod (
nccl-test-nccl-test-0) 的测试结果：kubectl logs $(kubectl get pods -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep allgather-worker-0-0)
输出类似于以下内容：
# nccl-tests version 2.17.6 nccl-headers=22809 nccl-library=22809 # Collective test starting: all_gather_perf # nThread 1 nGpus 1 minBytes 1024 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 100 agg iters: 1 validation: 1 graph: 0 # ... # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 1024 32 float none -1 45.49 0.02 0.02 0 45.29 0.02 0.02 0 2048 64 float none -1 45.52 0.04 0.04 0 45.37 0.05 0.04 0 4096 128 float none -1 46.02 0.09 0.08 0 45.83 0.09 0.08 0 8192 256 float none -1 63.93 0.13 0.11 0 46.98 0.17 0.15 0 16384 512 float none -1 46.51 0.35 0.31 0 47.11 0.35 0.30 0 32768 1024 float none -1 66.32 0.49 0.43 0 50.73 0.65 0.57 0 65536 2048 float none -1 49.89 1.31 1.15 0 50.04 1.31 1.15 0 131072 4096 float none -1 54.68 2.40 2.10 0 52.38 2.50 2.19 0 262144 8192 float none -1 54.66 4.80 4.20 0 54.06 4.85 4.24 0 524288 16384 float none -1 66.28 7.91 6.92 0 65.75 7.97 6.98 0 1048576 32768 float none -1 85.63 12.25 10.72 0 86.44 12.13 10.61 0 2097152 65536 float none -1 68.33 30.69 26.86 0 72.32 29.00 25.37 0 4194304 131072 float none -1 71.85 58.37 51.08 0 71.58 58.60 51.28 0 8388608 262144 float none -1 83.80 100.10 87.59 0 85.73 97.85 85.62 0 16777216 524288 float none -1 195.94 85.62 74.92 0 195.86 85.66 74.95 0 33554432 1048576 float none -1 240.84 139.32 121.91 0 210.82 159.16 139.27 0 67108864 2097152 float none -1 254.95 263.22 230.32 0 250.93 267.44 234.01 0 134217728 4194304 float none -1 411.09 326.49 285.68 0 386.11 347.61 304.16 0 268435456 8388608 float none -1 741.69 361.92 316.68 0 722.42 371.58 325.13 0 536870912 16777216 float none -1 1358.44 395.21 345.81 0 1343.63 399.57 349.62 0 1073741824 33554432 float none -1 2679.62 400.71 350.62 0 2585.68 415.26 363.36 0 2147483648 67108864 float none -1 5281.54 406.60 355.78 0 5074.73 423.17 370.28 0 4294967296 134217728 float none -1 10476.2 409.97 358.73 0 10027.5 428.32 374.78 0 8589934592 268435456 float none -1 20853.9 411.91 360.42 0 20194.7 425.36 372.19 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 126.85 # # Collective test concluded: all_gather_perf