| NodeManager: | |
| Node ID: 0df5098b63b61b261d6b38f00795725b5909c1b1e4c347b784459207 | |
| Node name: 192.168.0.2 | |
| InitialConfigResources: {GPU: 20000, node:192.168.0.2: 10000, CPU: 960000, object_store_memory: 42949672960000, node:__internal_head__: 10000, memory: 21474836480000, accelerator_type:A40: 10000} | |
| ClusterTaskManager: | |
| ========== Node: 0df5098b63b61b261d6b38f00795725b5909c1b1e4c347b784459207 ================= | |
| Infeasible queue length: 0 | |
| Schedule queue length: 0 | |
| Dispatch queue length: 0 | |
| num_waiting_for_resource: 0 | |
| num_waiting_for_plasma_memory: 0 | |
| num_waiting_for_remote_node_resources: 0 | |
| num_worker_not_started_by_job_config_not_exist: 0 | |
| num_worker_not_started_by_registration_timeout: 0 | |
| num_tasks_waiting_for_workers: 0 | |
| num_cancelled_tasks: 0 | |
| cluster_resource_scheduler state: | |
| Local id: -5584884104066417891 Local resources: {"total":{GPU: [10000, 10000], CPU: [960000], object_store_memory: [42949672960000], node:192.168.0.2: [10000], accelerator_type:A40: [10000], node:__internal_head__: [10000], memory: [21474836480000]}}, "available": {GPU: [10000, 10000], CPU: [960000], object_store_memory: [41399063440000], node:192.168.0.2: [10000], accelerator_type:A40: [10000], node:__internal_head__: [10000], memory: [21474836480000]}}, "labels":{"ray.io/node_id":"0df5098b63b61b261d6b38f00795725b5909c1b1e4c347b784459207",} is_draining: 0 is_idle: 0 Cluster resources: node id: -5584884104066417891{"total":{accelerator_type:A40: 10000, GPU: 20000, object_store_memory: 42949672960000, CPU: 960000, memory: 21474836480000, node:192.168.0.2: 10000, node:__internal_head__: 10000}}, "available": {accelerator_type:A40: 10000, GPU: 20000, node:192.168.0.2: 10000, CPU: 960000, memory: 21474836480000, object_store_memory: 41399063440000, node:__internal_head__: 10000}}, "labels":{"ray.io/node_id":"0df5098b63b61b261d6b38f00795725b5909c1b1e4c347b784459207",}, "is_draining": 0, "draining_deadline_timestamp_ms": -1} { "placment group locations": [], "node to bundles": []} | |
| Waiting tasks size: 0 | |
| Number of executing tasks: 0 | |
| Number of pinned task arguments: 0 | |
| Number of total spilled tasks: 0 | |
| Number of spilled waiting tasks: 0 | |
| Number of spilled unschedulable tasks: 0 | |
| Resource usage { | |
| - (language=PYTHON actor_or_task=_StatsActor.__init__ pid=28397 worker_id=2f2abf4327dfc8ec0c674120298d6cd1d20b4cf0c1448a7e6f0a3002): {} | |
| - (language=PYTHON actor_or_task=AutoscalingRequester.__init__ pid=28474 worker_id=a235db546c9cc02ab54504e49e89e366e21f24b693d4fbd560019419): {} | |
| } | |
| Backlog Size per scheduling descriptor :{workerId: num backlogs}: | |
| Running tasks by scheduling class: | |
| ================================================== | |
| ClusterResources: | |
| LocalObjectManager: | |
| - num pinned objects: 30 | |
| - pinned objects size: 155060952 | |
| - num objects pending restore: 0 | |
| - num objects pending spill: 0 | |
| - num bytes pending spill: 0 | |
| - num bytes currently spilled: 0 | |
| - cumulative spill requests: 0 | |
| - cumulative restore requests: 0 | |
| - spilled objects pending delete: 0 | |
| ObjectManager: | |
| - num local objects: 40 | |
| - num unfulfilled push requests: 0 | |
| - num object pull requests: 0 | |
| - num chunks received total: 0 | |
| - num chunks received failed (all): 0 | |
| - num chunks received failed / cancelled: 0 | |
| - num chunks received failed / plasma error: 0 | |
| Event stats: | |
| Global stats: 21 total (0 active) | |
| Queueing time: mean = 139.469 us, max = 547.937 us, min = 41.984 us, total = 2.929 ms | |
| Execution time: mean = 65.037 us, total = 1.366 ms | |
| Event stats: | |
| ObjectManager.FreeObjects - 21 total (0 active), Execution time: mean = 65.037 us, total = 1.366 ms, Queueing time: mean = 139.469 us, max = 547.937 us, min = 41.984 us, total = 2.929 ms | |
| PushManager: | |
| - num pushes in flight: 0 | |
| - num chunks in flight: 0 | |
| - num chunks remaining: 0 | |
| - max chunks allowed: 409 | |
| OwnershipBasedObjectDirectory: | |
| - num listeners: 0 | |
| - cumulative location updates: 197 | |
| - num location updates per second: 10.398 | |
| - num location lookups per second: 0.000 | |
| - num locations added per second: 10.198 | |
| - num locations removed per second: 8.998 | |
| BufferPool: | |
| - create buffer state map size: 0 | |
| PullManager: | |
| - num bytes available for pulled objects: 4139906814 | |
| - num bytes being pulled (all): 0 | |
| - num bytes being pulled / pinned: 0 | |
| - get request bundles: BundlePullRequestQueue{0 total, 0 active, 0 inactive, 0 unpullable} | |
| - wait request bundles: BundlePullRequestQueue{0 total, 0 active, 0 inactive, 0 unpullable} | |
| - task request bundles: BundlePullRequestQueue{0 total, 0 active, 0 inactive, 0 unpullable} | |
| - first get request bundle: N/A | |
| - first wait request bundle: N/A | |
| - first task request bundle: N/A | |
| - num objects queued: 0 | |
| - num objects actively pulled (all): 0 | |
| - num objects actively pulled / pinned: 0 | |
| - num bundles being pulled: 0 | |
| - num pull retries: 0 | |
| - max timeout seconds: 0 | |
| - max timeout request is already processed. No entry. | |
| WorkerPool: | |
| - registered jobs: 1 | |
| - process_failed_job_config_missing: 0 | |
| - process_failed_rate_limited: 0 | |
| - process_failed_pending_registration: 0 | |
| - process_failed_runtime_env_setup_failed: 0 | |
| - num PYTHON workers: 96 | |
| - num PYTHON drivers: 1 | |
| - num PYTHON pending start requests: 0 | |
| - num PYTHON pending registration requests: 0 | |
| - num object spill callbacks queued: 0 | |
| - num object restore queued: 0 | |
| - num util functions queued: 0 | |
| - num idle workers: 94 | |
| TaskDependencyManager: | |
| - task deps map size: 0 | |
| - get req map size: 0 | |
| - wait req map size: 0 | |
| - local objects map size: 40 | |
| WaitManager: | |
| - num active wait requests: 0 | |
| Subscriber: | |
| Channel WORKER_OBJECT_LOCATIONS_CHANNEL | |
| - cumulative subscribe requests: 1973 | |
| - cumulative unsubscribe requests: 1981 | |
| - active subscribed publishers: 0 | |
| - cumulative published messages: 361 | |
| - cumulative processed messages: 307 | |
| Channel WORKER_OBJECT_EVICTION | |
| - cumulative subscribe requests: 322 | |
| - cumulative unsubscribe requests: 292 | |
| - active subscribed publishers: 11 | |
| - cumulative published messages: 292 | |
| - cumulative processed messages: 292 | |
| Channel WORKER_REF_REMOVED_CHANNEL | |
| - cumulative subscribe requests: 0 | |
| - cumulative unsubscribe requests: 0 | |
| - active subscribed publishers: 0 | |
| - cumulative published messages: 0 | |
| - cumulative processed messages: 0 | |
| num async plasma notifications: 0 | |
| Remote node managers: | |
| Event stats: | |
| Global stats: 40377 total (129 active) | |
| Queueing time: mean = 31.288 ms, max = 13.587 s, min = 63.000 ns, total = 1263.319 s | |
| Execution time: mean = 6.827 ms, total = 275.673 s | |
| Event stats: | |
| ClientConnection.async_read.ProcessMessageHeader - 5982 total (97 active), Execution time: mean = 8.437 us, total = 50.470 ms, Queueing time: mean = 205.785 ms, max = 13.587 s, min = 3.606 us, total = 1231.007 s | |
| ClientConnection.async_read.ProcessMessage - 5885 total (0 active), Execution time: mean = 422.093 us, total = 2.484 s, Queueing time: mean = 79.394 us, max = 70.604 ms, min = 1.549 us, total = 467.236 ms | |
| NodeManagerService.grpc_server.ReportWorkerBacklog.HandleRequestImpl - 3343 total (0 active), Execution time: mean = 51.476 us, total = 172.086 ms, Queueing time: mean = 1.457 ms, max = 82.673 ms, min = 4.306 us, total = 4.872 s | |
| NodeManagerService.grpc_server.ReportWorkerBacklog - 3343 total (0 active), Execution time: mean = 2.043 ms, total = 6.830 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| CoreWorkerService.grpc_client.PubsubCommandBatch.OnReplyReceived - 2093 total (0 active), Execution time: mean = 230.389 us, total = 482.203 ms, Queueing time: mean = 627.604 us, max = 68.455 ms, min = 6.510 us, total = 1.314 s | |
| CoreWorkerService.grpc_client.PubsubCommandBatch - 2093 total (0 active), Execution time: mean = 2.704 ms, total = 5.660 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManagerService.grpc_server.RequestWorkerLease - 1848 total (0 active), Execution time: mean = 4.469 ms, total = 8.259 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManagerService.grpc_server.RequestWorkerLease.HandleRequestImpl - 1848 total (0 active), Execution time: mean = 1.169 ms, total = 2.160 s, Queueing time: mean = 1.424 ms, max = 72.018 ms, min = 4.843 us, total = 2.632 s | |
| WorkerPool.PopWorkerCallback - 1848 total (0 active), Execution time: mean = 124.262 us, total = 229.636 ms, Queueing time: mean = 1.044 ms, max = 72.858 ms, min = 9.822 us, total = 1.930 s | |
| NodeManagerService.grpc_server.ReturnWorker - 1845 total (0 active), Execution time: mean = 1.065 ms, total = 1.965 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManagerService.grpc_server.ReturnWorker.HandleRequestImpl - 1845 total (0 active), Execution time: mean = 108.758 us, total = 200.658 ms, Queueing time: mean = 460.806 us, max = 64.518 ms, min = 4.854 us, total = 850.187 ms | |
| CoreWorkerService.grpc_client.PubsubLongPolling - 1489 total (18 active), Execution time: mean = 157.885 ms, total = 235.091 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| CoreWorkerService.grpc_client.PubsubLongPolling.OnReplyReceived - 1471 total (0 active), Execution time: mean = 338.217 us, total = 497.517 ms, Queueing time: mean = 779.054 us, max = 71.939 ms, min = 4.956 us, total = 1.146 s | |
| ObjectManager.ObjectAdded - 419 total (0 active), Execution time: mean = 205.893 us, total = 86.269 ms, Queueing time: mean = 375.710 us, max = 51.160 ms, min = 4.844 us, total = 157.423 ms | |
| CoreWorkerService.grpc_client.UpdateObjectLocationBatch - 391 total (0 active), Execution time: mean = 2.583 ms, total = 1.010 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| CoreWorkerService.grpc_client.UpdateObjectLocationBatch.OnReplyReceived - 391 total (0 active), Execution time: mean = 51.840 us, total = 20.269 ms, Queueing time: mean = 509.894 us, max = 41.760 ms, min = 6.916 us, total = 199.368 ms | |
| ObjectManager.ObjectDeleted - 379 total (0 active), Execution time: mean = 40.361 us, total = 15.297 ms, Queueing time: mean = 558.460 us, max = 3.235 ms, min = 17.256 us, total = 211.656 ms | |
| RaySyncer.OnDemandBroadcasting - 373 total (1 active), Execution time: mean = 223.265 us, total = 83.278 ms, Queueing time: mean = 7.249 ms, max = 1.959 s, min = 12.531 us, total = 2.704 s | |
| NodeManager.CheckGC - 373 total (1 active), Execution time: mean = 3.501 us, total = 1.306 ms, Queueing time: mean = 7.466 ms, max = 1.959 s, min = 21.916 us, total = 2.785 s | |
| ObjectManager.UpdateAvailableMemory - 372 total (0 active), Execution time: mean = 5.462 us, total = 2.032 ms, Queueing time: mean = 670.544 us, max = 51.642 ms, min = 6.430 us, total = 249.442 ms | |
| NodeManagerService.grpc_server.PinObjectIDs - 322 total (0 active), Execution time: mean = 2.010 ms, total = 647.080 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManagerService.grpc_server.PinObjectIDs.HandleRequestImpl - 322 total (0 active), Execution time: mean = 889.059 us, total = 286.277 ms, Queueing time: mean = 720.177 us, max = 64.461 ms, min = 5.704 us, total = 231.897 ms | |
| Subscriber.HandlePublishedMessage_WORKER_OBJECT_LOCATIONS_CHANNEL - 307 total (0 active), Execution time: mean = 15.784 us, total = 4.846 ms, Queueing time: mean = 1.319 ms, max = 51.495 ms, min = 78.043 us, total = 404.877 ms | |
| Subscriber.HandlePublishedMessage_WORKER_OBJECT_EVICTION - 292 total (0 active), Execution time: mean = 62.123 us, total = 18.140 ms, Queueing time: mean = 701.385 us, max = 51.246 ms, min = 123.219 us, total = 204.804 ms | |
| RaySyncer.BroadcastMessage - 254 total (0 active), Execution time: mean = 177.719 us, total = 45.141 ms, Queueing time: mean = 807.929 ns, max = 1.813 us, min = 93.000 ns, total = 205.214 us | |
| - 254 total (0 active), Execution time: mean = 977.575 ns, total = 248.304 us, Queueing time: mean = 549.136 us, max = 30.169 ms, min = 6.296 us, total = 139.480 ms | |
| RayletWorkerPool.deadline_timer.kill_idle_workers - 189 total (1 active), Execution time: mean = 28.169 us, total = 5.324 ms, Queueing time: mean = 12.179 ms, max = 1.966 s, min = 15.398 us, total = 2.302 s | |
| MemoryMonitor.CheckIsMemoryUsageAboveThreshold - 146 total (1 active), Execution time: mean = 787.280 us, total = 114.943 ms, Queueing time: mean = 24.570 ms, max = 1.916 s, min = 15.605 us, total = 3.587 s | |
| ClientConnection.async_write.DoAsyncWrites - 98 total (0 active), Execution time: mean = 1.634 us, total = 160.144 us, Queueing time: mean = 39.453 us, max = 384.506 us, min = 7.207 us, total = 3.866 ms | |
| NodeManagerService.grpc_server.GetSystemConfig.HandleRequestImpl - 97 total (0 active), Execution time: mean = 56.241 us, total = 5.455 ms, Queueing time: mean = 327.679 us, max = 4.464 ms, min = 11.580 us, total = 31.785 ms | |
| NodeManagerService.grpc_server.GetSystemConfig - 97 total (0 active), Execution time: mean = 3.289 ms, total = 318.996 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManagerService.grpc_server.GetResourceLoad.HandleRequestImpl - 40 total (0 active), Execution time: mean = 113.612 us, total = 4.544 ms, Queueing time: mean = 46.334 ms, max = 1.403 s, min = 15.655 us, total = 1.853 s | |
| NodeManagerService.grpc_server.GetResourceLoad - 40 total (0 active), Execution time: mean = 51.644 ms, total = 2.066 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManager.ScheduleAndDispatchTasks - 39 total (1 active), Execution time: mean = 12.916 us, total = 503.717 us, Queueing time: mean = 32.346 ms, max = 1.166 s, min = 25.221 us, total = 1.261 s | |
| NodeManager.deadline_timer.spill_objects_when_over_threshold - 39 total (1 active), Execution time: mean = 5.206 us, total = 203.025 us, Queueing time: mean = 32.500 ms, max = 1.162 s, min = 15.645 us, total = 1.267 s | |
| NodeManager.deadline_timer.flush_free_objects - 39 total (1 active), Execution time: mean = 251.260 us, total = 9.799 ms, Queueing time: mean = 32.231 ms, max = 1.162 s, min = 16.470 us, total = 1.257 s | |
| NodeManagerService.grpc_server.CancelWorkerLease.HandleRequestImpl - 39 total (0 active), Execution time: mean = 49.727 us, total = 1.939 ms, Queueing time: mean = 730.121 us, max = 20.392 ms, min = 12.120 us, total = 28.475 ms | |
| NodeManagerService.grpc_server.CancelWorkerLease - 39 total (0 active), Execution time: mean = 1.204 ms, total = 46.948 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ClusterResourceManager.ResetRemoteNodeView - 14 total (1 active), Execution time: mean = 11.521 us, total = 161.288 us, Queueing time: mean = 3.930 ms, max = 52.391 ms, min = 32.476 us, total = 55.016 ms | |
| PeriodicalRunner.RunFnPeriodically - 13 total (0 active), Execution time: mean = 205.288 us, total = 2.669 ms, Queueing time: mean = 3.509 ms, max = 11.519 ms, min = 34.502 us, total = 45.612 ms | |
| Subscriber.HandleFailureCallback_WORKER_OBJECT_LOCATIONS_CHANNEL - 8 total (0 active), Execution time: mean = 291.490 us, total = 2.332 ms, Queueing time: mean = 269.496 us, max = 564.613 us, min = 111.922 us, total = 2.156 ms | |
| NodeManager.deadline_timer.record_metrics - 8 total (1 active), Execution time: mean = 1.385 ms, total = 11.081 ms, Queueing time: mean = 6.454 ms, max = 48.887 ms, min = 177.618 us, total = 51.630 ms | |
| ray::rpc::NodeInfoGcsService.grpc_client.CheckAlive - 8 total (0 active), Execution time: mean = 1.514 ms, total = 12.110 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManager.GcsCheckAlive - 8 total (1 active), Execution time: mean = 257.821 us, total = 2.063 ms, Queueing time: mean = 7.533 ms, max = 52.080 ms, min = 230.143 us, total = 60.266 ms | |
| ray::rpc::NodeInfoGcsService.grpc_client.CheckAlive.OnReplyReceived - 8 total (0 active), Execution time: mean = 55.690 us, total = 445.517 us, Queueing time: mean = 165.010 us, max = 341.070 us, min = 40.463 us, total = 1.320 ms | |
| NodeManager.deadline_timer.debug_state_dump - 4 total (1 active, 1 running), Execution time: mean = 1.685 ms, total = 6.740 ms, Queueing time: mean = 128.671 us, max = 304.977 us, min = 56.474 us, total = 514.683 us | |
| ray::rpc::InternalPubSubGcsService.grpc_client.GcsSubscriberCommandBatch - 2 total (0 active), Execution time: mean = 1.603 ms, total = 3.206 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.883 us, total = 3.765 us, Queueing time: mean = 216.000 ns, max = 369.000 ns, min = 63.000 ns, total = 432.000 ns | |
| ray::rpc::InternalPubSubGcsService.grpc_client.GcsSubscriberPoll - 2 total (1 active), Execution time: mean = 3.360 s, total = 6.720 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ray::rpc::InternalPubSubGcsService.grpc_client.GcsSubscriberCommandBatch.OnReplyReceived - 2 total (0 active), Execution time: mean = 202.470 us, total = 404.939 us, Queueing time: mean = 724.282 us, max = 1.335 ms, min = 113.127 us, total = 1.449 ms | |
| ray::rpc::JobInfoGcsService.grpc_client.AddJob.OnReplyReceived - 1 total (0 active), Execution time: mean = 59.264 us, total = 59.264 us, Queueing time: mean = 432.627 us, max = 432.627 us, min = 432.627 us, total = 432.627 us | |
| ray::rpc::JobInfoGcsService.grpc_client.GetAllJobInfo.OnReplyReceived - 1 total (0 active), Execution time: mean = 26.745 us, total = 26.745 us, Queueing time: mean = 212.085 us, max = 212.085 us, min = 212.085 us, total = 212.085 us | |
| Subscriber.HandlePublishedMessage_GCS_JOB_CHANNEL - 1 total (0 active), Execution time: mean = 89.941 us, total = 89.941 us, Queueing time: mean = 360.874 us, max = 360.874 us, min = 360.874 us, total = 360.874 us | |
| ray::rpc::InternalPubSubGcsService.grpc_client.GcsSubscriberPoll.OnReplyReceived - 1 total (0 active), Execution time: mean = 317.107 us, total = 317.107 us, Queueing time: mean = 148.715 us, max = 148.715 us, min = 148.715 us, total = 148.715 us | |
| ray::rpc::NodeInfoGcsService.grpc_client.GetAllNodeInfo.OnReplyReceived - 1 total (0 active), Execution time: mean = 151.759 us, total = 151.759 us, Queueing time: mean = 320.113 us, max = 320.113 us, min = 320.113 us, total = 320.113 us | |
| ray::rpc::NodeInfoGcsService.grpc_client.RegisterNode.OnReplyReceived - 1 total (0 active), Execution time: mean = 602.456 us, total = 602.456 us, Queueing time: mean = 93.381 us, max = 93.381 us, min = 93.381 us, total = 93.381 us | |
| ray::rpc::JobInfoGcsService.grpc_client.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.246 ms, total = 1.246 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ray::rpc::InternalKVGcsService.grpc_client.GetInternalConfig.OnReplyReceived - 1 total (0 active), Execution time: mean = 26.634 ms, total = 26.634 ms, Queueing time: mean = 118.282 us, max = 118.282 us, min = 118.282 us, total = 118.282 us | |
| ray::rpc::JobInfoGcsService.grpc_client.AddJob - 1 total (0 active), Execution time: mean = 1.800 ms, total = 1.800 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ray::rpc::NodeInfoGcsService.grpc_client.GetAllNodeInfo - 1 total (0 active), Execution time: mean = 1.845 ms, total = 1.845 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManager.GCTaskFailureReason - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| NodeManager.deadline_timer.print_event_loop_stats - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ray::rpc::InternalKVGcsService.grpc_client.GetInternalConfig - 1 total (0 active), Execution time: mean = 1.883 ms, total = 1.883 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| ray::rpc::NodeInfoGcsService.grpc_client.RegisterNode - 1 total (0 active), Execution time: mean = 1.902 ms, total = 1.902 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s | |
| DebugString() time ms: 2 |