<p>When I run <code>kubectl delete raycluster &lt;raycluster-name&gt;</code>, sometimes this command hangs. It looks like this is because Kubernetes finalizers for the raycluster are preventing deletion of the resource until some condition is met. Indeed, I see the raycluster gets marked with a deletion timestamp like below:</p>
<pre><code>  creationTimestamp: &quot;2022-02-17T06:06:40Z&quot;
  deletionGracePeriodSeconds: 0
  deletionTimestamp: &quot;2022-02-17T18:51:16Z&quot;
  finalizers:
  - kopf.zalando.org/KopfFinalizerMarker
</code></pre>
<p>Looking at the logs, if termination happens correctly, I should see termination requests on the operator logs:</p>
<pre><code>2022-02-16 16:57:26,326 VINFO scripts.py:853 -- Send termination request to `&quot;/home/ray/anaconda3/lib/python3.9/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server *:50343&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot; &quot;&quot;` (via SIGTERM)
2022-02-16 16:57:26,328 VINFO scripts.py:853 -- Send termination request to `/home/ray/anaconda3/lib/python3.9/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2022-02-16_16-41-20_595437_116/sockets/raylet --store_socket_name=/tmp/ray/session_2022-02-16_16-41-20_595437_116/sockets/plasma_store --object_manager_port=0 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_ip_address=10.1.0.34 --redis_address=10.1.0.34 --redis_port=6379 --maximum_startup_concurrency=1 --static_resource_list=node:10.1.0.34,1.0,memory,367001600,object_store_memory,137668608 &quot;--python_worker_command=/home/ray/anaconda3/bin/python /home/ray/anaconda3/lib/python3.9/site-packages/ray/workers/setup_worker.py /home/ray/anaconda3/lib/python3.9/site-packages/ray/workers/default_worker.py --node-ip-address=10.1.0.34 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-02-16_16-41-20_595437_116/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-02-16_16-41-20_595437_116/sockets/raylet --redis-address=10.1.0.34:6379 --temp-dir=/tmp/ray --metrics-agent-port=45522 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER --redis-password=5241590000000000&quot; --java_worker_command= &quot;--cpp_worker_command=/home/ray/anaconda3/lib/python3.9/site-packages/ray/cpp/default_worker --ray_plasma_store_socket_name=/tmp/ray/session_2022-02-16_16-41-20_595437_116/sockets/plasma_store --ray_raylet_socket_name=/tmp/ray/session_2022-02-16_16-41-20_595437_116/sockets/raylet --ray_node_manager_port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --ray_address=10.1.0.34:6379 --ray_redis_password=5241590000000000 --ray_session_dir=/tmp/ray/session_2022-02-16_16-41-20_595437_116 --ray_logs_dir=/tmp/ray/session_2022-02-16_16-41-20_595437_116/logs --ray_node_ip_address=10.1.0.34 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER&quot; --native_library_path=/home/ray/anaconda3/lib/python3.9/site-packages/ray/cpp/lib --redis_password=5241590000000000 --temp_dir=/tmp/ray --session_dir=/tmp/ray/session_2022-02-16_16-41-20_595437_116 --log_dir=/tmp/ray/session_2022-02-16_16-41-20_595437_116/logs --resource_dir=/tmp/ray/session_2022-02-16_16-41-20_595437_116/runtime_resources --metrics-agent-port=45522 --metrics_export_port=43650 --object_store_memory=137668608 --plasma_directory=/dev/shm --ray-debugger-external=0 &quot;--agent_command=/home/ray/anaconda3/bin/python -u /home/ray/anaconda3/lib/python3.9/site-packages/ray/dashboard/agent.py --node-ip-address=10.1.0.34 --redis-address=10.1.0.34:6379 --metrics-export-port=43650 --dashboard-agent-port=45522 --listen-port=0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/tmp/ray/session_2022-02-16_16-41-20_595437_116/sockets/plasma_store --raylet-name=/tmp/ray/session_2022-02-16_16-41-20_595437_116/sockets/raylet --temp-dir=/tmp/ray --session-dir=/tmp/ray/session_2022-02-16_16-41-20_595437_116 --runtime-env-dir=/tmp/ray/session_2022-02-16_16-41-20_595437_116/runtime_resources --log-dir=/tmp/ray/session_2022-02-16_16-41-20_595437_116/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --redis-password=5241590000000000&quot;` (via SIGTERM)
</code></pre>
<p>However, in the case above where the finalizer condition is not met, I don't see the termination requests in the logs:</p>
<pre><code>Demands:
 (no resource demands)
ray,ray:2022-02-16 17:21:10,145 DEBUG gcs_utils.py:253 -- internal_kv_put b'__autoscaling_status_legacy' b&quot;Cluster status: 2 nodes\n - MostDelayedHeartbeats: {'10.244.0.11': 0.17503762245178223, '10.244.1.33': 0.17499160766601562, '10.244.0.12': 0.17495203018188477}\n - NodeIdleSeconds: Min=3926 Mean=3930 Max=3937\n - ResourceUsage: 0.0/3.0 CPU, 0.0 GiB/1.05 GiB memory, 0.0 GiB/0.38 GiB object_store_memory\n - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0\nWorker node types:\n - rayWorkerType: 2&quot; True None
ray,ray:2022-02-16 17:21:10,145 DEBUG legacy_info_string.py:24 -- Cluster status: 2 nodes
 - MostDelayedHeartbeats: {'10.244.0.11': 0.17503762245178223, '10.244.1.33': 0.17499160766601562, '10.244.0.12': 0.17495203018188477}
 - NodeIdleSeconds: Min=3926 Mean=3930 Max=3937
 - ResourceUsage: 0.0/3.0 CPU, 0.0 GiB/1.05 GiB memory, 0.0 GiB/0.38 GiB object_store_memory
 - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0
Worker node types:
 - rayWorkerType: 2
ray,ray:2022-02-16 17:21:10,220 DEBUG autoscaler.py:1148 -- ray-ray-worker-type-f5gsr is not being updated and passes config check (can_update=True).
ray,ray:2022-02-16 17:21:10,245 DEBUG autoscaler.py:1148 -- ray-ray-worker-type-fwkp7 is not being updated and passes config check (can_update=True).
ray,ray:2022-02-16 17:21:10,268 DEBUG autoscaler.py:1148 -- ray-ray-worker-type-f5gsr is not being updated and passes config check (can_update=True).
ray,ray:2022-02-16 17:21:10,285 DEBUG autoscaler.py:1148 -- ray-ray-worker-type-fwkp7 is not being updated and passes config check (can_update=True).
ray,ray:2022-02-16 17:21:10,389 DEBUG resource_demand_scheduler.py:189 -- Cluster resources: [{'CPU': 1.0, 'node:10.244.0.11': 1.0, 'object_store_memory': 135078297.0, 'memory': 375809638.0}, {'node:10.244.1.33': 1.0, 'memory': 375809638.0, 'object_store_memory': 137100902.0, 'CPU': 1.0}, {'object_store_memory': 134204620.0, 'CPU': 1.0, 'node:10.244.0.12': 1.0, 'memory': 375809638.0}]
ray,ray:2022-02-16 17:21:10,389 DEBUG resource_demand_scheduler.py:190 -- Node counts: defaultdict(&lt;class 'int'&gt;, {'rayHeadType': 1, 'rayWorkerType': 2})
ray,ray:2022-02-16 17:21:10,389 DEBUG resource_demand_scheduler.py:201 -- Placement group demands: []
ray,ray:2022-02-16 17:21:10,389 DEBUG resource_demand_scheduler.py:247 -- Resource demands: []
ray,ray:2022-02-16 17:21:10,389 DEBUG resource_demand_scheduler.py:248 -- Unfulfilled demands: []
ray,ray:2022-02-16 17:21:10,389 DEBUG resource_demand_scheduler.py:252 -- Final unfulfilled: []
ray,ray:2022-02-16 17:21:10,440 DEBUG resource_demand_scheduler.py:271 -- Node requests: {}
ray,ray:2022-02-16 17:21:10,488 DEBUG gcs_utils.py:253 -- internal_kv_put b'__autoscaling_status' b'{&quot;load_metrics_report&quot;: {&quot;usage&quot;: {&quot;object_store_memory&quot;: [0.0, 406383819.0], &quot;memory&quot;: [0.0, 1127428914.0], &quot;node:10.244.0.11&quot;: [0.0, 1.0], &quot;CPU&quot;: [0.0, 3.0], &quot;node:10.244.1.33&quot;: [0.0, 1.0], &quot;node:10.244.0.12&quot;: [0.0, 1.0]}, &quot;resource_demand&quot;: [], &quot;pg_demand&quot;: [], &quot;request_demand&quot;: [], &quot;node_types&quot;: [[{&quot;memory&quot;: 375809638.0, &quot;CPU&quot;: 1.0, &quot;node:10.244.0.11&quot;: 1.0, &quot;object_store_memory&quot;: 135078297.0}, 1], [{&quot;object_store_memory&quot;: 137100902.0, &quot;node:10.244.1.33&quot;: 1.0, &quot;memory&quot;: 375809638.0, &quot;CPU&quot;: 1.0}, 1], [{&quot;object_store_memory&quot;: 134204620.0, &quot;memory&quot;: 375809638.0, &quot;node:10.244.0.12&quot;: 1.0, &quot;CPU&quot;: 1.0}, 1]], &quot;head_ip&quot;: null}, &quot;time&quot;: 1645060869.937817, &quot;monitor_pid&quot;: 68, &quot;autoscaler_report&quot;: {&quot;active_nodes&quot;: {&quot;rayHeadType&quot;: 1, &quot;rayWorkerType&quot;: 2}, &quot;pending_nodes&quot;: [], &quot;pending_launches&quot;: {}, &quot;failed_nodes&quot;: []}}' True None
ray,ray:2022-02-16 17:21:15,493 DEBUG gcs_utils.py:238 -- internal_kv_get b'autoscaler_resource_request' None
ray,ray:2022-02-16 17:21:15,640 INFO autoscaler.py:304 --
======== Autoscaler status: 2022-02-16 17:21:15.640853 ========
Node status
---------------------------------------------------------------
Healthy:
 1 rayHeadType
 2 rayWorkerType
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/3.0 CPU
 0.00/1.050 GiB memory
 0.00/0.378 GiB object_store_memory

Demands:
 (no resource demands)
ray,ray:2022-02-16 17:21:15,683 DEBUG gcs_utils.py:253 -- internal_kv_put b'__autoscaling_status_legacy' b&quot;Cluster status: 2 nodes\n - MostDelayedHeartbeats: {'10.244.0.11': 0.14760899543762207, '10.244.1.33': 0.14756131172180176, '10.244.0.12': 0.1475226879119873}\n - NodeIdleSeconds: Min=3932 Mean=3936 Max=3943\n - ResourceUsage: 0.0/3.0 CPU, 0.0 GiB/1.05 GiB memory, 0.0 GiB/0.38 GiB object_store_memory\n - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0\nWorker node types:\n - rayWorkerType: 2&quot; True None
ray,ray:2022-02-16 17:21:15,684 DEBUG legacy_info_string.py:24 -- Cluster status: 2 nodes
 - MostDelayedHeartbeats: {'10.244.0.11': 0.14760899543762207, '10.244.1.33': 0.14756131172180176, '10.244.0.12': 0.1475226879119873}
 - NodeIdleSeconds: Min=3932 Mean=3936 Max=3943
 - ResourceUsage: 0.0/3.0 CPU, 0.0 GiB/1.05 GiB memory, 0.0 GiB/0.38 GiB object_store_memory
 - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0
Worker node types:
 - rayWorkerType: 2
ray,ray:2022-02-16 17:21:15,775 DEBUG autoscaler.py:1148 -- ray-ray-worker-type-f5gsr is not being updated and passes config check (can_update=True).
ray,ray:2022-02-16 17:21:15,799 DEBUG autoscaler.py:1148 -- ray-ray-worker-type-fwkp7 is not being updated and passes config check (can_update=True).
ray,ray:2022-02-16 17:21:15,833 DEBUG autoscaler.py:1148 -- ray-ray-worker-type-f5gsr is not being updated and passes config check (can_update=True).
ray,ray:2022-02-16 17:21:15,850 DEBUG autoscaler.py:1148 -- ray-ray-worker-type-fwkp7 is not being updated and passes config check (can_update=True).
ray,ray:2022-02-16 17:21:15,962 DEBUG resource_demand_scheduler.py:189 -- Cluster resources: [{'memory': 375809638.0, 'node:10.244.0.11': 1.0, 'CPU': 1.0, 'object_store_memory': 135078297.0}, {'CPU': 1.0, 'node:10.244.1.33': 1.0, 'object_store_memory': 137100902.0, 'memory': 375809638.0}, {'memory': 375809638.0, 'node:10.244.0.12': 1.0, 'CPU': 1.0, 'object_store_memory': 134204620.0}]
ray,ray:2022-02-16 17:21:15,962 DEBUG resource_demand_scheduler.py:190 -- Node counts: defaultdict(&lt;class 'int'&gt;, {'rayHeadType': 1, 'rayWorkerType': 2})
ray,ray:2022-02-16 17:21:15,963 DEBUG resource_demand_scheduler.py:201 -- Placement group demands: []
ray,ray:2022-02-16 17:21:15,963 DEBUG resource_demand_scheduler.py:247 -- Resource demands: []
ray,ray:2022-02-16 17:21:15,963 DEBUG resource_demand_scheduler.py:248 -- Unfulfilled demands: []
ray,ray:2022-02-16 17:21:15,963 DEBUG resource_demand_scheduler.py:252 -- Final unfulfilled: []
ray,ray:2022-02-16 17:21:16,032 DEBUG resource_demand_scheduler.py:271 -- Node requests: {}
ray,ray:2022-02-16 17:21:16,081 DEBUG gcs_utils.py:253 -- internal_kv_put b'__autoscaling_status' b'{&quot;load_metrics_report&quot;: {&quot;usage&quot;: {&quot;memory&quot;: [0.0, 1127428914.0], &quot;object_store_memory&quot;: [0.0, 406383819.0], &quot;CPU&quot;: [0.0, 3.0], &quot;node:10.244.0.11&quot;: [0.0, 1.0], &quot;node:10.244.1.33&quot;: [0.0, 1.0], &quot;node:10.244.0.12&quot;: [0.0, 1.0]}, &quot;resource_demand&quot;: [], &quot;pg_demand&quot;: [], &quot;request_demand&quot;: [], &quot;node_types&quot;: [[{&quot;node:10.244.0.11&quot;: 1.0, &quot;object_store_memory&quot;: 135078297.0, &quot;CPU&quot;: 1.0, &quot;memory&quot;: 375809638.0}, 1], [{&quot;object_store_memory&quot;: 137100902.0, &quot;node:10.244.1.33&quot;: 1.0, &quot;CPU&quot;: 1.0, &quot;memory&quot;: 375809638.0}, 1], [{&quot;object_store_memory&quot;: 134204620.0, &quot;node:10.244.0.12&quot;: 1.0, &quot;CPU&quot;: 1.0, &quot;memory&quot;: 375809638.0}, 1]], &quot;head_ip&quot;: null}, &quot;time&quot;: 1645060875.4946475, &quot;monitor_pid&quot;: 68, &quot;autoscaler_report&quot;: {&quot;active_nodes&quot;: {&quot;rayHeadType&quot;: 1, &quot;rayWorkerType&quot;: 2}, &quot;pending_nodes&quot;: [], &quot;pending_launches&quot;: {}, &quot;failed_nodes&quot;: []}}' True None
</code></pre>
<p>Reading through documentations, I found 2 workarounds:</p>
<pre><code>1. Use kubectl patch to remove the finalizer
2. Kill and restart the operator, this lifts the finalizer condition 
</code></pre>
<p>However, I am not sure if either method is sustainable because:</p>
<pre><code>1. After I run kubectl patch, I can't seem to create new rayclusters with the same name. 
   This requires me to kill and restart the operator.
2. If I restart the operator to bring down a raycluster, 
   I am afraid this will affect other rayclusters that are currently running.
</code></pre>
<p>I am looking to understand the following:</p>
<ol>
<li>What happens if I restart the ray operator while other rayclusters are active?</li>
<li>What is the finalizer condition here and can I disable it?</li>
<li>Some suitable workarounds</li>
</ol>

<p>Assuming the operator is running when you try to delete the resource, the hanging behavior is a bug.
Would you mind filing a <a href="https://github.com/ray-project/ray/issues/new?assignees=&amp;labels=bug%2Ctriage&amp;template=bug-report.yml&amp;title=%5BBug%5D%20" rel="nofollow noreferrer">bug report</a> on the Ray GitHub with reproduction details?</p>
<p>Edit: I'm one of the maintainers of Ray's Kubernetes support.
Feel to tag me with &quot;@DmitriGekhtman&quot; in the bug report.</p>
