<p>I wrote a script that checks the kubernetes pods and in case of hanging it must write pod logs and delete it. Kube master is on another server and I try to connect to it by ssh. It's working good when I try to start script locally on kube master server, but when I try to enter the same command as ssh argument, I get pod deleted, but this pod's logs weren't written and there's no errors, so I don't understand what I do wrong. I use python 3.6, access permissions 777 for scripts and directories in which they are located.</p>
<p>I know about possibility of doing this using k8s api but I don't know how to do it now. I am still learning and will try to realize it a bit later but the system should work now.</p>
<p>PS. The reason that I do this is that I want to add 'check_radio' to cron and forget about hanging pods...
PPS. I am not master-pro at python, just a regular sysadmin that wants to become a devops, so if you have some ideas how to optimize my work you're welcome.</p>
<p>Here is the script that is on kube master (kube_reload.py):</p>
<pre><code>#!/bin/python3


import sys
import os
import re
import subprocess as sp
from contextlib import redirect_stdout as r_stdout


marker = False
pod_nums = ''
pod_nums_int = []
for_reload = []

get_pods = sp.check_output(['kubectl', 'get', 'pods'])
get_pods = str(get_pods)
get_pods = get_pods.split()

pods = []


'''write garbage pod names in array'''

for word in get_pods:
    if 'radio-hls' in word:
        pods.append(word) 

'''make usual pod names from garbage names'''

for num in range(len(pods)):

    while marker == False:
        if pods[num][:5] != 'radio':
            pods[num] = pods[num][1:]
        if pods[num][:5] == 'radio':
            break


'''Function that lists all pods'''

def pod_list():

    sp.run('clear')
    print('PODS LIST:\n')
    for num in range(len(pods)):
        print(num+1, '.\t', pods[num])

    print(len(pods)+1, '.\t Reload all')

'''Function for recursion in try-except statement in 'input_nums()' function'''

def return_for_except():
    pod_list()
    print('Error input. Try again\n\n')
    input_nums()

'''Function that asks which pods you want to reload'''

def input_nums():
    pod_nums = str(input('Select which pods you want to reload (type all numbers with a space):\nExample: 1 2 3 10\n\n'))
    pod_nums = pod_nums.split()

    try:
        global pod_nums_int
        pod_nums_int = [eval(i) for i in pod_nums]
    except:
        return return_for_except()

'''Function to write pod logs to a file'''

def write_logs():

    global for_reload

    if len(pods)+1 in pod_nums_int:

        all_reload = input('You selected &quot;Reload all&quot;. To continue, press Enter\n')

        if all_reload == '':
            for i in range(len(pods)):
                with open (f'{pods[i-1][10:-17]}.log', 'w') as pod_log:
                                    sp.run(['kubectl', 'logs', f'{pods[i-1]}'], stdout=pod_log)
                print(f'{pods[i-1]} logged successfully')
                for_reload.append(i)
        else:
            print('Something went wrong')
            quit()
    else:
        for i in pod_nums_int:

            with open (f'{pods[i-1][10:-17]}.log', 'w') as pod_log:
                sp.run(['kubectl', 'logs', f'{pods[i-1]}'], stdout=pod_log)
            print(f'{pods[i-1]} logged successfully')
            for_reload.append(i)


'''Function that reloads streams'''

def reload_pods(for_reload, pods):
    for i in for_reload:
        sp.run(['kubectl', 'delete', 'pod', f'{pods[i-1]}'])
        print(f'{pods[i-1]}', 'deleted')






'''Start'''

'''manual (with no arguments)'''
if len(sys.argv) == 1:
    pod_list()
    input_nums()
    write_logs()
    reload_pods(for_reload, pods)
    sp.run(['kubectl', 'get', 'pods'])
    print()
    sp.run(['ls', '-lah'])

'''auto (from nginx srver)'''

if len(sys.argv) &gt; 1:
    for arg in sys.argv:
        if arg == './kube_reload.py':
            continue
        else:
            for pod in pods:
                if arg in pod:
                    #write logs
                    with open (f'{arg}.log', 'w') as log:
                        sp.run(['kubectl', 'logs', f'{pod}'], stdout=log)
                    #reload pods
                    sp.run(['kubectl', 'delete', 'pod', f'{pod}'])
                else:
                    continue
</code></pre>
<p>Here is the script from another server (check_radio):</p>
<pre><code>#!/bin/python3


import requests as r
import subprocess as sp
import sys

'''IN CASE OF ADDING ADDITIONAL STREAM ADD IT TO &quot;streams&quot; '''

streams = [
'gold',
'tophits',
'worldchart',
'ukraine',
'rock',
'chill',
'rap',
'retromusic',
'elektro',
'sport',
'radionv',
'edyninov',
'hromadske'
]

'''IF IF THERE IS NO NEED TO CHECK SOME STREAMS YOU CAN ADD IT TO &quot;streams_not_to_check&quot; '''

streams_not_to_check = [
'radionv',
'edyninov',
'hromadske'
]

streams_to_delete = []

#CLUSTER_API = 'https://host_ip:6443'
#auth_header = 'Authorization: '

for stream in streams:
    check_stream = r.get(f'https://host.host/stream/{stream}/status/health').json()

    if check_stream['metadata'] == 'UNHEALTHY':
        streams_to_delete.append(stream)

for stream in streams_not_to_check:
    if stream in streams_to_delete:
        streams_to_delete.remove(stream)

print(streams_to_delete)




if len(streams_to_delete) &gt;= 1:

    for stream in streams_to_delete:

        sp.Popen(f'ssh developer@radio1 python3 ~/deekly/kube_reload.py {stream}', shell=True).communicate()

</code></pre>
<p>I try this from nginx server</p>
<pre><code>./check_radio
</code></pre>
<p>and get this</p>
<pre><code>[developer@radio-lb1 deekly]$ ./check_radio
['rap', 'rock']
pod &quot;radio-hls-rap-f4b86bd77-jpmr4&quot; deleted
pod &quot;radio-hls-rock-57fc8fcd64-m54k5&quot; deleted
[developer@radio-lb1 deekly]$

</code></pre>
<p>and this on kube server</p>
<pre><code>[developer@radio1 deekly]$ ls -la
total 8
drwxrwxrwx  2 developer developer   28 Aug  4 20:46 .
drwx------ 14 developer developer 4096 Aug  4 20:26 ..
-rwxrwxrwx  1 developer developer 2850 Aug  4 19:51 kube_reload.py
[developer@radio1 deekly]$

</code></pre>
<p>but when I try this from kube master</p>
<pre><code>./kube_reload.py rap rock
</code></pre>
<p>or this (doesn't matter)</p>
<pre><code>python3 kube_reload.py rap rock
</code></pre>
<p>I get this</p>
<pre><code>[developer@radio1 deekly]$ ./kube_reload.py rap rock
pod &quot;radio-hls-rap-f4b86bd77-4bvcs&quot; deleted
pod &quot;radio-hls-rock-57fc8fcd64-g5j96&quot; deleted
[developer@radio1 deekly]$ ls -la
total 32
drwxrwxrwx  2 developer developer   59 Aug  5 16:38 .
drwx------ 14 developer developer 4096 Aug  4 20:26 ..
-rwxrwxrwx  1 developer developer 2850 Aug  4 19:51 kube_reload.py
-rw-rw-r--  1 developer developer 8303 Aug  5 16:38 rap.log
-rw-rw-r--  1 developer developer 8345 Aug  5 16:38 rock.log

</code></pre>

Deekly
<p>Your <code>kube_reload.py</code> script writes its logs to its current working directory:</p>
<pre><code>with open (f'{arg}.log', 'w') as log:
...
</code></pre>
<p>When you're running kube_reload.py interactively, you're running it from this &quot;deekly&quot; directory, so &quot;deekly&quot; is the process's working directory and the logs are written there:</p>
<pre><code>[developer@radio1 deekly]$ ./kube_reload.py rap rock
...
[developer@radio1 deekly]$ ls -la
...
-rw-rw-r--  1 developer developer 8303 Aug  5 16:38 rap.log
-rw-rw-r--  1 developer developer 8345 Aug  5 16:38 rock.log
</code></pre>
<p>When this <code>check_radio</code> invokes <code>kube_reload.py</code> through <code>ssh</code>, the <code>kube_reload.py</code> process has this &quot;developer&quot; user's home directory as its working directory, not the &quot;deekly&quot; directory:</p>
<pre><code>sp.Popen(f'ssh developer@radio1 python3 ~/deekly/kube_reload.py {stream}', shell=True)...
</code></pre>
<p><code>kube_reload.py</code> will presumably write its log files to this &quot;developer&quot; user's home directory, not the &quot;deekly&quot; subdirectory.</p>
<p>If you want the logs to be written to this &quot;deekly&quot; directory, you should do one of three things:</p>
<ol>
<li><p>Modify <code>kube_reload.py</code> to put the logs where you want them, instead of its current working directory.</p>
</li>
<li><p>Modify <code>kube_reload.py</code> to change its working directory the desired directory before opening the logs.</p>
</li>
<li><p>Modify <code>check_radio</code> to invoke <code>kube_reload.py</code> on the remote host with the correct working directory. I'm not a python programmer so I can't give you the exact python syntax. But a command such as the following should do it:</p>
<p>ssh developer@radio1 'cd deekly &amp;&amp; python3 ./kube_reload.py args...'</p>
</li>
</ol>
<p>You will have to do whatever is necessary in python to escape those single quotes, so that they're present in the command actually being executed.</p>
