<p>I work on a project that maintains a data lake that centralizes public information from the Brazilian government. Our pipelines run on a Kubernetes cluster.</p>
<p>I'm currently building a pipeline for labor market data. This is the bash script I use to download the data:</p>
<pre class="lang-sh prettyprint-override"><code>#!/bin/bash

# To run this script the user must run 'bash download.sh group', where group is cagedmov | cagedfor | cageddex.
# See explanation in the next comment:

# The microdata resulting from the new consolidation are made available in accordance with the
# month of disclosure, as of January 2020, containing three files for each
# competence. Following a consistent naming pattern, CAGEDMOVAYYYMM files
# bring the movements declared within the deadline with declaration competence
# same as YYYYMM. The CAGEDFORAAAMM files bring the declared moves
# outside the deadline with declaration competence equal to YYYYMM. the files
# CAGEDEXCAAAAMM bring the excluded movements with declaration competence
# of exclusion equal to YYYYMM

lower_group=$1
upper_group=${lower_group^^}

mkdir -p /tmp/novo_caged/$lower_group/input
ufs=('RO' 'AC' 'AM' 'RR' 'PA' 'AP' 'TO' 'MA' 'PI' 'CE' 'RN' 'PB' 'PE' 'AL' 'SE' 'BA' 'MG' 'ES' 'RJ' 'SP' 'PR' 'SC' 'RS' 'MS' 'MT' 'GO' 'DF')
anos=(2020 2021 2022)
meses=($(seq 1 1 12))

for uf in &quot;${ufs[@]}&quot;
do
    for ano in &quot;${anos[@]}&quot;
    do
        for mes in &quot;${meses[@]}&quot;
        do
            mkdir -p /tmp/novo_caged/$lower_group/ano=$ano/mes=$mes/sigla_uf=$uf/
        done
    done
done

cd /tmp/novo_caged/$lower_group/input
ftp_path=&quot;ftp://anonymous:anonymous@ftp.mtps.gov.br/pdet/microdados/NOVO CAGED/&quot;

pad_meses=($(echo {01..12}))
folders=($(seq 202001 1 202012))

for ano in &quot;${anos[@]}&quot;
do
    for mes in &quot;${pad_meses[@]}&quot;
    do
        wget &quot;$ftp_path$ano/$ano$mes/$upper_group$ano$mes.7z&quot;
        7z x -y $upper_group$ano$mes.7z
        rm *7z
    done
done
</code></pre>
<p>The script runs perfectly on my computer, but when I deploy to the Kubernetes cluster, the script throws an error <code>Failed to connect to ftp.mtps.gov.br port 21: Connection timed out</code>. Apparently, the address <code>ftp.mtps.gov.br</code> only accepts requests from IP addresses from Brazil. Is there a way to get around this restriction? It would be very important for our project to automate this ETL and have this data released in an updated way.</p>

Lucas
<p>You can use Tor as a sockx5 proxy, and configure it to let traffic exit from a specific country.<br />
In the torrc configuration file add these lines, or eventually modify existing ones.</p>
<pre><code>ExitNodes {br}
StrictNodes 1
</code></pre>
<p>The last thing, you need to tell your bash script to use tor.<br />
That can be done in different ways, the easiest one is to use the <strong>torify</strong> command.<br />
I suggest to test everything adding this line at the top of the script</p>
<pre><code>#!/bin/bash

curl https://api.myip.com;exit
</code></pre>
<p>This will give you evidence of the country being used as tor exit node. If it's ok, remove this test line.</p>
<p><a href="https://www.torproject.org/" rel="nofollow noreferrer">https://www.torproject.org/</a><br />
<a href="https://linux.die.net/man/1/torify" rel="nofollow noreferrer">https://linux.die.net/man/1/torify</a></p>
