title: install python packages using init scripts in a databricks cluster
body: <p>I have installed the databricks cli tool by running the following command</p>
  <p><code>pip install databricks-cli</code> using the appropriate version of pip for your Python installation. If you are using Python 3, run pip3.</p>
  <p>Then by creating a PAT (personal-access token in Databricks) I run the following .sh bash script:</p>
  <pre class="lang-sh prettyprint-override"><code># You can run this on Windows as well, just change to a batch files
  # Note: You need the Databricks CLI installed and you need a token configued
  #!/bin/bash
  echo &quot;Creating DBFS direcrtory&quot;
  dbfs mkdirs dbfs:/databricks/packages
  
  echo &quot;Uploading cluster init script&quot;
  dbfs cp --overwrite python_dependencies.sh                     dbfs:/databricks/packages/python_dependencies.sh
  
  echo &quot;Listing DBFS direcrtory&quot;
  dbfs ls dbfs:/databricks/packages
  </code></pre>
  <p><strong>python_dependencies.sh script</strong></p>
  <pre class="lang-sh prettyprint-override"><code>#!/bin/bash
  # Restart cluster after running.
  
  sudo apt-get install applicationinsights=0.11.9 -V -y
  sudo apt-get install azure-servicebus=0.50.2 -V -y
  sudo apt-get install azure-storage-file-datalake=12.0.0 -V -y
  sudo apt-get install humanfriendly=8.2 -V -y
  sudo apt-get install mlflow=1.8.0 -V -y
  sudo apt-get install numpy=1.18.3 -V -y
  sudo apt-get install opencensus-ext-azure=1.0.2 -V -y
  sudo apt-get install packaging=20.4 -V -y
  sudo apt-get install pandas=1.0.3 -V -y
  sudo apt update
  sudo apt-get install scikit-learn=0.22.2.post1 -V -y
  status=$?
  echo &quot;The date command exit status : ${status}&quot;
  </code></pre>
  <p>I use the above script to install python libraries in the init-scripts of the cluster</p>
  <p><a href="https://i.stack.imgur.com/9WiU5.png" rel="noreferrer"><img src="https://i.stack.imgur.com/9WiU5.png" alt="enter image description here" /></a></p>
  <p>My problem is that even though everything seems to be fine and the cluster is started successfully, the libraries are not installed properly. When I click on the libraries tab of the cluster I get this:</p>
  <p><a href="https://i.stack.imgur.com/04K6g.png" rel="noreferrer"><img src="https://i.stack.imgur.com/04K6g.png" alt="enter image description here" /></a>
  Only 1 out of the 10 python libraries is installed.</p>
  <p>Appreciate your help and comments.</p>