title: How can i segment clients on granular level using python and ML?
body: <h3>Novice alert</h3>
  <p>I have been learning ML in python for the last few months and have had some great results. Currently, however, I am stuck with a project and require the guidance of someone with more experience (Google can only take you so far it appears  ).</p>
  <h3>What I am trying to achieve</h3>
  <p>I have a dummy data set full of clients and their transactions. I want to cluster or segment them into much smaller 'tribes' based on their demographic data, spending score and shopping behaviour. For example, one 'tribes' description could be something this granular : (men, aged 35 who primarily purchase music-based products on a Saturday afternoon within the first half of each month and have a high spending score)
  I want to find the sweet spot between granular segmentation and general segmentation eg: segmentation by income and spending score.</p>
  <h3>What I have tried</h3>
  <p>Firstly, I have allocated an int value representing the frequency of each categorical occurrence across each client's transactions. For example :</p>
  <pre><code>Client | Home  | Movies | Games 
      1      3        1       0
  </code></pre>
  <p>This indicates that Client 1 has purchased Home related items 3 times, Movie related items 1 time and they have never purchased any item in the Games category.</p>
  <p>I have done the same for  Days (i.e Sunday - Saturday), Week number (i.e 1-5 the week number in any given month), Hour (i.e hour_one - hour_twenty_four).</p>
  <p>This approach allows me to create a clean vector of purely numerical data.</p>
  <p>This is an example of my raw input data in JSON format (before processing):</p>
  <pre><code>[
      {
          &quot;id&quot;: 1,
          &quot;customer_id&quot;: 1,
          &quot;age&quot;: 47,
          &quot;gender&quot;: &quot;Female&quot;,
          &quot;first_name&quot;: &quot;Lea&quot;,
          &quot;last_name&quot;: &quot;Calafato&quot;,
          &quot;email&quot;: &quot;lcalafato0@cafepress.com&quot;,
          &quot;phone_number&quot;: &quot;612-170-5956&quot;,
          &quot;income_k&quot;: 24,
          &quot;location&quot;: &quot;Nottingham&quot;,
          &quot;sign_up_date&quot;: &quot;2/16/2019&quot;,
          &quot;transactions&quot;: [
              {
                  &quot;customer_id&quot;: &quot;1&quot;,
                  &quot;product_id&quot;: 42,
                  &quot;product_cat&quot;: &quot;Home&quot;,
                  &quot;price&quot;: 106.92,
                  &quot;time&quot;: &quot;8:15 PM&quot;,
                  &quot;date&quot;: &quot;04/15/2019&quot;,
                  &quot;day&quot;: &quot;Monday&quot;,
                  &quot;week_num&quot;: 3
              },
              {
                  &quot;customer_id&quot;: &quot;1&quot;,
                  &quot;product_id&quot;: 30,
                  &quot;product_cat&quot;: &quot;Movies&quot;,
                  &quot;price&quot;: 26.63,
                  &quot;time&quot;: &quot;10:12 AM&quot;,
                  &quot;date&quot;: &quot;09/17/2019&quot;,
                  &quot;day&quot;: &quot;Tuesday&quot;,
                  &quot;week_num&quot;: 4
              }
          ],
          &quot;number_of_purchases&quot;: 2,
          &quot;last_purchase&quot;: &quot;09/17/2019&quot;,
          &quot;total_spent&quot;: 133.55
      }
  ]
  </code></pre>
  <p>This is my dataframe after being processed and standardized :</p>
  <pre><code>age                  750 non-null int64
  income_k             750 non-null int64
  spending_score       750 non-null int64
  gender__Female       750 non-null uint8
  gender__Male         750 non-null uint8
  Home                 750 non-null float64
  Movies               750 non-null float64
  Games                750 non-null float64
  Grocery              750 non-null float64
  Music                750 non-null float64
  Health               750 non-null float64
  Beauty               750 non-null float64
  Sports               750 non-null float64
  Toys                 750 non-null float64
  Garden               750 non-null float64
  Computers            750 non-null float64
  Clothing             750 non-null float64
  Books                750 non-null float64
  Outdoors             750 non-null float64
  Industrial           750 non-null float64
  Kids                 750 non-null float64
  Tools                750 non-null float64
  Automotive           750 non-null float64
  Electronics          750 non-null float64
  Jewelery             750 non-null float64
  Baby                 750 non-null float64
  Shoes                750 non-null float64
  week_one             750 non-null float64
  week_two             750 non-null float64
  week_three           750 non-null float64
  week_four            750 non-null float64
  week_five            750 non-null float64
  Sunday               750 non-null float64
  Monday               750 non-null float64
  Tuesday              750 non-null float64
  Wednesday            750 non-null float64
  Thursday             750 non-null float64
  Friday               750 non-null float64
  Saturday             750 non-null float64
  hour_one             750 non-null float64
  hour_two             750 non-null float64
  hour_three           750 non-null float64
  hour_four            750 non-null float64
  hour_five            750 non-null float64
  hour_six             750 non-null float64
  hour_seven           750 non-null float64
  hour_eight           750 non-null float64
  hour_nine            750 non-null float64
  hour_ten             750 non-null float64
  hour_eleven          750 non-null float64
  hour_twelve          750 non-null float64
  hour_thirteen        750 non-null float64
  hour_fourteen        750 non-null float64
  hour_fithteen        750 non-null float64
  hour_sixteen         750 non-null float64
  hour_seventeen       750 non-null float64
  hour_eighteen        750 non-null float64
  hour_nineteen        750 non-null float64
  hour_twenty          750 non-null float64
  hour_twenty_one      750 non-null float64
  hour_twenty_two      750 non-null float64
  hour_twenty_three    750 non-null float64
  hour_twenty_four     750 non-null float64*
  </code></pre>
  <p>I have run this data through both k-means and DBSCAN algorithms to no avail. k-means gives me 4 clusters which are far too general for my requirements and DBSCAN gives me zero clusters with each data point being treated as noise.</p>
  <p>My apologies if anything is unclear, please feel free to ask me to clarify anything. Thanks in advance.</p>