title: tensorflow.js - image prediction - main color of image
body: <p>I hope you can help me. I'm fooling around with my first tensorflow.js Model a little and got stuck by improving it.</p>
  
  <p>The Task it should predict sounds "simple" : get color of object in Image.</p>
  
  <p>But I'm stuck maybe someone can give me a hint :-)</p>
  
  <p>My model:</p>
  
  <ul>
  <li><p>sequential</p></li>
  <li><p>conv2d:  filters: 32,
         kernelSize: [3, 3],
         padding: "same",
         inputShape: [32, 32, 3],
         kernelInitializer: "varianceScaling",</p></li>
  <li><p>activation: relu</p></li>
  <li><p>maxPooling2D: PoolSize: [2,2] strides: 2</p></li>
  <li><p>dropOut: rate 0.5</p></li>
  <li><p>conv2d:  filters: 64,
         kernelSize: [3, 3],
         padding: "same",
         kernelInitializer: "varianceScaling",</p></li>
  <li><p>activation: relu</p></li>
  <li><p>maxPooling2D: PoolSize: [2,2] strides: 2</p></li>
  <li><p>dropOut: rate 0.5</p></li>
  <li><p>flatten</p></li>
  <li><p>dense: units: 1024</p></li>
  <li><p>activation: relu</p></li>
  <li><p>batchNormalization: axis -1</p></li>
  <li><p>dropout: 0.5</p></li>
  <li><p>dense: units  (e.g.22)</p></li>
  <li><p>activation: softmax</p></li>
  </ul>
  
  <p><a href="https://i.stack.imgur.com/q5MNn.png" rel="nofollow noreferrer">summary of my Model</a></p>
  
  <hr>
  
  <p><strong>My Data</strong></p>
  
  <ul>
  <li>one List of all colors which exist inside data (total 22 entries)</li>
  <li>an array of image dir to their mapped color example [ [dir=>"img/image1.jpeg", label: "red"],[dir=>"img/image2.jpeg", label: "black"]</li>
  </ul>
  
  <hr>
  
  <p><strong>My Process</strong></p>
  
  <p><em>For Images</em>
  Read Image with tf.node.decodeJpeg</p>
  
  <p>Normalize it and push it into array of tensors</p>
  
  <pre><code>const offset = tf.scalar(127.5);
  var tensor=tf.node.decodeJpeg(img, 3).resizeNearestNeighbor([32,32])
  tensor = tensor.toFloat();
  tensor = tensor.sub(offset).div(offset);
  imageTensorArray.push(tensor);
  </code></pre>
  
  <p><em>for Label</em></p>
  
  <p>get Numeric Index of Labels from Alllabels Array and oneHot binarize it.</p>
  
  <pre><code>var imageLabels = imageLabelArray.map(function (d) {
          return allLabelsUnique.indexOf(d[1]); //index of label String in allLabels
      });
  const imageLabelsHot=tf.oneHot(imageLabels, allLabelsUnique.length);
  </code></pre>
  
  <p><em>compile Model</em></p>
  
  <pre><code>const optimizer = tf.train.adam(1e-3, 1e-3 / 20);
  model.compile({
      optimizer: optimizer,
      loss: "categoricalCrossentropy",
      metrics: ['accuracy']
  });
  </code></pre>
  
  <p><em>train Model</em></p>
  
  <pre><code> await model.fit(trainImageTensors //5000, trainImageLabels //5000, {  ///all in the right order 
          epochs: 20,
          verbose: 1,
          validationSplit: 0.3,
          shuffle: 1,
          batchSize: 32
      });
  </code></pre>
  
  <hr>
  
  <p>Now my Problem(s):</p>
  
  <ul>
  <li><p>acc is getting better - val_acc stucks at ~ 0.339, loss: decrease - val_loss not regardless of epochs or amount of training data</p></li>
  <li><p>prediction is always black</p></li>
  </ul>
  
  <p>I've played around a lot but i can't get the "one" hint for help </p>
  
  <ul>
  <li>trainImageTensors = tf.stack(buildImageToTensor(image)) - could this be a problem?</li>
  </ul>
  
  <p>Model Pipeline:</p>
  
  <p>I have a csv it is just 
  color_i, color_ii, url to Picture</p>
  
  <p>I Then check if it is already on my local disc if not i download it an save it in a smaller size:</p>
  
  <pre><code>img=canvas.loadImage(imageUrl) 
  var cvs = new canvas.Canvas(32, 32);
  var ctx = cvs.getContext('2d');
  ctx.drawImage(img, 0, 0, img.width, img.height, 0, 0, cvs.width, cvs.height);
  var data = ctx.canvas.toDataURL('image/jpeg').split(',');
              var buf = new Buffer.alloc(data[1].length, data[1], 'base64');
  var dir = getlocalDir(url);
  if (!fs.existsSync('img')) {
   fs.mkdirSync('img');
  }
  fs.writeFileSync(dir.toString(), buf, '');
  </code></pre>
  
  <p>After I have the picture i or if it already on the storage I read it as a Tensor using builtin functions</p>
  
  <pre><code> //if local exists
   var dir = getlocalDir(url)
   var img = fs.readFileSync(dir);
   var tensor = tf.node.decodeJpeg(img, 3);
   //if not exists - ctx still there at the moment
   var tensor = tf.browser.fromPixels(ctx.canvas);
  // then doing normalization
   tensor = tensor.toFloat();
   tensor = tensor.div(255.0);
   tensor = tf.image.cropAndResize(tensor.expandDims(0), [[15, 15, 30, 30]], [0], [15, 15], 'nearest').reshape([15, 15, 3]);
   imageTensorArray.push(tensor);
  </code></pre>
  
  <p>After this the work beginns</p>
  
  <pre><code>const trainImageTensors = tf.slice(imageTensorArray, 0, totalSum);
  
  ///dataArray.images: [0]=&gt;Url, [1]=&gt;color (red,blue,green, etc...)
  ///dataArray.labels: collection of all colors available in dataset
  var imageLabels = dataArray.images.map(function (d) {      
          return dataArray.labels.indexOf(d[1]);
      });
  
  const imageLabelsHot = tf.oneHot(imageLabels, dataArray.labels.length);
  const trainImageLabels = tf.slice(imageLabelsHot, 0, totalSum);
   const optimizer = tf.train.adam(0.00001, 0.00001 / 50,);
      model.compile({
          optimizer: optimizer,
          loss: "categoricalCrossentropy",
          metrics: ['accuracy']
      });
  
      await model.fit(trainImageTensors, trainImageLabels, {
          epochs: 50,
          verbose: 1,
          validationSplit: 0.4,
          shuffle: 1,
          batchSize: 100,
      });
  
  </code></pre>