<p>You can read about the work of Jacques Mattheij, he actually uses a customized version of Xception<sup>1</sup> running on <a href="https://keras.io/" rel="nofollow noreferrer">https://keras.io/</a>.</p>

<p>The introduction is <a href="https://jacquesmattheij.com/sorting-two-metric-tons-of-lego" rel="nofollow noreferrer">Sorting 2 Metric Tons of Lego</a>.</p>

<p>In <a href="https://jacquesmattheij.com/sorting-lego-the-software-side" rel="nofollow noreferrer">Sorting 2 Tons of Lego, The software Side</a> you can read:</p>

<blockquote>
  <p>The hard challenge to deal with next was to get a training set large
  enough to make working with 1000+ classes possible. At first this
  seemed like an insurmountable problem. I could not figure out how to
  make enough images and to label them by hand in acceptable time, even
  the most optimistic calculations had me working for 6 months or longer
  full-time in order to make a data set that would allow the machine to
  work with many classes of parts rather than just a couple.</p>
  
  <p>In the end the solution was staring me in the face for at least a week
  before I finally clued in: it doesn’t matter. All that matters is that
  the machine labels its own images most of the time and then all I need
  to do is correct its mistakes. As it gets better there will be fewer
  mistakes. This very rapidly expanded the number of training images.
  The first day I managed to hand-label about 500 parts. The next day
  the machine added 2000 more, with about half of those labeled wrong.
  The resulting 2500 parts where the basis for the next round of
  training 3 days later, which resulted in 4000 more parts, 90% of which
  were labeled right! So I only had to correct some 400 parts, rinse,
  repeat… So, by the end of two weeks there was a dataset of 20K images,
  all labeled correctly.</p>
  
  <p>This is far from enough, some classes are severely under-represented
  so I need to increase the number of images for those, perhaps I’ll
  just run a single batch consisting of nothing but those parts through
  the machine. No need for corrections, they’ll all be labeled
  identically.</p>
</blockquote>

<p>A recent update is <a href="https://jacquesmattheij.com/sorting-lego-many-questions-and-this-is-what-the-result-looks-like" rel="nofollow noreferrer">Sorting 2 Tons of Lego, Many Questions, Results</a>.</p>

<p><hr>
<sup>1</sup><a href="https://arxiv.org/abs/1610.02357" rel="nofollow noreferrer">CHOLLET, François. Xception: Deep Learning with Depthwise Separable Convolutions. <em>arXiv preprint arXiv:1610.02357</em>, 2016.</a></p>
Using tensorflow to identify lego bricks?
<p>having read <a href="https://cloud.google.com/blog/big-data/2016/08/how-a-japanese-cucumber-farmer-is-using-deep-learning-and-tensorflow" rel="nofollow" title="this article">this article</a> about a guy who uses tensorflow to sort cucumber into nine different classes I was wondering if this type of process could be applied to a large number of classes. My idea would be to use it to identify Lego parts. </p>

<p>At the moment, a site like Bricklink describes more than <a href="https://www.bricklink.com/catalogList.asp?catType=P&amp;catLike=W" rel="nofollow">40,000 different parts</a> so it would be a bit different than the cucumber example but I am wondering if it sounds suitable. There is no easy way to get hundreds of pictures for each part but does the following process sound feasible : </p>

<ul>
<li>take pictures of a part ; </li>
<li>try to identify the part using tensorflow ;</li>
<li>if it does not identify the correct part, take more pictures and feed the neural network with them ;</li>
<li>go on with the next part.</li>
</ul>

<p>That way, each time we encounter a new piece we "teach" the network its reference so that it can better be recognized the next time. Like that and after hundreds of iterations monitored by a human, could we imagine tensorflow to be able to recognize the parts? At least the most common ones?</p>

<p>My question might sound stupid but I am not into neural networks so any advice is welcome. At the moment I have not found any way to identify a lego part based on pictures and this "cucumber example" sounds promising so I am looking for some feedback.</p>

<p>Thanks.</p>