Is it possible to train a neural network to learn something via video footage?

In other words, if i have a video teaching me how to draw an animal from scratch, can i then use this video to teach the computer to draw the animal in the same way?


Video footage is essentially a sequence of images, any image processing capabilities available to us through machine learning are possible when applied to videos using a sequencial network (lstm, RNN etc.)

So i guess the difficult part becomes mapping the activity to an action like moving a pen or something


Short answer: No.

Long answer: A neural network is a function that maps input data (e.g. a picture) to output data (e.g. probability that the picture contains a dog). What you propose does not seem like the sort of task this tool is suited for, though I'm not willing to claim that it can't be done.

If you are interested in computer-generated artwork, I encourage you to learn about Generative Adversarial Networks and style transfer.

  • $\begingroup$ You have only described a neural network as a classification tool.. what about reinforcement learning and regression-based nets? $\endgroup$ – Mark Markrowave Charlton Jun 13 '17 at 5:19
  • $\begingroup$ The use case doesn't change the fundamental concept of the net - it's still a function mapping input variables to output variables. $\endgroup$ – Jessica Alan Jun 13 '17 at 15:52
  • $\begingroup$ Ok, am i able to map 3d coordinates from a 2d input? lets say i make a 3d object and screenshot it, can i map one to the other? $\endgroup$ – Mark Markrowave Charlton Jun 13 '17 at 22:27
  • $\begingroup$ So a network that maps a 2D representation of an object to its 3D vertices? That sounds feasible. $\endgroup$ – Jessica Alan Jun 14 '17 at 15:07
  • $\begingroup$ ok cool im on to it! $\endgroup$ – Mark Markrowave Charlton Jun 14 '17 at 23:06

Just like the other answer - it can't really be done like that. However, you might want to take a look at Google Deepdream, which actually enhances images to look more like what they detected. E.g., if a cat is detected, it will make the picture more cat-like.


About the screenshot you mentioned in comments: don't forget that NN build the mappings from learning. With only one screenshot, you'll only be able to construct a NN that outputs the correct 3D for THAT screenshot. With many screenshots, you may come closer to what you want (AFAI understand your question).
But remember that without 2 eyes, you have problems with 3D objects. Doesn't mean it's impossible.
May I suggest you to have a look at convolutional neural networks also https://en.wikipedia.org/wiki/Convolutional_neural_network

  • $\begingroup$ So am i correct in saying that if i can get enough visual information of an object (ie enough screenshots of different angles) and encode it into a higher representation (like identifying that theres a face) i can map that encoded information to a mesh of 3d coordinates? $\endgroup$ – Mark Markrowave Charlton Jun 13 '17 at 23:06
  • $\begingroup$ If you know the angles (more precisely the position of the...say camera), you don't even have to use a NN. ;) $\endgroup$ – jcm69 Jun 13 '17 at 23:18
  • $\begingroup$ if i have a 3d mesh with geometry (lets say 4 square faces, each face containing 4 vertices) is there a way to represent the polygons edges? becuase if i have 4 vertices i still dont know where the edges are $\endgroup$ – Mark Markrowave Charlton Jun 13 '17 at 23:21
  • $\begingroup$ Ok, say you have 2 views from 2 cameras (with known position) of a single point (say a light), then you can reconstruct it's position in a 3D space. Easy. Based on that, you can do many things. That's been the case for years. But it's complicated and specialized. While NN have incredible results that classical approaches can't achieve in a quite simple way. Here, quite means that you've got to be familiar with NN. Often takes time, really ;) $\endgroup$ – jcm69 Jun 13 '17 at 23:40
  • $\begingroup$ ok but how can i represent a 3d mesh in terms of its faces instead of its vertices? $\endgroup$ – Mark Markrowave Charlton Jun 13 '17 at 23:42

Your Answer

