Systems and methods for training networks are provided. A method for training networks comprises receiving an input from each of a plurality of neural networks differing from each other in at least one of architecture, input modality, and feature type, connecting the plurality of neural networks through a common output layer, or through one or more common hidden layers and a common output layer to result in a joint network, and training the joint network.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for training networks, the method comprising: receiving an input from each of a plurality of neural networks differing from each other in at least one of architecture, input modality, and feature type; connecting the plurality of neural networks through a common output layer, or through one or more common hidden layers and a common output layer to result in a joint network; and training the joint network, wherein the receiving, connecting and training steps are performed by a computer system comprising a memory and at least one processor coupled to the memory. 2. The method according to claim 1, further comprising defining a joint architecture where the plurality of neural networks are connected by at least one common hidden layer and one common output layer. 3. The method according to claim 2, wherein training the joint network comprises backpropagating an error signal through the at least one common hidden layer and the one common output layer and through the plurality of neural networks. 4. The method according to claim 3, wherein the error signal is a partial derivative of an error function which measures classification or regression error of the joint network. 5. The method according to claim 2, wherein training the joint network comprises updating weights of the at least one common hidden layer, the one common output layer and the plurality of neural networks through stochastic gradient descent. 6. The method according to claim 2, wherein training the joint network comprises updating weights of the at least one common hidden layer, the one common output layer and the plurality of neural networks through second-order optimization. 7. The method according to claim 2, wherein a first neural network of the plurality of neural networks comprises an input layer connected to the at least one common hidden layer, and a second neural network of the plurality of neural networks comprises a convolutional layer connected to the at least one common hidden layer. 8. The method according to claim 1, wherein the joint network is used as an acoustic model in an automatic speech recognition system. 9. The method according to claim 1, wherein the training is performed jointly on at least one of different modalities and feature streams. 10. The method according to claim 1, wherein the plurality of neural networks comprise a deep neural network and convolutional neural network. 11. A system for training networks, the system comprising: an input module capable of receiving an input from each of a plurality of neural networks differing from each other in at least one of architecture, input modality, and feature type; a connection module capable of connecting the plurality of neural networks through a common output layer, or through one or more common hidden layers and a common output layer to result in a joint network; and a training module capable of training the joint network. 12. The system according to claim 11, further comprising a defining module capable of defining a joint architecture where the plurality of neural networks are connected by at least one common hidden layer and one common output layer. 13. The system according to claim 12, wherein the training module is capable of performing the training by backpropagating an error signal through the at least one common hidden layer and the one common output layer and through the plurality of neural networks. 14. The system according to claim 13, wherein the error signal is a partial derivative of an error function which measures classification or regression error of the joint network. 15. The system according to claim 12, wherein the training module is capable of performing the training by updating weights of the at least one common hidden layer, the one common output layer and the plurality of neural networks through stochastic gradient descent. 16. The system according to claim 2, wherein the training module is capable of performing the training by updating weights of the at least one common hidden layer, the one common output layer and the plurality of neural networks through second-order optimization. 17. The system according to claim 12, wherein a first neural network of the plurality of neural networks comprises an input layer connected to the at least one common hidden layer, and a second neural network of the plurality of neural networks comprises a convolutional layer connected to the at least one common hidden layer. 18. The system according to claim 11, wherein the training module is capable of performing the training jointly on at least one of different modalities and feature streams. 19. The system according to claim 11, wherein the plurality of neural networks comprise a deep neural network and convolutional neural network. 20. A computer program product for training networks, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising: receiving an input from each of a plurality of neural networks differing from each other in at least one of architecture, input modality, and feature type; connecting the plurality of neural networks through a common output layer, or through one or more common hidden layers and a common output layer to result in a joint network; and training the joint network.