Training products of expert capsules with mixing by dynamic routing
Michael Hauser
cs.LG cs.NE stat.ML


Transparency enhancement in a double-barrier structure by the Fano antiresonance
    I.V. Gornyi
    December 30, 2023
================================================================================




This study develops an unsupervised learning algorithm for products of expert capsules with dynamic routing. Analogous to binary-valued neurons in Restricted Boltzmann Machines, the magnitude of a squashed capsule firing takes values between zero and one, representing the probability of the capsule being on. This analogy motivates the design of an energy function for capsule networks. In order to have an efficient sampling procedure where hidden layer nodes are not connected, the energy function is made consistent with dynamic routing in the sense of the probability of a capsule firing, and inference on the capsule network is computed with the dynamic routing between capsules procedure. In order to optimize the log-likelihood of the visible layer capsules, the gradient is found in terms of this energy function. The developed unsupervised learning algorithm is used to train a capsule network on standard vision datasets, and is able to generate realistic looking images from its learned distribution.




§ INTRODUCTION


Products of experts models <cit.> were designed as an alternative to mixture models, where the sum over distributions is replaced by a product. Because there is a product, as opposed to a sum, if an individual expert votes a low probability, then the entire product  will have low probability. This is in contrast to a standard mixture model, where it only takes a single distribution to vote high probability for the data point to have high probability of existing, effectively ignoring the votes of all of the other distributions. A product of experts allows each expert to specialize on a single aspect of the data, and to disqualify a data point from existing only on this single aspect, whereas a sum of experts requires each distribution to model all aspects of the data, since within the sum of experts each distribution is essentially acting independently of the others.



Capsule networks <cit.> with dynamic routing <cit.> depart in many significant ways from neural networks. Artificial neural networks originate from the McCulloch-Pitts model <cit.>, where a scalar neuron fires if the weighted sum of the neuron's scalar inputs reaches a threshold. For example, the outputs of the softmax and max-pool functions depend not only on the previous layer inputs, but also on the outputs from the other nodes at the current layer; besides these exceptions most standard activation functions, such as the sigmoid and ReLU, follow the McCulloch-Pitts design. 

As an alternative to scalar-valued neurons, capsules are vector valued, which has at least two advantages. First, because they are vector valued, much more complex routing mechanisms, such as routing by agreement <cit.>, can be used to pass data forward through the network, as opposed to a simple weighted sum of inputs. Second, vectors have a magnitude and an orientation, which allows capsules to fire if their magnitude reaches a threshold, while their orientation can be used to represent the instantiation parameters of the data.

A major difficulty with capsule networks is that because they are still in their infancy, far fewer algorithmic tools exist to train and develop them. For example capsule networks have been trained via backpropagation <cit.> as well as expectation-maximization <cit.>. For unsupervised learning they have been trained as an autoencoder <cit.> as well as in a generative-adversarial setting <cit.>, but these are built off of backpropagation, which itself was designed for neural networks without dynamic routing. In their supervised settings they have been paired with a fully-connected decoder <cit.>, but this is to compel the network to learn invariant vector representations and help defend against adversarial attacks <cit.>.

For these reasons we design a product of expert capsules in an analogous way to a product of expert neurons. From this, an efficient unsupervised learning procedure is developed to train this product of expert capsules in a bottom up fashion, using the dynamic routing procedure to mix the model distribution. With a given energy function compatible with dynamic routing, the contrastive divergence is minimized to learn the underlying density.



§ ALGORITHM REVIEW


This section will briefly review roduct of experts as well as dynamic routing between capsules.



 §.§ Product of experts learning


As mentioned in the Introduction, product of experts models combine densities as products as opposed to sums <cit.>. If we divide the experts into a visible layer and hidden layer of binary valued experts, the energy of this configuration is given as


    -E(v,h) = 
    ∑_i,j w_ij v_i,h_j +
    ∑_i b_i v_i + 
    ∑_j c_j h_j


From this, the probability of a visible-hidden configuration is p(v,h)=1/Ze^-E(v,h), and marginalized over the hidden layer gives p(v) = ∑_h p(v,h)=1/Z∑_h e^-E(v,h). Because there are no intra-layer connections, we can efficiently sample the hidden layer nodes as p(h_j=1|v ) = 1/1+exp( - ∑_i w_ijv_j + c_j ) and the visible layer nodes as p(v_i=1|h ) = 1/1+exp( - ∑_j w_ijh_i + b_i ). The gradient of the log-likelihood is given by


    ∂log p/∂ w_ij(v) =
      p( h_j=1 | v ) v_i - 
      ∑_v
      p(v)
      p( h_j=1 | v ) v_i


The first term on the right-hand side of this equation is generated by the data, while the second term is the expectation over the product of experts model itself. Truly estimating the second term with MCMC requires mixing the density to infinity. Instead one usually minimizes the contrastive divergence <cit.>, which requires running the Markov chain once.



 §.§ Routing by agreement in capsule networks


This section will briefly review capsule networks and dynamic routing <cit.>.

At layer l, given a collection i=1,2,…,I of vector-valued capsules x^(l)_i and matrix-valued prediction maps W^(l)_ij, the pre-activation, vector-valued predicted capsule j from capsule i is z^(l+1)_j|i=W^(l)_ij· x^(l)_i,
 where at layer l+1 we have the collection j=1,2,…,J of capsules. We then take a weighted average of all of the predictions to yield the final, pre-activation, vector-valued capsule j at layer l+1, i.e.

    z^(l+1)_j=∑_ic^(l)_ij W^(l)_ij· x^(l)_i

where the scalar-valued c^(l)_ij's are determined by routing by agreement and each lower layer capsule can only make a finite amount of predictions, i.e. ∑_ic^(l)_ij=1.

Routing by agreement is a procedure that iteratively re-weighs each of the individual predictions to yield a final, collective prediction. If an individual prediction z^(l+1)_j|i agrees well with the collective prediction z^(l+1)_j, then we would like to increase the routing weight c^(l)_ij connecting these capsules. Agreement has been measured as both the inner product <cit.>, as well as the cosine distance <cit.>, between the individual predictions and the squashed collective prediction. In this paper we use the cosine distance. The updated weights are then renormlized ∑_ic^(l)_ij=1, and the process is repeated. (If the cosine distance is used, then squashing can happen outside this loop since the cosine distance takes the inner product of unit vectors, so the magnitude isn't needed.)

The squashing function <cit.> we use is defined as follows:


    x^(l+1)_j = 
        squash(z^(l+1)_j)=
        ‖ z^(l+1)_j‖^2/1+‖ z^(l+1)_j‖^2z^(l+1)_j/‖ z^(l+1)_j‖


The intention of the squashing function is to scale the magnitude of z^(l+1)_j between 0 and 1 while keeping the orientation the same. We can then interpret capsule j is on if the magnitude of the squashing function is close to 1, and j is off if the magnitude is close to 0:

    P(j=on|x^(l)_1,…,x^(l)_I) = 
        ‖squash(z^(l+1)_j) ‖




Later in the paper we need the inverse map of the squash map, which we call unsquash:


    z^(l+1)_j = 
        unsquash(x^(l+1)_j)=
        √(‖ x^(l+1)_j‖/1-‖ x^(l+1)_j‖)x^(l+1)_j/‖ x^(l+1)_j‖


This is just the pre-image of the squashed vector so (unsquash∘squash) (z) = identity_ℝ^d(z) = z, in a similar way as the logit map is the inverse map of the sigmoid.



§ CAPSULE NETWORKS AS PRODUCT OF EXPERTS


This section develops the product of expert capsules model, where a capsule being either on or off, measured by the magnitude of the capsule vector after squashing, is analogous to the binary action of a neuron firing in a Restricted Boltzmann Machine.

We denote an individual capsule by x^(l)_i for i=1,2,…,I. We also write x^(l)=(x^(l)_1,x^(l)_2,…,x^(l)_I ) to be the collection of all capsules on that layer. Similarly ‖ x^(l+1)_j ‖ is the norm of capsule j for j=1,2,…,J, and ‖ x^(l+1)‖ = (‖ x^(l+1)_1 ‖,‖ x^(l+1)_2 ‖,…,‖ x^(l+1)_J ‖).


























 §.§ Energy function and conditional probabilities


Before beginning we would like to make a subtle, yet important point. We note that if we want the magnitude of the squashed capsule to play an analogous role to the probability of a binary neuron firing, P(‖ x^(l+1)_j ‖=1  |x^(l))=‖squash( z^(l+1)_j ) ‖, for z^(l+1)_j=∑_i c^(l)_ijW^(l)_ij· x^(l)_i, the squashing part of the squashing function can be rewritten as a sigmoid activation, ‖ z^(l+1)_j ‖^2/1 + ‖ z^(l+1)_j ‖^2 = σ( log‖ z^(l+1)_j ‖^2 ). Intuitively, ‖ z^(l+1)_j ‖≥ 0, whereas the sigmoid function should take arguments over all of ℝ, so taking the logarithm of the norm maps positive numbers to all of ℝ.

Define the energy across layers as follows:


    E( x^(l),‖ x^(l+1)‖) = 
        - ∑_j
        log( ‖∑_i c^(l)_ijW^(l)_ij· x^(l)_i ‖^2 )
        ‖ x^(l+1)_j ‖
















The probability of a certain configuration ( x^(l),‖ x^(l+1)‖) is then defined

    P(x^(l),‖ x^(l+1)‖) = 
        1/Zexp( -
        E( x^(l),‖ x^(l+1)‖)
        )













where Z is the normalizing partition function. 
This is a product of expert capsules:


    P(x^(l),‖ x^(l+1)‖) =
        1/ZΠ_j [
        exp(
        log( 
        ‖∑_i c^(l)_ijW^(l)_ij· x^(l)_i
        ‖^2
        ) ‖ x^(l+1)_j ‖)]


We marginalize this distribution over the layer l+1 capsules firing:


    P(x^(l)) =
        1/ZΠ_j [
        1 +
        exp(
        log( 
        ‖∑_i c^(l)_ijW^(l)_ij· x^(l)_i
        ‖^2
        ) 
        )]


In order to have an efficient means of sampling the hidden layer, it is necessary that at layer l+1 the nodes representing the capsules firing are not connected, so that the total probability is equal to the product of individual capsule probabilities, and so using equations <ref> and <ref> we have:


    P(‖ x^(l+1)‖ | x^(l)) =
        
        Π_j
        P(‖ x^(l+1)_j ‖ | x^(l))


From here it is straightforward to show that the conditional probability of capsule j firing is equal to the magnitude of the squashed capsule:


    P(‖ x^(l+1)_j ‖ = 1 | x^(l)) =
    ‖∑_i c^(l)_ijW^(l)_ij· x^(l)_i ‖^2 /1+‖∑_i c^(l)_ijW^(l)_ij· x^(l)_i ‖^2


In this way, the energy function defined in Equation <ref> is consistent with routing by agreement, in the sense that the magnitude of the squashed capsule represents the probability that the individual capsule is on. 


    P( j = on | x^(l)) =
        ‖ x^(l+1)_j ‖ =
        P(‖ x^(l+1)_j ‖ = 1 | x^(l))

where again x^(l+1)_j = squash( z^(l+1)_j ) for z^(l+1)_j = ∑_i c^(l)_ijW^(l)_ij· x^(l)_i.

Because of this consistency, we use dynamic routing between capsules for mixing in the MCMC estimation of the distribution. Note that we do not want to directly sample x^(l+1)_j from a graphical model P ( x^(l+1)_j | x^(l)_i ). This is because x^(l+1)_j is a squashed vector, implying that the components of the vector are dependent on each other from the squashing, so graphically the nodes at that given layer are connected, meaning that we cannot efficiently sample these nodes with MCMC. By explicitly decoupling x^(l+1)_j into its magnitude and orientation, we are able to decouple the parts of the dynamic routing that we want to use from the parts we do not want to use, allowing us to efficiently sample the magnitude ‖ x^(l+1)_j ‖∼ P ( ‖ x^(l+1)_j ‖ | x^(l)_i ) from this distribution, since graphically these nodes are not connected, from Equation <ref>.

For inference, the energy model only requires P( ‖ x^(l+1)_j ‖ = 1 | x^(l)) = ‖ x^(l+1)_j ‖, which holds by construction. We then sample x^(l+1)_j ∼ P( x^(l+1)_j | x^(l)) as x^(l+1)_j =  squash( ∑_i c^(l)_ijW^(l)_ij· x^(l)_i ), from the dynamic routing, and again the norm of this is consistent with the energy model. Taking the norm of x^(l+1)_j introduces a rotational invariance to the vector, which at first may seem like a problem since the instantiation parameters are stored in the rotational angles of the capsule. In fact this is not an issue, as we will see in Section <ref>, because we are taking the gradient of the log likelihood, and the gradient of the norm of a vector is dependent on the orientation of the vector itself, not just its magnitude. In this way, during the gradient descent, information from the orientation of the capsule is used to update the model parameters.













 §.§ Gradient of the log-likelihood



In order to optimize our parameter weights we need the gradient of the log-likelihood to be tractably computable; the log-likelihood is given by:

    log P(  x^(l)) =
        log∑^1_‖ x^(l+1)‖ =0 e^-E( x^(l) , ‖ x^(l+1)‖) 
        - 
        log∑_x^(l)∑^1_‖ x^(l+1)‖ =0 e^-E( x^(l) , ‖ x^(l+1)‖)

We can put the gradient of the log-likelihood in a compact form using the fact that P( ‖ x^(l+1)‖ | x^(l)) = 
P( x^(l) , ‖ x^(l+1)‖)/P( x^(l)) = 
1/Zexp-E( x^(l) , ‖ x^(l+1)‖)/1/Z∑^1_‖ x^(l+1)‖ =0exp-E( x^(l) , ‖ x^(l+1)‖).

    ∂/∂ W^(l)_ijlog P(  x^(l)) =
        ∑^1_‖ x^(l+1)‖ =0 
        P( ‖ x^(l+1)‖ | x^(l))
        ∂ E /∂ W^(l)_ij( x^(l) , ‖ x^(l+1)‖)
        - 
    ∑_x^(l) 
        P( x^(l))
        ∑^1_‖ x^(l+1)‖ =0 
        P( ‖ x^(l+1)‖ | x^(l))
        ∂ E /∂ W^(l)_ij( x^(l) , ‖ x^(l+1)‖)

Computing this is still not tractable as it involves an exponential number of sums. However, since the layer l+1 capsule activations are not connected, from Equation <ref>, we can apply an analogous factorization trick to that which is used in learning Restricted Boltzmann Machines. First we reduce the energy function:

    ∂ E /∂ W^(l)_ij( x^(l) , ‖ x^(l+1)‖) =
        ∂/∂ W^(l)_ij∑_j'log( ‖∑_i' c^(l)_i'j'W^(l)_i'j'· x^(l)_i'‖^2 )
        ‖ x^(l+1)_j'‖
        =
        
    ∂/∂ W^(l)_ijlog( ‖∑_i' c^(l)_i'jW^(l)_i'j· x^(l)_i'‖^2 )
        ‖ x^(l+1)_j‖

Using this, we are in a position to find the tractable gradient for the product of expert capsules:

    ∑^1_‖ x^(l+1)‖ =0 
        P( ‖ x^(l+1)‖ | x^(l))
        ∂ E /∂ W^(l)_ij( x^(l) , ‖ x^(l+1)‖)
        = 
    ∑^1_‖ x^(l+1)_j‖ =0
        P( ‖ x^(l+1)_j ‖ | x^(l))
        ∂/∂ W^(l)_ijlog( ‖
        z^(l+1)_j‖^2 )
        ‖ x^(l+1)_j‖∑^1_‖ x^(l+1)_-j‖ =0
        P( ‖ x^(l+1)_-j‖ | x^(l))
        
     = 
        P( ‖ x^(l+1)_j ‖ = 1 | x^(l))
        ∂/∂ W^(l)_ijlog( ‖
        z^(l+1)_j‖^2 )

where x^(l+1)_-j is analogous notation to the RBM case <cit.>, and refers to all capsules at layer l+1 other than the j^th capsule. 

To further reduce Equation <ref>, where we write z^(l+1)_j = ∑_i' c^(l)_i'jW^(l)_i'j· x^(l)_i', one has ∂/∂ W^(l)_ijlog( ‖ z^(l+1)_j‖^2 ) = 2 c^(l)_ij/‖  
z^(l+1)_j‖^2 z^(l+1)_j x^(l)_i. Similarly, P( ‖ x^(l+1)_j ‖ = 1 | x^(l)) = ‖ z^(l+1)_j‖^2/1+‖ z^(l+1)_j‖^2, and we have the final form of the gradient:


    ∂/∂ W^(l)_ijlog P( x^(l)) =
        2 c^(l)_ij(
        z^(l+1)_j x^(l)_i/1+‖  
        z^(l+1)_j‖^2
        -
        ∑_x^(l)
        P( x^(l))
        z^(l+1)_j x^(l)_i/1+‖  
        z^(l+1)_j‖^2)


For RBM learning, this is analogous to the update rule Δ w_ij = ⟨ v_i h_j ⟩_data - ⟨ v_i h_j ⟩_model. 

Thus, beginning with the data x^(l)_i, we use dynamic routing to sample z^(l+1)_j = ∑_i c^(l)_ijW^(l)_ij· x^(l)_i where the c^(l)_ij's are determined by routing by agreement and ∑_i c^(l)_ij = 1. 

We then squash this x^(l+1)_j = squash( z^(l+1)_j) and run dynamic routing in reverse to produce z̃^(l)_i = ∑_j c^(l)_ijW^(l)T_ij· x^(l+1)_j, using the same c^(l)_ij's as before, with ∑_i c^(l)_ij = 1. This is squashed producing x̃^(l)_i.
Finally, we repeat the first step, again with the same c^(l)_ij's such that ∑_i c^(l)_ij = 1 to produce z̃^(l+1)_j = ∑_i c^(l)_ijW^(l)_ij·x̃^(l)_i and its squashed counterparts x̃^(l+1)_j. This procedure is summarized in Algorithm <ref>.

We then use these values, which we are referring to as mixing by dynamic routing, for the gradient in Equation <ref>. As is usual, instead of minimizing the log likelihood we minimize the contrastive divergence, and so we only run this mixing once. The complete process is summarized in Algorithm <ref>.





§ EXPERIMENTS




 §.§ Network architecture


The outline of the experimental architecture can be seen in Figure <ref>. Other than the input/output channels being greyscale and rgb, the same architecture was used for both experiments. First an autoencoding convolutional network <cit.>, with dropout <cit.> was used to learn the convolutional filter weights in an unsupervised way. The first filter bank is of size [9,9,1,128] (for height × width × channels in × channels out), followed by a leaky ReLU activation, while the second filter bank is [9,9,128,128] followed by a leaky ReLU activation. The transposed filters are used in the decoder, with first a leaky ReLU and a sigmoid at the output, to scale the pixel intensities between 0 and 1.

Once the convolutional autoencoder is sufficiently trained, these weights are held fixed and the 128-dimensional hidden layer is reshaped to 6× 6× 128/8=576 capsules, each being 8-dimensional, and then squashed along these 8 dimensions. Using the unsupervised product of expert capsules training algorithm described above, these capsules are mapped to 20 capsules, each of 16-dimensions, to learn the W^(l)_ij's in z^(l+1)_j=∑_i c^(l)_ij W^(l)_ij· x^(l)_i, where i=1,2,…,576 and j=1,2,…,20.

After the W^(l)_ij's are trained, these weights are held fixed and we learn a decoder z^(l)_i=∑_j e^(l+1)_ji U^(l+1)_ji· x^(l+1)_j so that we can sample from the 16-dimensional capsule space to generate the images. The update rule is Δ U^(l+1)_ji = 2e^(l+1)_ji( z^(l)_ix^(l+1)_j/1+‖ z^(l)_i‖^2 - 
z̃^(l)_ix̃^(l+1)_j/1+‖z̃^(l)_i‖^2), where (the data) z^(l)_i and x^(l+1)_j are fixed by the W^(l)_ij's, while the (the model) z̃^(l)_i=∑_j e^(l+1)_ji U^(l+1)_ji x^(l+1)_j and x̃^(l+1)_j = squash( ∑_j e^(l+1)_ji U^(l+1)T_jix̃^(l)_j) are generated with the U^(l+1)_ji's. We implemented this network in TensorFlow <cit.>, and training on a single gpu took about fifteen minutes.





 §.§ Routing-Weighted Product of Expert Neurons


In Figure <ref> there are 20 columns for each of the 20 capsules, and the 4 rows are random samples from a 16-dimensional Gaussian distrbition, with the other 19 capsules having 0 as their input. 
It is seen that each of the individual capsules learn specific objects, with different random samples generating images with different instantiation parameters of similar objects, such as stroke thicknesses and angles, or sleeves coming out of pants to create shirts and jackets. Interestingly some of the images are negatives of what is expected, where if the input x∼ N(0,1) is replaced with -x the image generated are no longer negatives, but the ones that were normal become negatives, as is seen in Figures <ref> and <ref>.

We believe this stems from the fact that the 16 dimensional point, as understood by the capsule, lives on the manifold S^15×(0,1), where S^15 is the 15-dimensional sphere of angular orientations, and the (0,1) is the capsule magnitude representing off or on. Not all object instantiation parameters should live on the sphere. For example rotating an object in a circle should live on S^1, but increasing the size of an object should live on ℝ, since an object's size shouldn't return to where it started if the size is monotonically increasing, as would happen on S^1. In this way the pixel intensities can be reversed 180^∘ since they are coming from S^15 as opposed to ℝ^15. When we restrict our sampling domain to the half of S^15 that is visited during training the problem is alleviated, as seen in Figures <ref> and <ref>, suggesting that this idea is infact correct.

A more elegant solution, although outside the scope of this paper, would be to design a capsule on, say, S^m ×ℝ^n ×(0,1). Nevertheless, the unsupervised learning procedure developed here for the capsules is distinctly learning recognizable objects.



§ CONCLUSIONS


This work developed capsule networks with routing by agreement within a product of experts formulation. Observing that the magnitudes of the hidden layer capsules are not connected, we design an energy function that is consistent with dynamic routing, in the sense that the binary action of a hidden layer capsule firing is equal to the probability of a capsule being on, when calculated with dynamic routing by agreement. We then use dynamic routing to mix the distribution. The gradient of the log likelihood is found and used to minimize the contrastive divergence. A simple network architecture is set up to test this unsupervised learning algorithm, and is able to generate images similar to those of the datasets it was trained on.








10

hinton2002training
Geoffrey E Hinton.
Training products of experts by minimizing contrastive divergence.
Neural computation, 14(8):1771–1800, 2002.

hinton2011transforming
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang.
Transforming auto-encoders.
In International Conference on Artificial Neural Networks,
  pages 44–51. Springer, 2011.

sabour2017dynamic
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton.
Dynamic routing between capsules.
In Advances in neural information processing systems, pages
  3856–3866, 2017.

mcculloch1943logical
Warren S McCulloch and Walter Pitts.
A logical calculus of the ideas immanent in nervous activity.
The bulletin of mathematical biophysics, 5(4):115–133, 1943.

rumelhart1985learning
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.
Learning internal representations by error propagation.
Technical report, California Univ San Diego La Jolla Inst for
  Cognitive Science, 1985.

hinton2018matrix
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst.
Matrix capsules with em routing.
2018.

jaiswal2018capsulegan
Ayush Jaiswal, Wael AbdAlmageed, Yue Wu, and Premkumar Natarajan.
Capsulegan: Generative adversarial capsule network.
In Proceedings of the European Conference on Computer Vision
  (ECCV), pages 0–0, 2018.

goodfellow2014generative
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
  Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In Advances in neural information processing systems, pages
  2672–2680, 2014.

goodfellow2014explaining
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples.
arXiv preprint arXiv:1412.6572, 2014.

welling2005exponential
Max Welling, Michal Rosen-Zvi, and Geoffrey E Hinton.
Exponential family harmoniums with an application to information
  retrieval.
In Advances in neural information processing systems, pages
  1481–1488, 2005.

zeiler2014visualizing
Matthew D Zeiler and Rob Fergus.
Visualizing and understanding convolutional networks.
In European conference on computer vision, pages 818–833.
  Springer, 2014.

srivastava2014dropout
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
  Salakhutdinov.
Dropout: a simple way to prevent neural networks from overfitting.
The Journal of Machine Learning Research, 15(1):1929–1958,
  2014.

abadi2016tensorflow
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey
  Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.
Tensorflow: A system for large-scale machine learning.
In 12th {USENIX} Symposium on Operating Systems Design and
  Implementation ({OSDI} 16), pages 265–283, 2016.



