<p><code>in_channels</code> refers to the depth of the inputs to the constitutional layer. For example, if you are feed the layer with raw RGB images, then the depth will be 3, corresponding to the Red, Green, and Blue channels. This means that the kernels actually are 3D rather than 2D. The <code>out_channels</code> refer to the depth of output. Following picture from <a href="http://cs231n.github.io/convolutional-networks/" rel="nofollow noreferrer">here</a> illustrates an example with input depth of 3 and output depth of 5:</p>

<p><a href="https://i.stack.imgur.com/3bHh8.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3bHh8.jpg" alt="enter image description here"></a></p>

<p><code>properly define</code> is something done based on experiments. That is a network design issue. You may read about some of the famous architectures like <a href="http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf" rel="nofollow noreferrer">AlexNet</a> and <a href="https://arxiv.org/pdf/1409.1556/" rel="nofollow noreferrer">VGG-16</a> to see how network architectures are designed in practice.</p>
Tensorflow tf.nn.conv2d clarification
<p>In reading through the Tensorflow tutorial and API documentation, I do not understand how they defined the shape of the convolution input and filter arguments. The method is: <code>tf.nn.conv2d(input, filter, strides, padding, use_cudnn_on_gpu=None, data_format=None, name=None)</code>, where the input is shape: <code>[batch, in_height, in_width, in_channels]</code> and the filter is shape: <code>[filter_height, filter_width, in_channels, out_channels]</code>. If anyone could shed light on how to properly define the "in_channel" and "out_channel" sizes, that would be very helpful. </p>