<p><em>Cross Modality Pre-training</em> may be the method you need. Proposed by <a href="https://arxiv.org/abs/1608.00859" rel="nofollow noreferrer">Wang et al. (2016)</a>, this method averages the weights of the pre-trained model across the channels in the first layer and replicates the mean by the number of target channels. The experiment result indicates that the network gets better performance by using this kind of pre-training method even it has 20 input channels and its input modality is not RGB.</p>
<p>To apply this, one can refer to <a href="https://stackoverflow.com/questions/53251827/pretrained-tensorflow-model-rgb-rgby-channel-extension">another answer</a> that use layer.get_weights() and layer.set_weights() to manually set the weights in the first layer of the pre-trained model.</p>
Change number of input channels to pretrained keras.applications model?
<p>I am prototyping a deep learning segmentation model that needs six channels of input (two aligned 448x448 RGB images under different lighting conditions). I wish to compare the performance of several pretrained models to that of my current model, which I trained from scratch. Can I use the pretrained models in <code>tf.keras.applications</code> for input images with more than 3 channels?</p>

<p>I tried applying a convolution first to reduce the channel dimension to 3 and then passed that output to <code>tf.keras.applications.DenseNet121()</code> but received the following error:</p>

<pre><code>import tensorflow as tf
dense_input = tf.keras.layers.Input(shape=(448, 448, 6))
dense_filter = tf.keras.layers.Conv2D(3, 3, padding='same')(dense_input)
dense_stem = tf.keras.applications.DenseNet121(include_top=False, weights='imagenet', input_tensor=dense_filter)
*** ValueError: You are trying to load a weight file containing 241 layers into a model with 242 layers.
</code></pre>

<p>Is there a better way to use pretrained models on data with a different number of input channels in keras? Will pretraining even help when the number of input channels is different?</p>