K-Origins: Better Colour Quantification for Neural Networks
Lewis Mason Mark Martinez



Saturation of magnetised plasma turbulence by propagating zonal flows
    W. Dorland^4
    September 9, 2024
=====================================================================




§ ABSTRACT 


K-Origins is a neural network layer designed to improve image-based network performances when learning colour, or intensities, is beneficial. Over 250 encoder-decoder convolutional networks are trained and tested on 16-bit synthetic data, demonstrating that K-Origins improves semantic segmentation accuracy in two scenarios: object detection with low signal-to-noise ratios, and segmenting multiple objects that are identical in shape but vary in colour. K-Origins generates output features from the input features, X, by the equation Y_k = X-J· w_k for each trainable parameter w_k, where J is a matrix of ones. Additionally, networks with varying receptive fields were trained to determine optimal network depths based on the dimensions of target classes, suggesting that receptive field lengths should exceed object sizes. By ensuring a sufficient receptive field length and incorporating K-Origins, we can achieve better semantic network performance. Examples of these improvements are illustrated in Figure <ref>.







§ INTRODUCTION



Semantic segmentation classifies 2D or 3D images on a pixel-by-pixel basis. It is especially valuable for processing large datasets that are impractical to classify manually. In biomedical and materials science, semantic segmentation is particularly useful for two tasks: distinguishing objects from the background and differentiating tracer particles from non-tracer particles.



The first class of problems, object segmentation, involves distinguishing one, or more, target classes from the background. Examples include <cit.> where white blood cell nuclei are segmented, <cit.> where abdominal organs and regions of interest are segmented, and <cit.> where X-ray tomography images of materials such as liquid-solid composites and ore-particles are segmented. This segmentation problem is prevalent in engineering, biomedical research, and materials sciences.

On the other hand, tracer segmentation involves distinguishing objects that are nearly identical in shape, but vary by colour or intensity. An example of this problem is segmenting X-ray images where contrast enhancing agents have been used (<cit.> and <cit.>). Another example is cancer cell segmentation in pathology slides, like in <cit.>, where cancerous cells can vary from normal cells by colour. The tracer segmentation problem is also relevant for datasets that produces a large number of false positives during segmentation.

Convolutional neural networks (CNNs) have shown to be very good at semantic segmentation and one of the most popular architecture styles for this task is the encoder-decoder network. This architecture makes predictions by combining low-level and high-level image features, effectively integrating information from various fields of view to achieve optimal results. The encoder-decoder network was mainly popularized by U-Net (<cit.>) which has been cited over 89,000 times. Because of its wide spread use, U-Net serves as a basic blueprint for the architectures used in this paper.


The receptive field (RF) is a key characteristic of CNNs. It represents the network's 2D or 3D field of view, indicating how much of the input image is used at each feature layer. To differentiate the RF, an area or volume, from its side length, we refer to the side length as the receptive field length (RFL). The RFL can be calculated for one side using the recursive equation from <cit.>:


    r_l-1 = s_l · r_l + (k_l - s_l)


To determine the RFL before a layer in the network (r_l-1) given the RFL after that layer (r_l), use the layer's stride (s_l) and kernel size (k_l) in Equation <ref>. For semantic segmentation, start at the deepest set of features with an RFL of one (r_l=end = 1 pixel) and work backwards to determine the RFL at each feature layer. 

The RFL at the beginning of semantic networks can then be thought of as the side length of the area or volume used for a single pixel's prediction. For example, a 2D semantic network with an RFL of 11 uses an 11x11 pixel area to generate the features used for classifying the central pixel. If the RFL is symmetrical in all directions, it only needs to be calculated for one dimension.

RFs have been extensively studied in various articles, with some focusing on determining optimal sizes and the effective RFLs in networks: <cit.>. However, many studies use complex datasets, making it difficult to generalize the findings.


Increasing the complexity of neural networks often improves training accuracy but results in longer training times and higher hardware costs. With millions of trainable parameters, it also becomes difficult to understand what the network is learning. It would be beneficial if networks could be made smaller and more efficient without hurting their performance.

This paper aims to reduce neural network complexity by building architectures from the ground up with synthetic data, ensuring that the correct properties, such as colour and shape, are learned effectively. It deviates from the standard research structure, as it addresses no obvious deficiencies in CNN research. Instead, the work is motivated by testing neural networks on simple datasets to identify shortcomings which can then be resolved. We also look at how the RFL affects results using this simple dataset. Overall, our goal is to decrease network complexity without negatively affecting the results.

In Section <ref>, we discuss the data generation process for all trials. The motivating case for this study is presented in Section <ref>, demonstrating that a CNN can struggle with simple object detection. In Section <ref> we introduce some additional background material that is relevant for quantifying results. In Section <ref>, we introduce K-Origins, a layer designed to help neural networks quantify colours and intensity magnitudes. Section <ref> demonstrates that the motivating case can either be solved by using K-Origins or by increasing the depth and complexity of the network. Finally, in Section <ref>, we test the limits of segmentation across a range of colour distributions for two types of problems: object detection and tracer segmentation.




§ METHODS





 §.§ Synthetic data



Greyscale synthetic data is generated with a 16-bit colour channel for various test cases. This data contains a background with randomly placed squares, and the number of squares varies to ensure that the background remains visible. For each trial, 400 synthetic images with the dimensions 200x200 are created for training, and an additional set is used for testing. Square side lengths and class intensity distributions vary between trials. By using squares, which are simpler shapes and are easier to interpret, we can better assess the impact of K-Origins.

For greyscale data, a pixel's colour is represented by a single integer value. For 16-bit data, as used in this paper, the values range from 0 (pure black) to 65535 (pure white), with various shades of grey in between. In this work, a class's colour is represented by its intensity mean (μ_i) and the standard deviation of added Gaussian noise (σ_i). Figure <ref> shows the integer-intensity mapping and provides examples of the synthetic data used in this paper. Data intensity distributions are illustrated using normalized histograms (data = data/max(data)).







 §.§ Motivating Case: Network Failure



The motivation for K-Origins and this work is shown in Figure <ref>, where a small encoder-decoder network fails to classify noiseless squares from the background. The network lacks an understanding of colour magnitude; if it could recognize the lighter gray squares against the darker gray background, the task would be simple. However, the network does not directly leverage the 16-bit values—the greyness—of the squares in its predictions. For example, a straightforward solution to this problem is to compare a pixel's integer value to 25000 (the squares' colour) and classify it as a square if it matches, or as background if it does not. Despite having over 70,000 trainable parameters, the network fails to learn this behavior. 

Moreover, the network can only correctly classify squares within 4 to 5 pixels from the object border. This suggests the network detects gradients rather than colour magnitudes and does so over a specific length. Convolutions are known for detecting gradient-related behavior so this is almost expected, but it would be highly beneficial if the non-linearity of neural networks could be used to leverage colour magnitudes more directly.



In Figure <ref>, the network's RFL is calculated by setting the bottom-right feature (the deepest point) to r_l=end = 1 pixel and recursively determining the RFL at previous layers. Using Equation <ref>, we calculate that the motivating network has an RFL of 8 pixels, which is twice the distance that gets correctly classified from the object border, plus or minus one pixel. Being twice the correct prediction distance should be expected because the deepest features in that network can "see" about 4-5 pixels on either side of the pixel it wishes to classify. We hypothesize that this network classifies pixels by detecting a square edge in any direction; if no edge is detected within the RF, the pixel is classified as background.

Figure <ref> shows that the network struggles to classify pixels far from the object border and that it also fails to understand intensity magnitudes. We will address both of these issues separately and will use greyscale data because the single channel results extend to additional colour channels (RGB).




 §.§ Metrics



In this section we introduce important equations and data properties that will be used throughout the rest of the paper.

To quantify the distance between intensity distributions, we use the Hellinger distance for two classes represented by Gaussian probability density functions (PDFs). The HD for two Gaussian distributions is given by:


    HD(𝒩(μ_1,σ_1),𝒩(μ_2,σ_2)) = √(1 - √(2σ_1 σ_2/σ_1^2 + σ_2^2)exp( -1/4(μ_1 - μ_2)^2/σ_1^2 + σ_2^2))


where 𝒩 is a normal distribution with means μ_1 or μ_2 and standard deviations σ_1 or σ_2 respectively (<cit.>). This equation produces a value between 0 and 1, where an HD of 0 indicates identical distributions, and an HD of 1 indicates completely distinguishable distributions.

Next, we introduce a modified accuracy metric to address class imbalance. In this paper, the background (class zero) is so large that it inflates the accuracy score. To counter this, we exclude the background class from all accuracy calculations. The resulting modified accuracy is given by:


    MAcc =  1/C-1∑_i≠background^CTP_i/TP_i + FP_i + FN_i


where MAcc is the custom accuracy with background bias removed, C is the total number of classes including the background, TP_i represents true positives, FP_i false positives, and FN_i false negatives for class i. A target class is any class that is not background (i ≠ background). This turns out to be the Jaccard index (<cit.>) for multiple classes and throughout this paper all mentions of accuracy are referring to MAcc.




 §.§ K-Origins Layer: The Colour Solution



We first develop a layer to help networks quantify colour magnitudes. To the best of the authors' knowledge at the time of writing, this approach is unique. Given features X, a K-Origins layer with K trainable weights produces output features for each trainable weight w_k ∈ [w_1,w_2,...,w_K], as follows:


    Y_k=X-J· w_k
 

where Y_k is the output given from a single weight w_k, and J is a matrix of ones matching the dimensions of X. This layer produces K copies of the input image, each with a different scalar subtracted from it, resulting in K images with different origins. All values less than the weight w_k, or origin, become negative in Y_k and all data greater than w_k stays positive. 

For 2D and 3D image data there is normally one origin (zero), making all data positive relative to it. If we immediately use K-Origins on this input data, then future layers such as convolutions can use the sign changes to determine the relative intensity locations for each pixel. Similar behaviour can be done for deeper features in a network. For the first K-Origins layer the weights w_k must match the data's (or features) order of magnitude. For un-signed 16-bit data we see w_k ∈ [0, 65535] for the first layer, requiring learning rates of 1-100 for significant parameter changes during training.

Figure <ref> shows a small network that takes an input image and concatenates it with the output of a K-Origins layer with one weight, w_1. Concatenating the output of K-Origins with the input provides stable reference features for the rest of the network, which is essential for convergence as Y_k constantly changes. The network then applies a softmax-activated 1x1 convolution with a learning rate of 1E-3 for pixel-wise predictions. Because this network has an RFL of one pixel, it can only use information from a single pixel for its predictions, extracting no spatial information.

The weight w_1 was initialized at 50000 with a learning rate of 100 and ended at w_1,final = 20200 after 33 epochs. This final value lies between the intensity values of the two classes, μ_0 =20000 and μ_1 = 25000. This small network with only 5 trainable parameters achieved 100% accuracy segmenting the case from Figure <ref>, whereas the encoder-decoder network with 71,042 trainable parameters achieved only a 67% accuracy. This small network was also tested with more weights on a 7-class case and achieved 100% accuracy. However, accuracy decreased when the class intensity distributions had an HD less than unity, suggesting that a combination of K-Origins and shape recognition would perform better.



Because supervised learning problems have ground truths, K-Origins weights can be initialized based on known class distributions with learning rates of zero, or near zero. For example, in the first problem of Figure <ref>b, initializing the K-Origins weight as w_1=22500, right between both classes, achieves 100% accuracy in just one epoch. This technique is used later in this article by "clamping" distributions, where a weight is placed above and below the known distribution of a target class to clamp those intensities.
 
Convolutional neural network layers are generally defined as:


    Y = f(X∗c + b)


where X∗c is the convolution of features X with kernel c, f(z) is the activation function, and b is the bias. Often for semantic segmentation networks the reluctance (ReLU) activation function is used (f(z)=ReLU(z)), such as in <cit.>. ReLU forces negative numbers to be near zero and positive, making it hard for a network to learn the behavior of K-Origins without directly implementing it. While a convolutional network could theoretically learn similar behavior, it would be challenging. 

Next we look at setting various neural network depths and compare accuracies with and without K-Origins for a range of RFL's. 




 §.§ RFL's: Length Scale Solution



In Figure <ref>, the network fails for larger objects. In this section, we investigate the required RFL for various object sizes. We use a set of small encoder-decoder networks, shown in Figure <ref>, with additional details in Table <ref>. The six architectures used are RFL8, RFL18, RFL38, KRFL8, KRFL18, and KRFL38, where "KRFLX" refers to an identical architecture to "RFLX" with K-Origins. All networks in Table <ref> use "same" padding, where applicable, to prevent cropping. We hypothesize that the RFL should be larger than the dominant length scale, or the minimum length required to differentiate two objects.





We first train the six networks on noiseless data (μ_0 = 20000, μ_1 = 25000, σ_0=σ_1 = 0) containing squares with a side length of 25 pixels, similar to the scenarios in Figures <ref> and <ref>. We also train on noisy data (μ_0 = 20000, μ_1 = 25000, σ_0=σ_1 = 2000) to simulate the failure case in Figure <ref>. This shows us the effect of increasing the RFL for a fixed object size. 

Training runs for 10 epochs with a batch size of 3. Learning rates are set to 1E-3 for convolution layers and 100 for K-Origins layers. The highest-level K-Origins weights are initialized by placing a weight two standard deviations above and below the intensity mean for each class (w_i1,i2 = μ_i ± 2σ_i). This effectively clamps each class's intensity distribution with two K-Origins parameters. For the noiseless case this corresponds to w_i={20000,20000,25000,25000}, and for the noisy case, w_i={16000,24000,21000,29000}. All other K-Origins layers have three weights initialized from Gaussian random variables with μ = 20000 and σ=5000. 

The results from these trials are shown in Figure <ref>. Networks without K-Origins increase in accuracy as the RFL approaches the object length, achieving high accuracies when the RFL exceeds the object length.  In contrast, networks with K-Origins achieve near-perfect validation accuracy regardless of their RFL, demonstrating a more efficient solution. Achieving a near-perfect accuracy without K-Origins requires about 1.4 million trainable parameters, while using K-Origins achieves the same accuracy with only 187,000 trainable parameters. Additionally, an even smaller network with K-Origins could be possible, as this test did not determine a network size lower bound.



Next, we perform a sweep of square side lengths to RFL ratios, L/RFL, for the six networks using the same training parameters as before. This is done with and without noise. For each RFL, we examine L/RFL≈{0.3,0.6,0.95,1.3,2,3}. These fractions are approximated since side lengths may be rounded. The summary of these tests is shown in Figure <ref> and in almost every case, using K-Origins increases accuracy. We also observe that accuracy decreases when L/RFL is small. This is because a small L/RFL results in very small squares, making segmentation difficult in noisy conditions regardless of the architecture used. All numerical results are found in Appendix <ref>.



In almost every case, networks with K-Origins outperform those without it. For this problem, KRFL8, KRFL18, and KRFL38 achieved nearly 100% accuracy in about 3 epochs, compared to the 10 epochs for their RFLX counterparts. While it might be argued that this is due to the high learning rate of K-Origin layers, training with a learning rate of zero for K-Origins produces similar results with the same initialization. The use of K-Origins may enable smaller and more efficient networks without sacrificing performance.

Networks without K-Origins (RFLX) succeed when the RFL is larger than the object size, which aligns with the preference for very deep networks in most research. These networks also seem to perform better on noisy data than on noiseless data. 

So far we have demonstrated a solution to the motivational problem (Figure <ref>) using both network length scales (ensuring sufficient RFL) and intensity quantification (K-Origins) for a noisy and noiseless case. In the noiseless case we set Δμ = 5000 and σ = 0, giving a unity HD. In the noisy case we set Δμ is the same, but σ = 2000 resulting in an HD of 0.73. The next logical step is to sweep across various HDs by adjusting Δμ and Δσ to determine the effectiveness of K-Origins for different intensity distributions.




 §.§ Mixed Solution for a Range of Intensity Distributions



In this section we explore how changing the HD affects segmentation by varying Δμ and Δσ. Setting the first class to μ=20000 and σ = 1000, we produce the HD heatmap shown in Figure <ref>. This heatmap will be used to determine HDs for the upcoming trials, allowing us to compare the accuracy of each trial with the HD.



This comparison is done for two network architectures: RFL14 and KRFL14, shown in Figure <ref> with additional parameters listed in Table <ref>. The key difference is the inclusion of K-Origins in KRFL14. 

We consider two scenarios: a single target class on a noisy background and two target classes on a noisy background. These problems have two and three output classes, respectively, with the background considered a class. Noise parameter sweeps are performed for two object sizes: squares with side lengths randomized between 6 to 12 pixels (L<RLF) and 20 to 30 pixels (L>RFL). This results in a total of four cases: object detection with L<RFL and L>RFL, and the tracer problem with L<RLF and L>RFL.

Networks are trained for 10 epochs with a batch size of 3. Learning rates are set to 1E-4 for convolution layers and 100 for K-Origins layers. K-Origins initialization follows the same method as in Section <ref> and this time KRFL14 has fewer parameters than RFL14. There are 50 randomly placed squares for L<RFL and 25 for L>RFL. All numerical results are found in Appendix <ref>.






  §.§.§ One Target Class: Object Detection



In this section, we segment a single target class (squares) from a background with an intensity mean of μ_0 = 20000 and noise with a standard deviation of σ_0 = 1000. We vary the target class mean and standard deviation, where Δμ = μ_1-μ_0 and Δσ = σ_1 - σ_0. For each mean and standard deviation, we train both RFL14 and KRFL14 and save the validation accuracies in a heatmap. The x-axis represents the change in mean (Δμ), and the y-axis represents the change in standard deviation (Δσ).

Figure <ref> shows results for L < RLF, and Figure <ref> shows results for L > RFL. In both figures, part (a) presents the heatmap with training results for each network. These accuracies can be compared to the HD found in Figure <ref>. Part (b) shows a simple example with an HD of 0.694, and two extreme cases with HDs of 0.176. The first example case has a different distribution than those discussed earlier in this work.





The validation accuracy heatmaps show that the network with K-Origins consistently outperforms the one without it. The network without K-Origins struggles most when Δσ = 0, indicating a pure mean shift. The results also suggest that having L > RLF is beneficial, but this is specific to the dataset used in this paper. Just as small, noisy squares are hard to detect, larger noisy squares become easier to detect with such controlled data. 

KRFL14 also makes relatively good predictions when the HD is 0.176 which is an extremely challenging segmentation task for both machines and humans. This demonstrates the effectiveness of intensity quantification for tasks such as object detection.

After adding K-Origins, the accuracy heatmap is almost directly correlated to the class HDs. This is evident by comparing Figure <ref> to the KRFL14 accuracy plots in Figure <ref> and Figure <ref>. As the HD decreases, so does the accuracy, and vice versa. This correlation is not observed in the traditional network without K-Origins.




  §.§.§ Two Target Classes: Tracer Segmentation



In this section, we segment two identically shaped target classes (both squares) from a background and differentiate these classes from each other, addressing the "Tracer Problem." This involves varying the intensity distributions of the two target classes to make them more or less similar.

The background has μ_0 = 16500 and noise with a standard deviation σ_0 = 900 to minimize interference with the target classes. The first target class has μ_1 = 20000 and σ_1 = 1000 , while the second target class varies based on Δμ = μ_2-μ_1 and Δσ = σ_2 - σ_1. Results for L<RFL are shown in Figure <ref>, and results for L>RFL are shown in Figure  <ref>. In part (a), we present the heatmaps showing validation accuracy results for RFL14 and KRFL14, which can be compared to a trials HD using Figure <ref>. Part (b) provides a straightforward example followed by the two most challenging cases tested.





KRFL14 consistently outperforms RFL14 in this task, especially when the standard deviation remains constant while Δμ varies. Accuracy increases with L>RFL are for the same reason as mentioned before. This shows extremely promising results, with useful segmentation even at an HD of 0.176. As in Section <ref>, the accuracy plots for the network using K-Origins correlates well with the HD plot.




§ DISCUSSION



The most significant improvements from K-Origins occur when the primary difference between class intensity distributions is a mean shift. In reality, this scenario is common because classes often differ by colour or intensity means.

The RFL-related experiments suggest that a network's depth should be set such that the RFL is larger than all target object sizes (L<RFL). Soon after this point, the primary author hypothesizes that making the architecture deeper is less beneficial and less efficient than making it wider. K-Origins is an example of making it wider, as is increasing the number of filters in layers. This could possibly be presented as a guideline for how deep a neural network should be, and could make the design process more deterministic. There is, however, the possibility that these results stem from using such controlled data. This could be studied in the future. 

A potential use case for data without such well-defined intensity distributions, such as general image data, is to use N equally spaced weights along the entire intensity or colour spectrum in each channel, dividing it into N+1 different regions for the network to leverage. This approach would make it easier, for example, to determine if a picture of a dog has white snow or green grass in the background. This would likely also help determine the exact colour of the dog. For this reason, K-Origins is likely useful for other classification problems, not just semantic segmentation. The downside, however, is that as N increases, the memory requirements grow significantly due to the number of image copies being created. There are likely ways to make this more efficient.

Additionally, it is unclear if K-Origin layers after the first impact classification results significantly. These weights likely need to be much smaller than those used in this paper and this could be explored in future studies. There is also the possibility to extend the application of K-Origins to un-supervised problems, perhaps by using a modified version of the simple colour network in Figure <ref>.

The experiments in this paper did not involve any hyperparameter tuning, which would likely improve results significantly, nor were the networks necessarily trained to steady state. Our goal was to demonstrate that even with minimal tuning, the ability to understand colour magnitudes is beneficial for predictions. 




§ CONCLUSION



The experimental results from this study suggest that encoder-decoder networks struggle with classifications that require an understanding of colour or intensity magnitudes, as opposed to gradients alone. The custom layer K-Origins, which can be added to any network, was tested by incorporating it in modified U-Net architectures. By adding K-Origins and ensuring a sufficient RFL, there were significant accuracy improvements for the object detection and tracer segmentation problems. This approach allows for the development of smaller and more efficient networks.

These improvements are likely relevant to many fields, as object detection and tracer segmentation problems are common. Additionally, as new network architectures are being studied it would be valuable to test the impact of K-Origins on these emerging architectures, given its compatibility with any network.



  







§ CODE



The Python (TensorFlow) code used in this paper can be found in the GitHub repository associated with the primary author's thesis: <https://github.com/lewismmason/Thesis-Public>.




§ FIGURE 1



The networks used for Figure <ref> are RFL32 and KRFL32, deeper versions of RFL14 and KRFL14. The additional level is added the same way RFL8 is extended to RFL18. This depth satisfies RFL requirements, and still demonstrates that adding K-Origins is beneficial.




§ EXPERIMENTAL DATA




























