This folder contains all the sources files of the "frame_field_learning" Python package.

We briefly introduce each script in the following.


## data_transform.py

Contains functions that return transformations applied to input data before feeding it to the network.
It includes pre-processing (whose result is store on disk), 
CPU transforms which are applied on the loaded pre-processed data before being transferred to the GPU, 
and finally GPU transforms which are applied on the GPU for speed (such as data augmentation).


## evaluate.py

Defines the "evaluate" function called by ```main.py``` when ```--mode=eval``` which setups and instantiates an Evaluator object whose evaluate method is then called.


## evaluator.py

Defines the "Evaluator" class used to run inference on a trained model followed by computing some measures and saving all results for all samples of a evaluation fold of a dataset.


## frame_field_utils.py

This script defines all "frame field"-related functions. 
For example the ```framefield_align_error``` function is used to compute the "align", 
"align90" losses as well as the "frame field align" energy for ASm optimization in our paper.

The ```LaplacianPenalty``` class is used for the "smooth" loss to ensure a smooth frame field.

Both ```compute_closest_in_uv``` and ```detect_corners``` are use to detect corners sing the frame field.
They are somewhat redundant but the first is applied on torch tensors while the second is applied on numpy arrays.


## ictnet.py

This script implements ICTNet (https://theictlab.org/lp/2019ICTNet) in PyTorch to try and add frame field learning to ICTNet.
We can use it as a backbone model with the ```ICTNetBackbone``` class.

## inference.py

This script defines the ```inference``` function to run a trained model on one image tile. It is used by the evaluation code.

If the "patch_size" parameter is set in the "eval_params", then the image tile is split in small patches of size "patch_size", 
inference is run on bacth_size patches at a time, 
finally the result tile is stoched together from the results of all patches.
Typically those patches overlap by a few pixels and the result is linearly interpolated in overlapping areas.

If there is no "patch_size" parameter, inference is run directly on the whole image tile.


## inference_from_filepath.py

This script defines the ```inference_from_filepath``` function which is called by ```main.py``` when the ```--in_filepath``` argument is set.
It runs inference + polygonization on the images specified by the ```--in_filepath``` argument and saves the result.


## local_utils.py

This script holds several utility functions for the project that do not really belong elsewhere.


## losses.py

The script defines a ```Loss``` class that is the base class to be used for all loss functions. 
It makes it easy to compute normalized losses for easier balancing. 
It thus also defines a ```MultiLoss``` class to combine all necessary losses and apply multiplicative coefficients to their normalized version.

All individual losses (segmentation loss, frame field align loss, frame field smooth loss, etc.) are defined as classes inheriting from the ```Loss``` class.

Lastly the ```build_combined_loss``` function instantiates a ```MultiLoss``` object with all required losses depending on the specified config file.


## measures.py

This script defines the ```iou``` function to compute the Intersection over Union (Iou), also calles the Jaccard index, and 
the ```dice``` function to compute the differentiable version of the IoU in the form of the Dice coefficient.


## model.py

This script defines the ```FrameFieldModel``` class which implements the final network. 
It needs a backbone to be specified model and adds the necessary convolutions for the segmentation and frame field outputs.
Its forward method also performs transforms to be done on the GPU before pushing the result into the backbone. 


## plot_utils.py

This script defines several functions to plot segmentations, frame fields, polygons and polylines for visualization.


## polygonize.py

This script is the entrypoint for all polygonization algorithms. It can then call these polygonization methods:
- Active Contours Model (ACM) from polygonize_acm.py
- Active Skeleton Model (ASM) from polygonize_asm.py
- Simple polygonization (Marching Cubes contour detection + Ramer-Douglas-Peucker simplification) from polygonize_simple.py


## polygonize_acm.py

This script implements the Active Contours Model (ACM) polygonization algorithm. 
It starts with contours detected with Marching Squares which are then optimized on the GPU with PyTorch to align to the frame field (in addition to other objectives, see paper).
It does not handle common walls between adjoining buildings. 
For that reason we have also developed the Active Skeleton Model (ASM), see next section.


## polygonize_asm.py

This script implements the Active Skeleton Model (ASM) polygonization algorithm. 
It starts with a skeleton graph detected from the wall segmentation map. 
It is then optimized on the GPU with PyTorch to align to the frame field (in addition to other objectives, see paper).
It can also be initialized with Marching Squares and essentially implements the ACM as well. 
Thus, the polygonize_acm.py script should be redundant 
(we still keep it around as it follows a different approach to the data structure which can be interesting).


## polygonize_simple.py

This script implements the simple polygonization (Marching Cubes contour detection + Ramer-Douglas-Peucker simplification) used as baseline in the paper.


## polygonize_utils.py

This scripts implements a few utility functions used by several of the above polygonization methods. 
For example, the marching squares contour detection and computation of polygon probability computed from the building interior probability map.


## save_utils.py

This script implements functions for saving results produced by the network and polygonization in various formats.


## train.py

This script defines the ```train``` function which sets up the traiing procedure. 
It instantiates a ```Trainer``` object which will then run the optimization.

## trainer.py

This script implements the ```Trainer``` class which is responsible for training the given model.
It implements multi-GPU training, loss normalization, restarting traiing from a checkpoint, etc.


## tta_utils.py

When performing inference with the model, 
Test Time Augmentation (TTA) can be used to increase the quality of the result.
We augment the input 8 times with right-angled rotations and flips. 
The outputs are then merged with a given aggregation function. 
We implemented several but it seems averaging works best.
The aggregation function is used on the 8 segmentation maps.
However, for the frame field, neither the uv nor the c0, c2 representations can be averaged easily (because of non-linearity).
Thus, we first search for the one segmentation out of the 8 ones that agress the most with the aggregated one.
Then the corresponding frame field is selected.
TTA is performed in the ```forward``` method of ```FrameFieldModel```.


## unet.py

This script implements the ```UNetBackbone``` class for the original U-Net network to be used as a backbone.


## unet_resnet.py

This script is an adapted code from https://github.com/neptune-ai/open-solution-mapping-challenge/blob/master/src/unet_models.py
to use the ResNet network as an encoder for a U-Net. This is used to instantiate the Unet-Resnet101 backbone we use in the paper.


### :warning: TODO: complete this README