When combining spectral and LiDAR data, the parametric models have been quite rarely used for predicting forest attributes. In contrast, relatively more studies were carried out using combined data made use of nonparametric methods (especially MSN and RF), probably as the models are generally assumed as rather 'distribution-free methods' which can potentially be applied regardless of the underlying distribution of the population. A further reason could be the ability of more advanced methods such as MSN and RF to handle high-dimensional feature spaces. However, examples of the joint use of spectral and laser scanner data for parametric modelling can be e.g. (Fransson et al., 2004) and (Hudak et al., 2006), in both of which the magnitude of candidate predictors were notably less than those making use of nonparametric methods. (Fransson et al., 2004) built regression models to predict stem volume using SPOT5 data aided by TopEye laser scanner data in Swedish coniferous landscapes. The SPOT5 data was used to develop features including multi-spectral bands, ditto squared, and the band ratios. LiDAR- derived features included height and forest density measures at stand level. The single as well as combined datasets were tested, from which the combined use of laser height data with the spectral features surpassed the individual use of the datasets. Later on,(Hudak et al., 2006) linearly regressed basal area and tree density on 26 predictors derived from height/intensity of LiDAR and Advanced Land Imager (ALI) multispectral data. They found laser height (to a higher extent) added by laser intensity metrics as most relevant predictors of both responses (The LiDAR-dominated models explained around 90 %

In terms of applying conventional distance-based k-NN methods, (McInerney et al., 2010) can be referred who combined airborne laser scanner and spaceborne Indian Remote Sensing (IRS) multispectral data to model stand canopy height using k-NN method. They apparently reported laser height data as the major means of canopy height retrieval, and achieved a relative *RMSE* between 28 and 31 %. (Maltamo, Malinen, Packalén, Suvanto & Kangas, 2006) applied a k-MSN (MSN using multiple *k*) method to combine the LiDAR data with aerial images and terrestrial stand information in Finland. The laser-based models were reported to outperform aerial photography in stand volume estimation, and the combination improved the models at plot and stand levels. (Wallerman & Holmgren, 2007) have also highlighted the combined application of predictive features derived from optical (SPOT) and laser (TopEye) data, according to which the combined dataset yielded the mean standing volume and stem density models with *RMSE* = 20% and *RMSE* = 22%, respectively. Combining satellite-based (TM) spectral features with laser metrics was also carried out by (Latifi et al., 2010) who reported that TM-extracted metrics can be used as alternatives to those derived from aerial photography for area-based models. Using k-MSN approach, (Packalén & Maltamo, 2006) conducted a survey to achieve species-specific stand information using sets

et al., 2008), (Straub et al., 2009)).

of variance for both response variables).

of aerial photography and ALS data. The procedure consisted of two methods including 1) simultaneous k-MSN estimation and 2) a two- phase prediction (prediction of the responses using regression analysis of ALS data and then allocation of the variables using a fuzzy classification approach). The k-MSN achieved better results than the fuzzy classifications. Although the study still proposed some further developments of the predictor variables from both datasets, the results were assessed satisfactory in cases of Norway spruce (*Picea abies* L.) and Scots pine (*Pinus sylvestris* L.). Soon after, (Packalén & Maltamo, 2007) made stand level models of volume and height using the similar dataset as before. A set of Haralick textural features(Haralick, 1979) from the optical data were additionally combined with the calculated ALS height features to produce predictive models. Accuracy of the predicted responses was finally found to be comparable to stand-level field assessments, though the attributes of conifers were estimated more accurately than those from the deciduous stands. In a further study by those authors, (Packalén & Maltamo, 2008) made use of the similar data to develop k-MSN models of diameter distribution by tree species. Based on the results of growing stock estimation in the previous research work(s), two approaches were compared including 1) field-based modelling using the Weibull distribution and 2) k-MSN prediction, in which the latter was assessed to outperform the former method. Nevertheless, the need to have more comprehensive reference field data (i.e. a common small-scale problem) to cover the spectral variations of the remote sensing data was highlighted as a major concern which supports those already acknowledged by precedent studies. (Nothdurft et al., 2009) represents an attempt towards solving this, in which bootstrap-simulated prediction errors of MSN inferences of volume based on sole use of LiDAR height metrics were smaller than those of design-based sampling.

Few studies e.g. (Straub et al., 2010) and (Latifi et al., 2011) compared parametric and nonparametric methods for forest attribute estimation in presence of both LiDAR and multispectral datasets. Whereas the former study compared Ordinary Least Squares (OLS) regression and a yield table-estimated stem volume with that from Euclidean distance-based k-NN method, the latter made a comparison between RF and OLS outputs. Nevertheless, both studies made relatively similar conclusions, in that they stated that using nonparametric methods cannot b expected to remarkably contribute to the improvement of forest attribute estimates. Besides, it supports (Yu et al., 2011)who also tested pure LiDAR metrics and achieved a similar performance of RF and OLS in a single tree scale. The rationale behind this is that non-parametric imputations do not share the same mix of error components as regression predictions. Imputation errors are often greater than regression errors because the errors do not result from a least-squares minimisation, but from selection of a most similar element in a pool of neighbouring observations (Stage & Crookston, 2007). However, K-NN methods (especially in single- neighbour setting) yield predictions with similar variance structure to that of the observations (Moeur & Stage, 1995), and are thus advantageous over the higher accuracies achievable by the use of OLS (Hudak et al., 2008).

The selection of proper predictor variables for a k-NN model (i.e. an absent element of conventional k-NN approaches) is a time-consuming task which and needs to be automated. (Packalén & Maltamo, 2007) used an iterative cost- minimizing variable selection algorithm which aimed at minimizing the weighted average of the relative *RMSE*. In contrast, studies like (Hudak et al., 2008) and (Straub et al., 2009)applied stepwise selection methods, where the former study based its stepwise iteration on the *Gini* index of variable importance used by (Breiman, 2001) as a built-in feature in RF. As such, other variable screening methods such as

regional-scale spatial resolution such as Landsat images, has been constantly approved to bear practical values when combined with laser scanner data ((Fransson et al., 2004), (McInerney et al., 2010)) and even as an alternative to aerial photography for area-based applications (Latifi et al., 2010). Furthermore, image spectroscopy data showed positive potentials for forest modelling ((Foster et al., 2002), (Schlerf et al., 2005)) and could potentially complement LiDAR-based models. However, one should bear in mind that the experimental results of surveys is by no means an eventual justification for the small- scale end users to take the

Characterizing Forest Structure by Means of Remote Sensing: A Review 21

In terms of various modelling methods used, both parametric and nonparametric modelling categories were frequently employed to describe the forest structural attributes. However, the latter approaches received more attention during the recent years to be run for high dimensional predictor datasets as well as for simultaneous predictions. The k-NN methods (especially MSN and RF) have been successfully coupled with LiDAR information and thus caused a rapid increase in the number of research projects during recent years. As it was shown here, much work has been done on area-based methods e.g. stand and plot levels, whereas single-tree approaches still lack some research, mainly due to high computational

In terms of handling predictor feature space induced by remote sensing features, some examples were previously referred. Whereas studies such as (Breidenbach, Nothdurft & Kändler, 2010)made the general necessity of variable screening in k-NN context questionable, some other studies acknowledge the requirement to selecting an effective strategy of pruning of predictor dataset (e.g. (Hudak et al., 2008), (Latifi et al., 2010)) and showed some decisive influences on the outcomes of the forest attribute models. The proper pruning of predictor feature space has been proved to help producing robust models (Latifi & Koch, 2011). Reducing the sensitivity of models has been also shown to greatly contribute to increasing the robustness of the models. Using resampling methods e.g. bootstrapping to reproduce the underlying population (e.g. (Nothdurft et al., 2009),(Breidenbach, Nothdurft & Kändler, 2010), and (Latifi et al., 2011) increases the potential and robustness of applying nonparametric models in small-scale forest inventory, where the shortage of reference data for validating the models is a major constraint. Robust models enable the analyst to apply them under other natural growing conditions except of the underlying test site, and can thus open up new

Along with the rapid advancements in launching the active/passive remote sensing instruments, the general access to high resolution products (particularly to laser scanner data) at reasonable costs is increasing. Therefore, the efforts towards thorough description of tree and forest stand structure are currently following a boosting trend all over the world. However, it is necessary to emphasize, again, that much care should be taken in terms of producing valid and robust results, as well as to get the best out of the available data and modelling facilities. Whereas the rapid and accurate modelling of standing volume, biomass and tree density is still important, some remaining open areas of research still require further research. These include, for example, efforts towards advanced classification tasks (especially on single-tree level or in complicated mixed stands), modelling understory and regenerations (e.g. important for intermediate silvicultural practices), and modelling rare and

acquisition of (relatively) expensive airborne hyperspectral data for granted.

requirements and the need for high resolution data.

operational applications for the yielded models (e.g. (Koch, 2010)).

ecologically-valuable populations.

parametric univariate correlation analysis (Breidenbach, Næsset, Lien, Gobakken & Solberg, 2010), Built-in schemes of RF such as stepwise iterative method (Vauhkonen et al., 2010) and forward selection (Breidenbach, Nothdurft & Kändler, 2010)were also used to complete this task in the recent literature. Each of those screening methods has been reported to be satisfying in terms of reducing the dimensionality of the feature space, though no rationale (e.g. comparison to other methods) has been presented. (Latifi et al., 2010) used a GA on categorised response variables to optimise the high-dimensional feature space formed by numerous correlated predictors. Even though this GA prototype was evaluated to efficiently reduce the relative RMSE of standing volume and AGB compared to the stepwise selection of predictors, the method was reported to produce unstable subsets attributed to strong correlations amongst the predictors. By using a Tau-squared index on continuous responses, GA was later shown to yield stable parsimonious variable subsets (Latifi et al., 2011). GA is a search algorithm which works via numerous solutions and generations and thus explores the entire possible combinations of candidate predictor variables. It provides the consequent NN models with the optimum range of refined, pre-processed feature space formed of relevant (and uncorrelated) remote sensing descriptors and is shown to be able to be adjusted to the k-NN modelling approaches (e.g. (Tomppo & Halme, 2004)). In this context, fitness functions to optimise continuous responses are preferable for regression scenarios. Those functions can even be linear as long as no highly non-linear trend/prediction is observed in the entire underlying dataset.

In a review by (Koch, 2010), the importance of combined use of laser and optical data for such purposes was highlighted. She stated that combining the altimetric height information with physical values derived from laser intensity is appropriate for modelling forest structure. As 3D data has already been shown to be plausible for AGB modelling, and due to the expected future technical innovations of those data for biomass assessments, it is assumed that it will further play a prominent role in major forest monitoring tasks e.g. those related to AGB modelling.

## **3. Conclusion**

Amongst the available active/passive remote sensing instruments, information derived from laser scanner (especially the height information) is definitely of major importance for studies regarding forest structure. According to (Koch, 2010), the significance of using LiDAR data for biomass assessment has been confirmed by variety of investigations which repeatedly showed comparatively higher performance of those data. However, the use of LiDAR intensity data is still limited. The intensity data has been shown to be able to add useful complementary information to LiDAR height data for forest attribute modelling (e.g. (Hudak et al., 2006)). Yet, a direct physical connection between those intensity metrics and forest structure still cannot be drawn. The reason for this complication is stated to be the dependency of intensity on a range of factors affecting reflected laser data including range, incidence angle, bidirectional reflectance function effects, and transmission of atmosphere (Hyyppä et al., 2008).

Apart from few exceptional studies which reported the incapability of spectral data for explaining the variation beyond the variation that could be explained by laser metrics (Hudak et al., 2008), adding spectral information to pure LiDAR-based models has been confirmed to be useful, as they provide continuous information over long time series and are spectrally sensitive for differentiating tree species. The ability of multispectral data, even in 18 Will-be-set-by-IN-TECH

parametric univariate correlation analysis (Breidenbach, Næsset, Lien, Gobakken & Solberg, 2010), Built-in schemes of RF such as stepwise iterative method (Vauhkonen et al., 2010) and forward selection (Breidenbach, Nothdurft & Kändler, 2010)were also used to complete this task in the recent literature. Each of those screening methods has been reported to be satisfying in terms of reducing the dimensionality of the feature space, though no rationale (e.g. comparison to other methods) has been presented. (Latifi et al., 2010) used a GA on categorised response variables to optimise the high-dimensional feature space formed by numerous correlated predictors. Even though this GA prototype was evaluated to efficiently reduce the relative RMSE of standing volume and AGB compared to the stepwise selection of predictors, the method was reported to produce unstable subsets attributed to strong correlations amongst the predictors. By using a Tau-squared index on continuous responses, GA was later shown to yield stable parsimonious variable subsets (Latifi et al., 2011). GA is a search algorithm which works via numerous solutions and generations and thus explores the entire possible combinations of candidate predictor variables. It provides the consequent NN models with the optimum range of refined, pre-processed feature space formed of relevant (and uncorrelated) remote sensing descriptors and is shown to be able to be adjusted to the k-NN modelling approaches (e.g. (Tomppo & Halme, 2004)). In this context, fitness functions to optimise continuous responses are preferable for regression scenarios. Those functions can even be linear as long as no highly non-linear trend/prediction is observed in the entire

In a review by (Koch, 2010), the importance of combined use of laser and optical data for such purposes was highlighted. She stated that combining the altimetric height information with physical values derived from laser intensity is appropriate for modelling forest structure. As 3D data has already been shown to be plausible for AGB modelling, and due to the expected future technical innovations of those data for biomass assessments, it is assumed that it will further play a prominent role in major forest monitoring tasks e.g. those related to AGB

Amongst the available active/passive remote sensing instruments, information derived from laser scanner (especially the height information) is definitely of major importance for studies regarding forest structure. According to (Koch, 2010), the significance of using LiDAR data for biomass assessment has been confirmed by variety of investigations which repeatedly showed comparatively higher performance of those data. However, the use of LiDAR intensity data is still limited. The intensity data has been shown to be able to add useful complementary information to LiDAR height data for forest attribute modelling (e.g. (Hudak et al., 2006)). Yet, a direct physical connection between those intensity metrics and forest structure still cannot be drawn. The reason for this complication is stated to be the dependency of intensity on a range of factors affecting reflected laser data including range, incidence angle, bidirectional

reflectance function effects, and transmission of atmosphere (Hyyppä et al., 2008).

Apart from few exceptional studies which reported the incapability of spectral data for explaining the variation beyond the variation that could be explained by laser metrics (Hudak et al., 2008), adding spectral information to pure LiDAR-based models has been confirmed to be useful, as they provide continuous information over long time series and are spectrally sensitive for differentiating tree species. The ability of multispectral data, even in

underlying dataset.

modelling.

**3. Conclusion**

regional-scale spatial resolution such as Landsat images, has been constantly approved to bear practical values when combined with laser scanner data ((Fransson et al., 2004), (McInerney et al., 2010)) and even as an alternative to aerial photography for area-based applications (Latifi et al., 2010). Furthermore, image spectroscopy data showed positive potentials for forest modelling ((Foster et al., 2002), (Schlerf et al., 2005)) and could potentially complement LiDAR-based models. However, one should bear in mind that the experimental results of surveys is by no means an eventual justification for the small- scale end users to take the acquisition of (relatively) expensive airborne hyperspectral data for granted.

In terms of various modelling methods used, both parametric and nonparametric modelling categories were frequently employed to describe the forest structural attributes. However, the latter approaches received more attention during the recent years to be run for high dimensional predictor datasets as well as for simultaneous predictions. The k-NN methods (especially MSN and RF) have been successfully coupled with LiDAR information and thus caused a rapid increase in the number of research projects during recent years. As it was shown here, much work has been done on area-based methods e.g. stand and plot levels, whereas single-tree approaches still lack some research, mainly due to high computational requirements and the need for high resolution data.

In terms of handling predictor feature space induced by remote sensing features, some examples were previously referred. Whereas studies such as (Breidenbach, Nothdurft & Kändler, 2010)made the general necessity of variable screening in k-NN context questionable, some other studies acknowledge the requirement to selecting an effective strategy of pruning of predictor dataset (e.g. (Hudak et al., 2008), (Latifi et al., 2010)) and showed some decisive influences on the outcomes of the forest attribute models. The proper pruning of predictor feature space has been proved to help producing robust models (Latifi & Koch, 2011). Reducing the sensitivity of models has been also shown to greatly contribute to increasing the robustness of the models. Using resampling methods e.g. bootstrapping to reproduce the underlying population (e.g. (Nothdurft et al., 2009),(Breidenbach, Nothdurft & Kändler, 2010), and (Latifi et al., 2011) increases the potential and robustness of applying nonparametric models in small-scale forest inventory, where the shortage of reference data for validating the models is a major constraint. Robust models enable the analyst to apply them under other natural growing conditions except of the underlying test site, and can thus open up new operational applications for the yielded models (e.g. (Koch, 2010)).

Along with the rapid advancements in launching the active/passive remote sensing instruments, the general access to high resolution products (particularly to laser scanner data) at reasonable costs is increasing. Therefore, the efforts towards thorough description of tree and forest stand structure are currently following a boosting trend all over the world. However, it is necessary to emphasize, again, that much care should be taken in terms of producing valid and robust results, as well as to get the best out of the available data and modelling facilities. Whereas the rapid and accurate modelling of standing volume, biomass and tree density is still important, some remaining open areas of research still require further research. These include, for example, efforts towards advanced classification tasks (especially on single-tree level or in complicated mixed stands), modelling understory and regenerations (e.g. important for intermediate silvicultural practices), and modelling rare and ecologically-valuable populations.

Foster, J., Kingdon, C. & Townsend, P. (2002). Predicting tropical forest carbon from

Characterizing Forest Structure by Means of Remote Sensing: A Review 23

Franco-Lopez, H., Ek, A. R. & Bauer, M. E. (2001). Estimation and mapping of forest stand

Franklin, J. (1986). Thematic mapper analysis of coniferous forest structure and composition,

Fransson, J., Gustavsson, A., Ulander, L. & Walter, F. t. (2000). Towards an operational use of

Fransson, J., Magnusson, M. & Holmgren, J. (2004). Estimation of forest stem volume using

Gebreslasie, M. T., Ahmed, F, B. & Van Aardt, J. (2010). Predicting forest structural attributes

Ghosh, M. & Rao, J, N. K. (1994). Small area estimation: An appraisal, *Statistical Science*

Gonzalez-Alonso, F., Marino-De-Miguel, S., Roldan-Zamarron, A., Garcia-Gigorro, S. &

Guo, X. J. A. (2005). *climate- sensitive analysis of lodgepole pine site index in alberta*, Master's

Haapanen, R., Ek, A. R., Bauer, M. E. & Finley, A. O. (2004). Delineation of forest/nonforest

Haralick, R. M. (1979). Statistical and structural approaches to texture. proceedings,

Heinzel, J., Weinacker, H. & Koch, B. (2008). Full automatic detection of tree species based

Heurich, M. & Thoma, F. (2008). Estimation of forestry stand parameters using laser scanning

Hölmstrom, H. (2002). Estimation of single tree characteristics using the knn method

Holmström, H. & Fransson, E. S. (2003). Combining remotely sensed optical and radar data

Härdle, W. (1990). *Econometric society monographs*, Econometric society monographs, Cambridge University Press, chapter Applied nonparametric regression. Härdle, W., Müller, M., Sperlich, S. & Werwatz, A. (2004). *Non-parametric and semiparametric*

in knn estimation of forest variables, *Forest Science* 49(3): 409–418.

norway spruce (picea abies) forests, *Forestry* 81(5): 645–661.

pp. 3108–3110.

pp. 2318–2322.

9(1): 55–76.

89: 265–271.

167: 303–314.

*models*, Springer, New York.

*Sensing of Environment* 77: 251?274.

*International Journal of Remote Sensing* 7: 1287 – 1301.

*IGARSS 2000*, IEEE, Piscataway, NJ., p. 399U401. ˝

*Observation and Geoinformation* 125: 523–526.

*Proceedings of the IEEE*, Vol. 67(5), pp. 786–804.

*of Remote Sensing* 27(24): 5409–5415.

eo-1 hyperspectral imagery in noel kempff mercado national park, bolivia, *. IEEE International Geoscience and Remote Sensing Symposium, 2002. IGARSS '02. Vol. 6,*,

density, volume, and cover type using the k-nearest neighbours method, *Remote*

vhf sar data for forest mapping and forest management, *in* T. Stein (ed.), *Proceedigs of*

optical spot-5 satellite and laser data in combination, *Proceedings of IGARSS 2004*,

using ancillary data and aster satellite data, *International Journal of Applied Earth*

Cuevas, J. M. (2006). Forest biomass estimation through ndvi composites. the role of remotely sensed data to assess spanish forests as carbon sinks, *International Journal*

thesis, Dept. of Mathematics and Statistics. Concordia University, Montreal-Canada.

land use classes using nearest neighbour methods, *Remote Sensing of Environment*

on delineated single tree crowns - a data fusion approach for airborne laser scanning data and aerial photographs, *Proceedings of SilviLaser 2008*, Edinburgh, UK, pp. 76–85.

data in temperate, structurally rich natural european beech (fagus sylvatica) and

and plotwise aerial photograph interpretations, *Forest Ecology and Management*

#### **4. References**

20 Will-be-set-by-IN-TECH

Acker, S., Sabin, T., Ganio, L. & McKee, W. (1998). Development of old-growth structure

BMU (2009). National biomass action plan for germany, *Technical report*, Bundesministerium für Umwelt, Naturschutz und Reaktorsicherheit (BMU), 11055 Berlin, Germany. Boyd, D. S. & Hill, R. A. (2007). Validation of airborne lidar intensity values from a forested

*SLaser Scanning 2007 and SilviLaser 2007Š Part 3 / W52, Espoo-Finland ´* . Breidenbach, J., Kublin, E., McGaughey, R., Andersen, H. & Reutebuch, S. (2008).

airborne laser scanner data, *Photogrammetric Journal of Finland* 21(1): 4–15. Breidenbach, J., Nothdurft, A. & Kändler, G. (2010). Comparison of nearest neighbour

Breidenbach, J., Næsset, E., Lien, V., Gobakken, T. & Solberg, S. (2010). Prediction of species

Cabaravdic, A, A. (2007). *Efficient Estimation of Forest Attributes with k NN*, PhD thesis, Faculty

Crookston, N. L., Moeur, M. & Renner, D. (2002). *Users guide to the most similar neighbor*

Dalponte, M., Coops, N. C., Bruzzone, L. & Gianelle, D. (2009). Analysis on the use of multiple

Davey, S. (1984). *Possums and Gliders*, Australian Mammal Society, Sydney, chapter Habitat

Efron, B. & Tibshirani, R. J. (1993). *An introduction to the bootstrap*, New York: Chapman &

Evans, J. S., Hudak, A. T., Faux, R. & Smith, M. (2009). Discrete return lidar in natural

Falkowski, M. J., Hudak, A. J., Crookston, N. L., Gessler, P. E., Uebler, E. H. & Smith, A.

Finley, A., Ek, A. R., Bai, Y. & Bauer, M. E. K. (2003). Nearest neighbour estimation of

Finley, A. O. & McRoberts, R. E. (2008). Efficient k-nearest neighbour searches for multi-source forest attribute mapping, *Remote Sensing of Environment* 112: 2203–2211.

*Topics in Applied Earth Observations and Remote Sensing* 2(4): 310–318.

of Forest and Environmental Studies, University of Freiburg.

and timber volume growth trends in maturing douglas-fir stands, *Forest Ecology and*

landscape using hymap data: preliminary analysis, *Proceedings of the ISPRS Workshop*

Mixed-effects models for estimating stand volume by means of small footprint

approaches for small area estimation of tree species-specific forest inventory attributes in central europe using airborne laser scanner data, *European Journal of*

specific forest inventory attributes using a nonparametric semi-individual tree crown approach based on fused airborne laser scanning and multispectral data, *Remote*

*imputation program version 2.00*, RMRS-GTR-96.Ogden, UT: USDA Forest Service

returns lidar data for the estimation of tree stems volume, *IEEE Journal of Selected*

preferences of arboreal marsupials within a coastal forest in southern New South

resources: Recommendations for project planning, data processing, and deliverables,

M. S. (2010). Landscape-scale parameterization of a tree-level forest growth model: a k-nearest neighbor imputation approach incorporating lidar data, *Canadian Journal of*

forest attributes: Improving mapping efficiency, *Proceedings of the fifth Annual Forest*

**4. References**

*Management* 104: 265– 280.

*Forest Research* 129(5): 833–846.

*Sensing of Environment* 114: 911–924.

Rocky Mountain Research Station.

Wales, pp. 509– 516.

*Remote Sensing* 1: 776–794.

*Forest Research* 40: 184–199.

*Inventory and Analysis Symposium*, pp. 61–68.

Hall.

Breiman, L. (2001). Random forests, *Machine Learning* 45: 5–32.

Latifi, H., Nothdurft, A., Straub, C. & Koch, B. (2011). Modelling stratified forest attributes

Characterizing Forest Structure by Means of Remote Sensing: A Review 25

LeMay, V. & Temesgen, H. (2005). Camparison of nearest neighbour methods for estimating

Liaw, A. & Wiener, M. (2002). Classification and regression by randomforest, *R News* 2: 18–22. Lim, K., Treitz, P., Wulder, M., St-Onge, B. & Flood, M. (2003). Lidar remote sensing of forest

MacLean, G. & Krabill, W. (1986). Gross merchantable timber volume estimation using an

Maltamo, M., Bollandsås, O. M., Vauhkonen, J., Breidenbach, J., Gobakken, T. & E, N. (2010).

Maltamo, M. & Eerikäinen, K. (2001). The most similar neighbour reference in the yield prediction of pinus kesiya stands in zambia, *Silva Fennica* 35(4): 437–451. Maltamo, M., Eerikäinen, K., Packalén, P. & Hyyppä, J. a. (2006). Estimation of stem volume using laser scanning-based canopy height metrics, *Forestry* 79(2): 217–229. Maltamo, M., Hyyppä, J. & Malinen, J. (2006). A comparative study of the use of laser scanner

Maltamo, M., Malinen, J., Packalén, P., Suvanto, A. & Kangas, J. (2006). Non-parametric

Maltamo, M., Næsset, E., Bollandsås, O., Gobakken, T. & Packalén, P. (2009). Non-parametric

McElhinny, C., Gibbons, P., Brack, C. & Bauhus, J. (2005). Forest and woodland stand

McInerney, D. O., Suarez-Minguez, J., Valbuena, R. & Nieuwenhuis, M. (2010). Forest

McRoberts, R. E. & Tomppo, E. O. (2007). Remote sensing support for national forest

Mäkelä, H. & Pekkarinen, A. (2004). Estimation of forest stand volumes by landsat tm imagery and stand-level field-inventory data, *Forest Ecology and Managament* 196: 245–255. Moeur, M. & Stage, A. R. (1995). Most similar neighbour: An improved sampling inference

Mohammadi, J. & Shataee, S. (2010). Possibility investigation of tree diversity mapping using

Muukkonen, P. & Heiskanen, A. J. (2007). Biomass estimation over a large area based on

verify carbon inventories, *Remote Sensing of Environment* 107: 607–624. Nelson, R., Krabill, W. & Tonelli, J. (1988). Estimating forest biomass and volume using airborne laser scanner data, *Remote Sensing of Environment* 24(2): 247–267.

landsat etm+ data in the hyrcanian forests of iran, *Remote Sensing of Environment*

standwise forest inventory data and aster and modis satellite data: A possibility to

procedure for natural resource planning, *Forest Science* 41: 337U359. ˝

stand-register data, *Canadian Journal of Forest Research* 36: 426–436.

knn estimation in aberfoyle, scotland, *Forestry* 83(2): 195–206.

inventories, *Remote Sensing of Environment* 110: 412–419.

airborne lidar system, *Canadian Journal of Remote Sensing* 12: 7U18. ˝

stands using airborne laser scanner data, *Forestry* 83(3): 257–268.

*Digital Earth* DOI:10.1080/17538947.2011.583992.

structure, *Progress in Physical Geography* 27(1): 88–106.

*Scandinavian Journal of Forest Research* 21: 231–238.

*Journal of Forest Research* 24: 541–553.

218: 1–24.

104(7): 1504–1512.

51(2): 109–119.

using optical/lidar features in a central european landscape, *International Journal of*

basal area and stems per hectare using aerial auxiliary variables, *Forest Science*

Comparing different methods for prediction of mean crown height in norway spruce

data and field measurements in the prediction of crown height in boreal forests,

estimation of stem volume using airborne laser scanning, aerial photography, and

prediction of diameter distribution using airborne laser scanner data, *Scandinavian*

structural complexity: Its definition and measurement, *Forest Ecology and Management*

canopy height retrieval using lidar data, medium resolution satellite imagery and

22 Will-be-set-by-IN-TECH

Hudak, A., Crookston, N., Evans, J., Hall, D. & Falkowski, M. (2008). Nearest neighbour

Hudak, A. T., Crookston, N. L., Evans, J. S., Falkowski, M. J., Smith, A. M. S. & Gessler, P.

Hyyppä, J., Hyyppä, H., Leckie, D., Gougon, F., Yu, X. & Maltamo, M. (2008). Review of

data in boreal forests, *International Journal of Remote Sensing* 29(5): 1339–1336. Imhoff, M. (1995). Radar backscatter and biomass saturation: ramifications for global biomass inventory, *IEEE Transactions on Geoscience and Remote Sensing* 33(2): 510–518. Iverson, L. R., Cook, E. A. & Graham, R. L. (1994). Regional forest cover estimation via remote sensing: the calibration center concept, *Landscape Ecology* 9(3): 159–174. Jochem, A., Hollaus, M., Rutzinger, M. & Höfle, B. (2011). Estimation of aboveground biomass

Katila, M., T. E. (2002). Stratification by ancillary data in multisource forest inventories

Kilkki, P. & Päivinen, R. (1987). Reference sample plots to combine field measurements and

Koch, B. (2010). Status and future of laser scanning, synthetic aperture radar and hyperspectral

Koch, B., Straub, C., Dees, M., Wang, Y. & Weinacker, H. (2009). Airborne laser data for

Korhonen, L., Peuhkurinen, J., Malinen, J., Suvanto, A., Malatamo, M., Packalén, P. & Kangas,

Kutzer, C. (2008). *Potential of the kNN Method for Estimation and Monitoring off-Reserve*

Latifi, H. & Koch, B. (2011). Generalized spatial models of forest structure using airborne

Latifi, H., Nothdurft, A. & Koch, B. (2010). Non-parametric prediction and mapping of

˝ *Forestry* 83(4): 395–407.

*Department of Forest Mensuration and Management, University of Helsinki*.

*Remote Sensing of Environment* 112: 2232–2245.

derived from airborne lidar data, *Sensors* 11: 278–295.

Kimmins, J. (1996). *Forest ecology*, Macmillan Inc., New York.

*and Remote Sensing* 65: 581–590.

*of Remote Sensing* 32: 126–138.

32: 1548–1561.

30(4): 935–963.

81(4): 499–510.

Germany.

University of Freiburg.

optical/lidar Uderived predictors,

imputation of species-level, plot-scale forest structure attributes from lidar data,

(2006). Regression modeling and mapping of coniferous forest basal area and tree density from discrete- return lidar and multispectral satellite data, *Canadian Journal*

methods of small-footprint airborne laser scanning for extracting forest inventory

in alpine forests: A semi-empirical approach considering canopy transparency

employing k-nearest neighbour estimation, *Canadian Journal of Forest Research*

satellite data in forest inventory, *Remote Sensing-Aided Forest Inventory. Proceedings of Seminars organised by SNS, 10-12 Dec. 1986, Hyytiälä, Finland. Research Notes No 19.*

remote sensing data for forest biomass assessment, *ISPRS Journal of Photogrammetry*

stand delineation and information extraction, *International Journal of Remote Sensing*

J. (2008). The use of airborne laser scanning to estimate sawlog volumes, *Forestry*

*Forest Resources in Ghana*, PhD thesis, Faculty of Forest and Environmental Studies,

multispectral and laser scanner data, *Proceedings of ISPRS Workshop: High resolution earth imaging for geospatial information,*, Vol. XXXVIII-4/W19. of *International Archives of the Photogrammetry, Remote sensing and Spatial Information Sciences,*, Hannover,

standing timber volume and biomass in a temperate forest: application of multiple

Straub, C., Weinacker, H. & Koch, B. (2010). A comparison of different methods for

Characterizing Forest Structure by Means of Remote Sensing: A Review 27

Thessler, S., Sesnie, S., Bendana, Z., Ruokolainen, K., Tomppo, E. & Finegan, B. (2008). Using

Tomppo, E. (1991). Satellite image-based national forest inventory of finland, *International*

Tomppo, E., Gagliano, C., De Natale, F., Katila, M. & McRoberts, R. E. (2009). Predicting

Tomppo, E. & Halme, M. (2004). Using coarse scale forest variables as ancillary information

Tomppo, E., Korhonen, K. T., Heikkinen, J. & Yli-Kojola, H. (2001). Multi-source inventory

Treuhaft, R. N., Asner, G. P. & Law, B. E. (2003). Structure-based forest biomass from fusion of radar and hyperspectral observations, *Geophysical Research Letters* 30(9): 1472. Tyrrell, L. & Crow, T. (1994). Structural characteristics of old-growth hemlock-hardwood

Uuttera, J., Maltamo, M. & Hotanen, J. (1997). The structure of forest stands in virgin

Van Den Meersschaut, D. & Vandekerkhove, K. (1998). Development of a standscale forest

Vauhkonen, J., Korpela, I., Maltamo, M. & Tokola, T. (2010). Imputation of single-tree

Vohland, M., Stoffels, J., Hau, C. & Schüler, G. (2007). Remote sensing techniques for

Wallerman, J. & Holmgren, J. (2007). Estimating field-plot data of forest stands using airborne laser scanning and spot hrg data, *Remote Sensing of Environment* 110: 501–508. Wehr, A. & Lohr, O. (1999). Airborne laser scanningUan introduction and overview, ˚ *ISPRS*

Wood, S. (2006). *Generalized additive models: an introduction with R*, Chapman & Hall/CRC,

over northern costa rica, *Remote Sensing of Environment* 112: 2485– 2494. Tommola, M., Tynkkynen, M., Lemmetty, J., Herstela, P. & Sikanen, L. (1999). Estimating the

orthophotos, *European Journal of Forest Research* 129: 1069–1080.

*Archives of Photogrammetry and Remote Sensing* 28 (7-1): 419U 424. ˝ Tomppo, E. (1993). Multi-source national forest inventory of finland, *in* J. R. A. Nyysso´

landsat imagery, *Remote Sensing of Environment* 113(3): 500–517.

*Forest Engineering* pp. 75–81.

*Sensing of Environment* 92: 1–20.

forests in relation to age, *Ecology* 75(2): 370–386.

metrics, *Remote Sensing of Environment* 114: 1263–1276.

*Journal of Photogrammetry and Remote Sensing* 54: 68–82.

*Ecology and Management* 96: 125–138.

analysis, *Silva Fennica* 41(3): 441–456.

Idaho, USA, pp. 340–34.

Boca Raton, Florida.

35(3): 309U328. ˝

forest resource estimation using information from airborne laser scanning and cir

k-nn and discriminant analyses to classify rain forest types in a landsat tm image

characteristics of a marked stand using k-nearest- neighbour regression, *Journal of*

S. Poso (ed.), *Proceedings of Ilvessalo symposium on national forest inventories*, p. 53 U 61. ˝

categorical forest variables using an improved k-nearest neighbour estimator and

and weighting of variables in k-nn estimation: a genetic algorithm approach, *Remote*

of the forests of the hebei forestry bureau, heilongjiang, china, *Silva Fennica*

and managed peat-lands: a comparison between finnish and russian keralia, *Forest*

biodiversity index based on the state forest inventory, *in* M. Hansen & T. Burk (eds), *Integrated Tools for Natural Resources Inventories in the 21st Century*, USDA, Boise,

attributes using airborne laser scanning-based height, intensity, and alpha shape

forest parameter assessment: Multispectral classification and linear spectral mixture

lnen,

24 Will-be-set-by-IN-TECH

Nothdurft, A., Soborowski, J. & Breidenbach, J. (2009). Spatial prediction of forest stand

Næsset, E. (2002). Predicting forest stand characteristics with airborne scanning laser

Næsset, E. (2004). Practical large-scale forest stand inventory using a small airborne scanning

Packalén, P. & Maltamo, M. (2006). Predicting the plot volume by tree species using airborne

Packalén, P. & Maltamo, M. (2007). The k-msn method for the prediction of species-specific

Packalén, P. & Maltamo, M. (2008). Estimation of species-specific diameter distributions using

Pesonen, A., Maltamo, M., Packalén, P. & Eerikäinen, K. (2008). Airborne laser scanning-based

Päivinen, R., Van Brusselen, J. & Schuck (2009). A the growing stock of european forests using

Rahman, M., Csaplovics, E. & Koch, B. (2007). An efficient regression strategy for extracting

Schlerf, M., Atzberger, C. & Hill, J. (2005). Remote sensing of forest biophysical variables using hymap imaging spectrometer data, *Remote Sensing of Environment* 95(2): 177–194. Sexton, J. O., Bax, T., Siquiera, P., Swenson, J. J. & Hensley, S. (2009). comparison of lidar,

southeastern north america, *Forest Ecology and Management* 257: 1136U1147.

Stage, A. R. & Crookston, N. L. (2007). Partitioning error components for accuracy-assessment of near- neighbor methods of imputation, *Forest Science* 53(1): 62?72. Stümer, W. . D. (2004). *Kombination vor terrestischen Aufnahmen und Fernerkundungsdaten mit*

Stoffels, J. (2009). *Einsatz einer lokal adaptiven Klassifikationsstrategie zur satellitengestützten*

Stone, J. & Porter, J. (1998). What is forest stand structure and how to measure it?, *Northwest*

Straub, C., Dees, M., Weinacker, H. & Koch, B. (2009). Using airborne laser scanner data

Straub, C. & Koch, B. (2011). Estimating single tree stem volume of pinus sylvestris

Geography/Geesciences, University of Trier.

*Fernerkundung, GeoInformation* 3/2009: 277–287.

remote sensing and forest inventory data, *Forestry* 82(5): 479–490.

using a practical two-stage procedure and field data, *Remote Sensing of Environment*

stand attributes using airborne laser scanning and aerial photographs, *Remote Sensing*

airborne laser scanning and aerial photographs, *Canadian Journal of Forest Research*

prediction of coarse woody debris volumes in a conservation area, *Forest Ecology and*

forest biomass information from satellite sensor data, *International Journal of Remote*

radar, and field measurements of canopy height in pine and hardwood forests of

*Hilfe der kNN-Methode zur Klassifizierung und Kartierung von Wäldern*, PhD thesis, Fakultät für Forst-, Geo- und Hydrowissenschaften der Technischen Universität

*Waldinventur in einem heterogenen Mittelgebirgsraum.*, PhD thesis, Faculty of

and cir orthophotos to estimate the stem volume of forest stands, *Photogrammetrie,*

using airborne laser scanner and multispectral line scanner data, *Remote Sensing*

˝

variables, *European Journal of Forest Research* 128(3): 241–251.

laser, *Scandinavian Journal of Forest Research* 19: 164–179.

Oliver, C. & Larson, B. (1996). *Forest Stand Dynamics*, McGraw-Hill Inc., New York.

laser scanning and aerial photographs, *Forest Science* 52(6): 611–622.

80(1): 88–99.

38: 1750–1760.

Dresden.

*Science* 72(2): 25–26.

3(5): 929–944.

*of Environment* 109: 328–341.

*Management* 255: 3288–3296.

*Sensing* 26(7): 1511–1519.

**2**

**Fusion of Optical and Thermal Imagery and**

**LiDAR Data for Application to 3-D Urban**

**Environment and Structure Monitoring**

*2Remote Sensing Laboratory, Department of Geography and Environment,*

For many years, panchromatic aerial photographs have been the main source of remote sensing data for detailed inventories of urban areas. Traditionally, building extraction relies mainly on manual photo-interpretation which is an expensive process, especially when a large amount of data must be processed (Ameri, 2000). The characterization of a given object bases on its visible information, such as: shape (external form, outline, or configuration), size, patterns (spatial arrangement of an object into distinctive forms), shadow (indicates the outlines, length, and is useful to measure height, or slopes of the terrain), tone (color or brightness of an object, smoothness of the surface, etc.) (Ridd 1995). Automated assessment of urban surface characteristics has been investigated due to the high costs of visual interpretation. Most of those studies used multispectral satellite imagery of medium to low spatial resolution (Landsat-TM, SPOT-HRV, IRS-LISS, ALI and CHRIS-PROBA) and were based on common image-analysis techniques (e.g. maximum likelihood (ML) classification, principal components analysis (PCA) or spectral indices (Richards and Jia 1999)). The problems of limited spatial resolution over urban areas have been overcome with the wider availability of space-borne systems, which characterized by large swath and high spatial and temporal resolutions (e.g. Worl-View2). However, the limits on spectral information of nonvegetative material render their exact identification difficult. In this regard, the hyperspectral remote sensing (HRS) technology, using data from airborne sensors (e.g. AVIRIS, GER, DAIS, HyMap, AISA-Dual), has opened up a new frontier for surface differentiation of homogeneous material based on spectral characteristics (Heiden et al. 2007). This capability also offers the potential to extract quantitative information on biochemical, geochemical and chemical parameters of the targets in question (Roessner et al.

The most common approach to characterizing urban environments from remote sensing imagery is land-use classification, i.e. assigning all pixels in the image to mutually exclusive classes, such as residential, industrial, recreational, etc. (Ridd 1995, Price 1998). In contrast, mapping the urban environment in terms of its physical components preserves the

**1. Introduction**

1998).

Anna Brook1, Marijke Vandewal1 and Eyal Ben-Dor2

*1Royal Military Academy, CISS Department, Brussels*

*Tel-Aviv University, Tel-Aviv*

*1Belgium 2Israel*

## **Fusion of Optical and Thermal Imagery and LiDAR Data for Application to 3-D Urban Environment and Structure Monitoring**

Anna Brook1, Marijke Vandewal1 and Eyal Ben-Dor2 *1Royal Military Academy, CISS Department, Brussels 2Remote Sensing Laboratory, Department of Geography and Environment, Tel-Aviv University, Tel-Aviv 1Belgium 2Israel*

## **1. Introduction**

26 Will-be-set-by-IN-TECH

28 Remote Sensing – Advanced Techniques and Platforms

Yu, X., Hyyppä, J., Holopainen, M. & Vastaranta, M.... (2010). Comparison of area-based

Yu, X., Hyyppä, J., Vstarana, M., Holopainen, M. & Viitala, R. (2011). Predicting individual tree

*ISPRS Journal of Photogrammetry and Remote Sensing* 66(1): 28–37.

*Sensing* 2: 1481–1495.

and individual tree-based methods for predicting plot-level forest attributes, *Remote*

attributes from airborne laser point clouds based on the random forests technique,

For many years, panchromatic aerial photographs have been the main source of remote sensing data for detailed inventories of urban areas. Traditionally, building extraction relies mainly on manual photo-interpretation which is an expensive process, especially when a large amount of data must be processed (Ameri, 2000). The characterization of a given object bases on its visible information, such as: shape (external form, outline, or configuration), size, patterns (spatial arrangement of an object into distinctive forms), shadow (indicates the outlines, length, and is useful to measure height, or slopes of the terrain), tone (color or brightness of an object, smoothness of the surface, etc.) (Ridd 1995). Automated assessment of urban surface characteristics has been investigated due to the high costs of visual interpretation. Most of those studies used multispectral satellite imagery of medium to low spatial resolution (Landsat-TM, SPOT-HRV, IRS-LISS, ALI and CHRIS-PROBA) and were based on common image-analysis techniques (e.g. maximum likelihood (ML) classification, principal components analysis (PCA) or spectral indices (Richards and Jia 1999)). The problems of limited spatial resolution over urban areas have been overcome with the wider availability of space-borne systems, which characterized by large swath and high spatial and temporal resolutions (e.g. Worl-View2). However, the limits on spectral information of nonvegetative material render their exact identification difficult. In this regard, the hyperspectral remote sensing (HRS) technology, using data from airborne sensors (e.g. AVIRIS, GER, DAIS, HyMap, AISA-Dual), has opened up a new frontier for surface differentiation of homogeneous material based on spectral characteristics (Heiden et al. 2007). This capability also offers the potential to extract quantitative information on biochemical, geochemical and chemical parameters of the targets in question (Roessner et al. 1998).

The most common approach to characterizing urban environments from remote sensing imagery is land-use classification, i.e. assigning all pixels in the image to mutually exclusive classes, such as residential, industrial, recreational, etc. (Ridd 1995, Price 1998). In contrast, mapping the urban environment in terms of its physical components preserves the

Fusion of Optical and Thermal Imagery and LiDAR Data for

faces some challenges: it is time-consuming and expensive.

data type or analysis approach (Allen & Lu 2003).

reference system WGS 84.

reliable (Margarit et al. 2007).

information of the environment.

Application to 3-D Urban Environment and Structure Monitoring 31

altitude of it determined by GPS/INS, therefore, the raw data are collected in the GPS

Generally, 3-D urban built environment models are created using CAD (computer-aided design) tools. There have been many successful projects which have produced detailed and realistic 3-D models for a diverse range of cities (Dodge et al. 1998, Bulmer 2001, Jepson et al. 2001). These city models were created with accurate building models compiled with orthophotographs and exhibited an impressive, realistic urban environment (Chan et al. 1998). However, the creation of 3-D city models using CAD tools and orthophotographs

The analysis of InSAR (Interferometric Synthetic Aperture Radar) and SAR (Synthetic Aperture Radar) data for urban built targets has several important benefits, such as the ability to adopt numerical tools, and the ability to provide results resembling the real-world situation. In addition, a relation can be found between target geometry and the measured scattering, and according to target-scattering properties, height-retrieval algorithms can be developed. The limitation of this method is that the targets in urban models have to be as detailed as possible; otherwise the results obtained in the modeled environment will be not

The use of 3-D high-spatial-resolution applications in urban built environments is a mainstay of architecture and engineering practice. However, engineering practices are increasingly incorporating different data sets and alternative dissemination systems. Understanding, modeling and forecasting the trends in urban environments are important to recognize and assess the impact of urbanization for resource managers and urban planners. Many applications are suitable sources of reliable information on the multiple facets of the urban environment (Jensen & Cowen 1999, Donnay et al. 2001, Herold et al. 2003). These models have provided simulations of urban dynamics and an understanding of the patterns and processes associated with urbanization (Herold et al. 2005). However, the complexity of urban systems makes it difficult to adequately address changes using a single

This chapter presents techniques for data fusion and data registration. The ability to include an accurate and realistic 3-D position, quantitative spectral information, thermal properties and temporal changes provides a near-real-time monitoring system for photogrammetric and urban planning purposes. The method is focusing on registration of multi-sensor and multitemporal information for 3-D urban environment monitoring applications. Generally, data registration is a critical pre-processing procedure in all remote-sensing applications that utilizes multiple sensors inputs, including multi-sensor data fusion, temporal change detection, and data mosaicking. The main objective of this research is a fully controlled, nearreal-time, natural and realistic monitoring system for an urban environment. This task led us first to combine the image-processing and map-matching procedures, and then to incorporate remote sensing and GIS tools into an integrative method for data fusion and registration. To support this new data model, traditional spatial databases were extended to support 5-D data. This chapter is organized as follows. Section 2 describes the materials and methods, which are implemented in the 3-D urban environment model presented in Section 3. Section 4 addresses to the generic 3-D urban application, which involves data fusion and contextual

heterogeneity of urban land cover better than traditional land-use classification (Jensen & Cowen, 1999), characterizes urban land cover independent of analyst-imposed definitions and more accurately captures changes with time (Rashed et al. 2001).

Hyperspectral thermal infrared (TIR) remote sensing has rapidly advanced with the development of airborne systems and follows years of laboratory studies (Hunt & Vincent 1968, Conel 1969, Vincent & Thomson 1972, Logan et al. 1975, Salisbury et al. 1987). The radiance emitted from a surface in thermal infrared (4-13μm) is a function of its temperature and emissivity. Emittance and reflectance are complex processes that depend not only on the absorption coefficient of materials but also on their reflective index, physical state and temperature. Most urban built environment studies are taking into account both temperature and emissivity variations, since these relate to the targets identification, mapping and monitoring and provide a mean for practical application.

The hyperspectral thermal imagery provides the ability for mapping and monitoring temperatures related to the man-made materials. The urban heat island (UHI) has been one of the most studied and the best-known phenomena of urban climate investigated by thermal imagery (Carlson et al., 1981; Vukovich, 1983; Kidder & Wu, 1987; Roth et al., 1989; Nichol, 1996). The preliminary studies have reported similarities between spatial patterns of air temperature and remotely sensed surface temperature (Henry et al., 1989; Nichol 1994), whereas progress studies suggest significant differences, including the time of day and season of maximum UHI development and the relationship between land use and UHI intensity (Roth et al., 1989). The recent high-resolution airborne systems determine the thermal performance of the building that can be used to identify heating and cooling loss due to poor construction, missing or inadequate insulation and moisture intrusion.

The spectral (reflective and thermal) characteristics of the urban surfaces are known to be rather complex as they are composed of many materials. Given the high degree of spatial and spectral heterogeneity within various artificial and natural land cover categories, the application of remote sensing technology to mapping built urban environments requires specific attention to both 3-D and spectral domains (Segl et al. 2003). Segl confirms that profiling hyperspectral TIR can successfully identify and discriminate a variety of silicates and carbonates, as well as variations in the chemistry of some silicates. The integration of VNIR-SWIR and TIR results can provide useful information to remove possible ambiguous interpretations in unmixed sub-pixel surfaces and materials. The image interpretation is based on the thematic categories (Roessner et al. 2001), which are defined by the rules of urban mapping and land-uses.

The ultimate aim in photogrammetry in generating an urban landscape model is to show the objects in an urban area in 3-D (Juan et al. 2007). As the most permanent features in the urban environment, an accurate extraction of buildings and roads is significant for urban planning and cartographic mapping. Acquisition and integration of data for the built urban environment has always been a challenge due to the high cost and heterogeneous nature of the data sets (Wang 2008). Thus, over the last few years, LiDAR (LIght Detection And Ranging) has been widely applied in the field of photogrammetry and urban 3-D analysis (Tao 2001, Zhou 2004). Airborne LiDAR technique provides geo-referenced 3-D dense points ("cloud") measured roughly perpendicular to the direction of flight over a reflective surface on the ground. This system integrates three basic data-collection tools: a laser scanner, a global positioning system (GPS) and an inertial measuring unit (IMU). The position and

heterogeneity of urban land cover better than traditional land-use classification (Jensen & Cowen, 1999), characterizes urban land cover independent of analyst-imposed definitions

Hyperspectral thermal infrared (TIR) remote sensing has rapidly advanced with the development of airborne systems and follows years of laboratory studies (Hunt & Vincent 1968, Conel 1969, Vincent & Thomson 1972, Logan et al. 1975, Salisbury et al. 1987). The radiance emitted from a surface in thermal infrared (4-13μm) is a function of its temperature and emissivity. Emittance and reflectance are complex processes that depend not only on the absorption coefficient of materials but also on their reflective index, physical state and temperature. Most urban built environment studies are taking into account both temperature and emissivity variations, since these relate to the targets identification,

The hyperspectral thermal imagery provides the ability for mapping and monitoring temperatures related to the man-made materials. The urban heat island (UHI) has been one of the most studied and the best-known phenomena of urban climate investigated by thermal imagery (Carlson et al., 1981; Vukovich, 1983; Kidder & Wu, 1987; Roth et al., 1989; Nichol, 1996). The preliminary studies have reported similarities between spatial patterns of air temperature and remotely sensed surface temperature (Henry et al., 1989; Nichol 1994), whereas progress studies suggest significant differences, including the time of day and season of maximum UHI development and the relationship between land use and UHI intensity (Roth et al., 1989). The recent high-resolution airborne systems determine the thermal performance of the building that can be used to identify heating and cooling loss

due to poor construction, missing or inadequate insulation and moisture intrusion.

urban mapping and land-uses.

The spectral (reflective and thermal) characteristics of the urban surfaces are known to be rather complex as they are composed of many materials. Given the high degree of spatial and spectral heterogeneity within various artificial and natural land cover categories, the application of remote sensing technology to mapping built urban environments requires specific attention to both 3-D and spectral domains (Segl et al. 2003). Segl confirms that profiling hyperspectral TIR can successfully identify and discriminate a variety of silicates and carbonates, as well as variations in the chemistry of some silicates. The integration of VNIR-SWIR and TIR results can provide useful information to remove possible ambiguous interpretations in unmixed sub-pixel surfaces and materials. The image interpretation is based on the thematic categories (Roessner et al. 2001), which are defined by the rules of

The ultimate aim in photogrammetry in generating an urban landscape model is to show the objects in an urban area in 3-D (Juan et al. 2007). As the most permanent features in the urban environment, an accurate extraction of buildings and roads is significant for urban planning and cartographic mapping. Acquisition and integration of data for the built urban environment has always been a challenge due to the high cost and heterogeneous nature of the data sets (Wang 2008). Thus, over the last few years, LiDAR (LIght Detection And Ranging) has been widely applied in the field of photogrammetry and urban 3-D analysis (Tao 2001, Zhou 2004). Airborne LiDAR technique provides geo-referenced 3-D dense points ("cloud") measured roughly perpendicular to the direction of flight over a reflective surface on the ground. This system integrates three basic data-collection tools: a laser scanner, a global positioning system (GPS) and an inertial measuring unit (IMU). The position and

and more accurately captures changes with time (Rashed et al. 2001).

mapping and monitoring and provide a mean for practical application.

altitude of it determined by GPS/INS, therefore, the raw data are collected in the GPS reference system WGS 84.

Generally, 3-D urban built environment models are created using CAD (computer-aided design) tools. There have been many successful projects which have produced detailed and realistic 3-D models for a diverse range of cities (Dodge et al. 1998, Bulmer 2001, Jepson et al. 2001). These city models were created with accurate building models compiled with orthophotographs and exhibited an impressive, realistic urban environment (Chan et al. 1998). However, the creation of 3-D city models using CAD tools and orthophotographs faces some challenges: it is time-consuming and expensive.

The analysis of InSAR (Interferometric Synthetic Aperture Radar) and SAR (Synthetic Aperture Radar) data for urban built targets has several important benefits, such as the ability to adopt numerical tools, and the ability to provide results resembling the real-world situation. In addition, a relation can be found between target geometry and the measured scattering, and according to target-scattering properties, height-retrieval algorithms can be developed. The limitation of this method is that the targets in urban models have to be as detailed as possible; otherwise the results obtained in the modeled environment will be not reliable (Margarit et al. 2007).

The use of 3-D high-spatial-resolution applications in urban built environments is a mainstay of architecture and engineering practice. However, engineering practices are increasingly incorporating different data sets and alternative dissemination systems. Understanding, modeling and forecasting the trends in urban environments are important to recognize and assess the impact of urbanization for resource managers and urban planners. Many applications are suitable sources of reliable information on the multiple facets of the urban environment (Jensen & Cowen 1999, Donnay et al. 2001, Herold et al. 2003). These models have provided simulations of urban dynamics and an understanding of the patterns and processes associated with urbanization (Herold et al. 2005). However, the complexity of urban systems makes it difficult to adequately address changes using a single data type or analysis approach (Allen & Lu 2003).

This chapter presents techniques for data fusion and data registration. The ability to include an accurate and realistic 3-D position, quantitative spectral information, thermal properties and temporal changes provides a near-real-time monitoring system for photogrammetric and urban planning purposes. The method is focusing on registration of multi-sensor and multitemporal information for 3-D urban environment monitoring applications. Generally, data registration is a critical pre-processing procedure in all remote-sensing applications that utilizes multiple sensors inputs, including multi-sensor data fusion, temporal change detection, and data mosaicking. The main objective of this research is a fully controlled, nearreal-time, natural and realistic monitoring system for an urban environment. This task led us first to combine the image-processing and map-matching procedures, and then to incorporate remote sensing and GIS tools into an integrative method for data fusion and registration. To support this new data model, traditional spatial databases were extended to support 5-D data.

This chapter is organized as follows. Section 2 describes the materials and methods, which are implemented in the 3-D urban environment model presented in Section 3. Section 4 addresses to the generic 3-D urban application, which involves data fusion and contextual information of the environment.

Fusion of Optical and Thermal Imagery and LiDAR Data for

the frame rate is 33 fps with adjustable spectral sampling.

are processed by applying the surface-based clustering methods.

**2.3.1 Hyperspectral airborne and ground imagery**

assessed and quantified for each mission.

(Brook & Ben-Dor 2011b).

infrared camera (FLIR Systems, Inc.).

**2.3 Data processing**

Application to 3-D Urban Environment and Structure Monitoring 33

The ground spectral camera HS (Specim Ltd.) is a pushbroom scan camera that integrate ImSpector imaging spectrograph and an area monochrome camera. The camera's sensitive high speed interlaced CCD (Charge-Coupled Device) detector simultaneously acquires images in 850 contiguous spectral bands and covers the 0.4 to 1 µm spectral region with bandwidths of 2.8 nm. The spatial resolution is 1600 pixels in the cross-track direction, and

The ground truth reflectance data were measured for the calibration/validation targets by the ASD "FieldSpec Pro" (ASD.Inc, Boulder, CO) VNIR-SWIR spectrometer. Internally averaged scans were 100 ms each. The wavelength-dependent signal-to-noise ratio (S/N) is estimated by taking repeat measurements of a Spectralon white-reference panel over a 10 min interval and analyzing the spectral variation across this period. For each sample, three spectral replicates were acquired and the average was used as the representative spectrum. The ground truth thermal data were collected by a thermometer and thermocouples installed within calibration/validation targets (water bodies) and a thermal radiometer

This research integrates multi-sensor (airborne sensor, ground camera and field devises) and multi-temporal information into fully operational monitoring application. The aim of this sub-paragraph is to present several techniques for imagery and LiDAR data processing. The classification approaches for airborne and ground hyperspectral imagery are firstly presented. The radiance measured by these sensors strongly depends on the atmospheric conditions, which might bias the results of material identification/classification algorithms that rely on hyperspectral image data. The desire to relate imagery data to intrinsic surface properties has led to the development of atmospheric correction algorithms that attempt to recover surface reflectance or emission from at-sensor radiance. Secondly, the LiDAR data

Accurate spectral reflectance information is a key factor in retrieving correct thematic results. In general, the quality of HRS sensors varies from very high to moderate (and even very poor) in terms of signal-to-noise ratio, radiometric accuracy and sensor stability. Instability of the sensors' radiometric performance (stripes, saturation, etc.) might be caused by either known or unknown factors encountered during sensor transport, installation and/or even data acquisition. As part of data pre-processing, these distortions have to be

A full-chain atmospheric calibration SVC (supervised vicarious calibration) method (Brook & Ben-Dor 2011a) is applied to extract reflectance information from hyperspectral imagery. This method is based on a mission-by-mission approach, followed by a unique vicarious calibration site. In this study, the acquired AISA-Dual and HS images were subjected to the SVC method, which includes two radiometric recalibration techniques (F1 and F2) and two atmospheric correction approaches (F3 and F4). The atmospheric correction incorporate deshadow algorithm, which is applied o the map provided by the boresight ratio band

## **2. Materials and methods**

## **2.1 Study area**

Two separate datasets were utilized in this study. The first dataset was acquired over the suburban Mediterranean area on 10 Oct 2006 at 03h37 UTC and at 11h20 UTC. This area combines natural and engineered terrains (average elevation of 560m above sea level), a hill in the north of the studied polygon area and a valley in the center. The entire scene consists of rows of terraced houses located at the center of the image. The neighborhood consists of cottage houses (two and three floors) with tile roofs, flat white-colored concrete roofs and balconies, asphalt roads and parking lots, planted and natural vegetation, gravel paths and bare brown forest soil. The height of large buildings ranges from 8 to 16 m. A group of tall pine trees with various heights and shapes are located on the streets and the Mediterranean forest can be found in the corner of the scene.

The second dataset was acquired over urban settlement, on 15 Aug 2007 at 02h54 UTC and at 12h30 UTC. This area combines natural, agriculture and engineered terrains (average elevation of 30m above sea level). The urban settlement consists of houses (two and three floors) and public buildings (schools and municipalities buildings) with flat concrete, asphalt or whitewash roofing, asphalt roads and parking lots, planted and natural vegetation, gravel paths, bare brown reddish Mediterranean and agriculture soils, greenhouses and whitewash henhouse roofing. The height of large buildings ranges from 3 to 21 m.

## **2.2 Data-acquisition systems**

The research combines airborne and ground data collected from different platforms and different operated systems. The collected imagery data were validated and compared to the ground truth in situ measurements collected during the campaigns.

The first airborne platform carries AISA-Dual hyperspectral system. The airborne imaging spectrometer AISA-Dual (Specim Ltd.) is a dual hyperspectral pushbroom system, which combines the Aisa EAGLE (VNIR region) and Aisa HAWK (SWIR region) sensors. For the selected campaigns, the sensor simultaneously acquired images in 198 contiguous spectral bands, covering the 0.4 to 2.5 µm spectral region with bandwidths of ~10 nm for Aisa EAGLE and ~5 nm for Aisa HAWK. The sensor altitude was 10,000 ft, providing a 1.6 m spatial resolution for 286 pixels in the cross-track direction. A standard AISA-Dual data set is a 3-D data cube in a non-earth coordinate system (raw matrix geometry).

The second airborne platform carries hyperspectral TIR system, which is a line-scanner with 28 spectral bands in the thermal ranges 3-5 μm and 8-13 μm. It has 328 pixels in the crosstrack direction and hundreds of pixels in the along-track direction with a spatial resolution of 1.4m.

The third airborne platform carries the LiDAR system. This system operates at 1500 nm wavelength with a 165 kHz laser repetition rate and 100 Hz scanning rate and provides a spatial/footprint resolution of 0.5 m and an accuracy of 0.1 m. The scanner has a multi-pulse system that could record up to five different returns, but in this study, only the first return was recorded and analyzed.

The ground spectral camera HS (Specim Ltd.) is a pushbroom scan camera that integrate ImSpector imaging spectrograph and an area monochrome camera. The camera's sensitive high speed interlaced CCD (Charge-Coupled Device) detector simultaneously acquires images in 850 contiguous spectral bands and covers the 0.4 to 1 µm spectral region with bandwidths of 2.8 nm. The spatial resolution is 1600 pixels in the cross-track direction, and the frame rate is 33 fps with adjustable spectral sampling.

The ground truth reflectance data were measured for the calibration/validation targets by the ASD "FieldSpec Pro" (ASD.Inc, Boulder, CO) VNIR-SWIR spectrometer. Internally averaged scans were 100 ms each. The wavelength-dependent signal-to-noise ratio (S/N) is estimated by taking repeat measurements of a Spectralon white-reference panel over a 10 min interval and analyzing the spectral variation across this period. For each sample, three spectral replicates were acquired and the average was used as the representative spectrum. The ground truth thermal data were collected by a thermometer and thermocouples installed within calibration/validation targets (water bodies) and a thermal radiometer infrared camera (FLIR Systems, Inc.).

## **2.3 Data processing**

32 Remote Sensing – Advanced Techniques and Platforms

Two separate datasets were utilized in this study. The first dataset was acquired over the suburban Mediterranean area on 10 Oct 2006 at 03h37 UTC and at 11h20 UTC. This area combines natural and engineered terrains (average elevation of 560m above sea level), a hill in the north of the studied polygon area and a valley in the center. The entire scene consists of rows of terraced houses located at the center of the image. The neighborhood consists of cottage houses (two and three floors) with tile roofs, flat white-colored concrete roofs and balconies, asphalt roads and parking lots, planted and natural vegetation, gravel paths and bare brown forest soil. The height of large buildings ranges from 8 to 16 m. A group of tall pine trees with various heights and shapes are located on the streets and the Mediterranean

The second dataset was acquired over urban settlement, on 15 Aug 2007 at 02h54 UTC and at 12h30 UTC. This area combines natural, agriculture and engineered terrains (average elevation of 30m above sea level). The urban settlement consists of houses (two and three floors) and public buildings (schools and municipalities buildings) with flat concrete, asphalt or whitewash roofing, asphalt roads and parking lots, planted and natural vegetation, gravel paths, bare brown reddish Mediterranean and agriculture soils, greenhouses and whitewash henhouse roofing. The height of large buildings ranges from 3

The research combines airborne and ground data collected from different platforms and different operated systems. The collected imagery data were validated and compared to the

The first airborne platform carries AISA-Dual hyperspectral system. The airborne imaging spectrometer AISA-Dual (Specim Ltd.) is a dual hyperspectral pushbroom system, which combines the Aisa EAGLE (VNIR region) and Aisa HAWK (SWIR region) sensors. For the selected campaigns, the sensor simultaneously acquired images in 198 contiguous spectral bands, covering the 0.4 to 2.5 µm spectral region with bandwidths of ~10 nm for Aisa EAGLE and ~5 nm for Aisa HAWK. The sensor altitude was 10,000 ft, providing a 1.6 m spatial resolution for 286 pixels in the cross-track direction. A standard AISA-Dual data set

The second airborne platform carries hyperspectral TIR system, which is a line-scanner with 28 spectral bands in the thermal ranges 3-5 μm and 8-13 μm. It has 328 pixels in the crosstrack direction and hundreds of pixels in the along-track direction with a spatial resolution

The third airborne platform carries the LiDAR system. This system operates at 1500 nm wavelength with a 165 kHz laser repetition rate and 100 Hz scanning rate and provides a spatial/footprint resolution of 0.5 m and an accuracy of 0.1 m. The scanner has a multi-pulse system that could record up to five different returns, but in this study, only the first return

ground truth in situ measurements collected during the campaigns.

is a 3-D data cube in a non-earth coordinate system (raw matrix geometry).

**2. Materials and methods**

forest can be found in the corner of the scene.

**2.1 Study area**

to 21 m.

of 1.4m.

was recorded and analyzed.

**2.2 Data-acquisition systems**

This research integrates multi-sensor (airborne sensor, ground camera and field devises) and multi-temporal information into fully operational monitoring application. The aim of this sub-paragraph is to present several techniques for imagery and LiDAR data processing.

The classification approaches for airborne and ground hyperspectral imagery are firstly presented. The radiance measured by these sensors strongly depends on the atmospheric conditions, which might bias the results of material identification/classification algorithms that rely on hyperspectral image data. The desire to relate imagery data to intrinsic surface properties has led to the development of atmospheric correction algorithms that attempt to recover surface reflectance or emission from at-sensor radiance. Secondly, the LiDAR data are processed by applying the surface-based clustering methods.

## **2.3.1 Hyperspectral airborne and ground imagery**

Accurate spectral reflectance information is a key factor in retrieving correct thematic results. In general, the quality of HRS sensors varies from very high to moderate (and even very poor) in terms of signal-to-noise ratio, radiometric accuracy and sensor stability. Instability of the sensors' radiometric performance (stripes, saturation, etc.) might be caused by either known or unknown factors encountered during sensor transport, installation and/or even data acquisition. As part of data pre-processing, these distortions have to be assessed and quantified for each mission.

A full-chain atmospheric calibration SVC (supervised vicarious calibration) method (Brook & Ben-Dor 2011a) is applied to extract reflectance information from hyperspectral imagery. This method is based on a mission-by-mission approach, followed by a unique vicarious calibration site. In this study, the acquired AISA-Dual and HS images were subjected to the SVC method, which includes two radiometric recalibration techniques (F1 and F2) and two atmospheric correction approaches (F3 and F4). The atmospheric correction incorporate deshadow algorithm, which is applied o the map provided by the boresight ratio band (Brook & Ben-Dor 2011b).

Fusion of Optical and Thermal Imagery and LiDAR Data for

Application to 3-D Urban Environment and Structure Monitoring 35

The validation of the thematic map is performed by comparing ground truth and image reflectance data of the selected targets. The ten well-known targets (areas of approximately 30-40 pixels) were spectrally measured (using ASD SpecPro) and documented. The overall accuracy for the Ma'alot Tarshiha images was 96.8 and for the Qalansawe images it was 97.4. The exact location of each target within the scenes was captured using aerial orthophoto and ground truth field survey. The confusion matrices (Tables 1 and 2) and ROC (receiver operating characteristic) curve (Table 3) were calculated by comparison between number of pixels in each class (concrete, asphalt, scuffed asphalt) and ground truth maps. The overall accuracy of both images stands in good agreement, thus it can be concluded that the

Ground truth (%)

Ground truth (%)

suggested classification algorithm (Figure 1) performance is stable and accurate.

Unclassified 0 0 0 Concrete **96.2** 1.7 2.1 Asphalt 1.1 **98** 0.9 Scuffed Asphalt 2.8 0.2 **97**

Table 1. Confusion matrix of the Ma'alot Tarshiha image for selected classes

(Correspondence accuracies are in bold.)

accuracies are in bold.)

Class Concrete Asphalt Scuffed asphalt

Class Concrete Asphalt Scuffed asphalt

Table 2. Confusion matrix of the Qalansawe image for selected classes (Correspondence

DR 0.97 0.97 0.94 0.98 0.96 0.93 Area 0.99 0.99 0.96 0.99 0.98 0.95 Table 3. Detection rates (DR) of concrete, asphalt and scuffed asphalt for false alarm

Atmospheric correction is a key processing step for extracting information from thermal infrared imagery. The ground-leaving radiance combined with temperature/emissivity separation (TES) algorithms are generated and supplied to in-scene atmospheric

probability 0.1 according to ROC and area under the curve

**2.3.2 Thermal airborne and ground imagery**

Ma'alot image Qalansawe image Concrete Asphalt Scuffed asphalt Concrete Asphalt Scuffed asphalt

Unclassified 0 0 0 Concrete **96.3** 0.8 2.9 Asphalt 0 **98.4** 1.6 Scuffed Asphalt 4.5 0 **95.5**

The hyperspectral reflectance images are subjected to the data processing stage, which is operated in four steps (Figure 1). First step is a general coarse classification. Each "pure" pixel is assigned to a class in order to predefine the threshold of the probabilistic output of a support vector machine (SVM) algorithm, or remains unclassified (Villa et al., 2011). The unclassified pixels might associate with mixed spectra pixels, thus their classification is addressed at the third stage by the unmixing method in order to obtain the abundance fraction of each endmember class. Prior to this step, a second step is applied, where spectral data are reduced by the selected algorithm. The input variables in terms of absorption features can be reduced through a sequential forward selection (SFS) algorithm (Whitney, 1971). This method starts with the inclusion of feature sets one by one to minimize the prediction error of a linear regression model and focuses on conditional exclusion based on feature significance (Pudil et al., 1994). This step is proven to enhance overall performance of spectral models.

Fig. 1. Flow chart scheme of the classification approach for hyperspectral airborne and ground data

The nonnegative matrix factorization (NMF) was offered as an alternative method for linear unmixing (Lee et al., 2000). This algorithm search for the source and the transform by factorizing a matrix subject to positive constraints based on gradient optimization and Euclidean norm designation (Pauca et al, 2006; Robila & Maciak, 2006). We generated an algorithm that starts with the random linear transform to the nonnegative source data. The algorithm is continuously computing scalar factors that are chosen to produce the "best" intermediate source and transform. At each step of the algorithm the source and transform should remain positive. The final stage is a method for image segmentation combined with a Markov random field (MRF) model under a Bayesian framework (Yang & Jiang, 2003).

The hyperspectral reflectance images are subjected to the data processing stage, which is operated in four steps (Figure 1). First step is a general coarse classification. Each "pure" pixel is assigned to a class in order to predefine the threshold of the probabilistic output of a support vector machine (SVM) algorithm, or remains unclassified (Villa et al., 2011). The unclassified pixels might associate with mixed spectra pixels, thus their classification is addressed at the third stage by the unmixing method in order to obtain the abundance fraction of each endmember class. Prior to this step, a second step is applied, where spectral data are reduced by the selected algorithm. The input variables in terms of absorption features can be reduced through a sequential forward selection (SFS) algorithm (Whitney, 1971). This method starts with the inclusion of feature sets one by one to minimize the prediction error of a linear regression model and focuses on conditional exclusion based on feature significance (Pudil et

al., 1994). This step is proven to enhance overall performance of spectral models.

Fig. 1. Flow chart scheme of the classification approach for hyperspectral airborne and

The nonnegative matrix factorization (NMF) was offered as an alternative method for linear unmixing (Lee et al., 2000). This algorithm search for the source and the transform by factorizing a matrix subject to positive constraints based on gradient optimization and Euclidean norm designation (Pauca et al, 2006; Robila & Maciak, 2006). We generated an algorithm that starts with the random linear transform to the nonnegative source data. The algorithm is continuously computing scalar factors that are chosen to produce the "best" intermediate source and transform. At each step of the algorithm the source and transform should remain positive. The final stage is a method for image segmentation combined with a Markov random field (MRF) model under a Bayesian framework (Yang & Jiang, 2003).

ground data

The validation of the thematic map is performed by comparing ground truth and image reflectance data of the selected targets. The ten well-known targets (areas of approximately 30-40 pixels) were spectrally measured (using ASD SpecPro) and documented. The overall accuracy for the Ma'alot Tarshiha images was 96.8 and for the Qalansawe images it was 97.4. The exact location of each target within the scenes was captured using aerial orthophoto and ground truth field survey. The confusion matrices (Tables 1 and 2) and ROC (receiver operating characteristic) curve (Table 3) were calculated by comparison between number of pixels in each class (concrete, asphalt, scuffed asphalt) and ground truth maps. The overall accuracy of both images stands in good agreement, thus it can be concluded that the suggested classification algorithm (Figure 1) performance is stable and accurate.

Table 1. Confusion matrix of the Ma'alot Tarshiha image for selected classes (Correspondence accuracies are in bold.)

Table 2. Confusion matrix of the Qalansawe image for selected classes (Correspondence accuracies are in bold.)

Table 3. Detection rates (DR) of concrete, asphalt and scuffed asphalt for false alarm probability 0.1 according to ROC and area under the curve

## **2.3.2 Thermal airborne and ground imagery**

Atmospheric correction is a key processing step for extracting information from thermal infrared imagery. The ground-leaving radiance combined with temperature/emissivity separation (TES) algorithms are generated and supplied to in-scene atmospheric

Fusion of Optical and Thermal Imagery and LiDAR Data for

0.9

0.92

0.94

0.96

**Emissivity**

0.98

1

A

Application to 3-D Urban Environment and Structure Monitoring 37

**Tile Roof**

3 5 7 911 13

סידרה1

Image Ground Truth

סידרה2

סידרה 1 סידרה 2

Image Ground Truth

**Wavelength (µm)**

**Bitumen**

Fig. 3. Emissivity calculated from the thermal radiance. A is a tile roof and B is a bitumen

origins in mining and geostatistical applications involving spatially and temporally

3 5 7 9 11 13

**Wavelength (µm)**

The surface analysis (Figure 4) is first represented as a DEM (digital elevation model) of the scanned scene, where data are separated into on-terrain and off-terrain points (Masaharu and Ohtsubo 2002). In this study, the Kriging Gaussian correlation function was utilized to visualize and illustrate the edited DEM as a surface-response function. Note that the

The DTM (digital terrain model) was created by a morphological scale-opening filter, using square structural elements (Rottensteiner et al., 2003). Then, according to the filter, the slope

interpolation converts irregularly spaced LiDAR data to a self-adaptive DSM.

roof

correlated data (Cressie 1993).

0.84 0.86 0.88 0.9 0.92 0.94 0.96 0.98 1

**Emissivity**

B

compensation ISAC1 (Young et al., 2002). This model requires only the calibrated, ataperture radiance data to estimate the upwelling radiance and transmittance of the atmosphere. It is an effective atmospheric correction that produces spectra that compare favorably to the Planck function.

The ground truth must include several targets as water, sand or soil continuously measured by installed thermocouples. The generating atmospheric data cube may be used as an input to a temperature emissivity separation algorithm (normalized emissivity method). The proposed thermal classification method follows the same four stages of data processing (SVM's probabilistic map; data reduction; umnixing; classification) applied to the preprocessed emissivity imagery (Figure 2).

Fig. 2. Flow chart scheme of the thermal airborne and ground data preprocessing

From the physical definition, the spectral characteristics of urban materials in the reflective and thermal ranges are related. Segl (Segl et al., 2003) showed that materials with high albedos in the reflective range produce low albedos in the thermal range and vice versa, due to a better energy absorption in the reflective region. However, it is reported that bitumen roofing and asphalt pavement generate distinct spectral differences in the thermal wavelength range. The thermal measurements remain a compelling focus on a climate research in the built urban areas. However, the thermal airborne and ground imagery permit definition of UHI (for the ground surface) and resolve streets, roofs and walls. The successful numerical model of the urban areas is acquired during night-time conditions, when solar shading is absent and turbulent interactions are minimal.

The validation of the thematic map is performed by comparing ground truth and image emissivity data. The five targets (concrete, sand lot, bitumen, tile roof and polyethylene) were measured and documented. The resulting emissivity signatures are in good agreement with ground-truth data (two examples in Figure 3A and 3B). The results presented here confirm the robustness and stability of the suggested algorithm.

## **2.3.3 Airborne LiDAR data**

LiDAR data provides precise information about the geometrical properties of the surfaces and can reflect the different shapes and formations in the complex urban environment. The point cloud (irregularly spaced points) was interpolated into the digital surface model (DSM) by applying the Kriging technique (Sacks et al. 1989). The Kriging model has its

<sup>1</sup> ISAC (in-scene atmospheric compensations) model is implemented in ENVI®

compensation ISAC1 (Young et al., 2002). This model requires only the calibrated, ataperture radiance data to estimate the upwelling radiance and transmittance of the atmosphere. It is an effective atmospheric correction that produces spectra that compare

The ground truth must include several targets as water, sand or soil continuously measured by installed thermocouples. The generating atmospheric data cube may be used as an input to a temperature emissivity separation algorithm (normalized emissivity method). The proposed thermal classification method follows the same four stages of data processing (SVM's probabilistic map; data reduction; umnixing; classification) applied to the pre-

Fig. 2. Flow chart scheme of the thermal airborne and ground data preprocessing

Input Images: Airborne Thermal Ground Thermal

Preprocessing: Radiometric correction by system operator

when solar shading is absent and turbulent interactions are minimal.

confirm the robustness and stability of the suggested algorithm.

1 ISAC (in-scene atmospheric compensations) model is implemented in ENVI®

**2.3.3 Airborne LiDAR data**

From the physical definition, the spectral characteristics of urban materials in the reflective and thermal ranges are related. Segl (Segl et al., 2003) showed that materials with high albedos in the reflective range produce low albedos in the thermal range and vice versa, due to a better energy absorption in the reflective region. However, it is reported that bitumen roofing and asphalt pavement generate distinct spectral differences in the thermal wavelength range. The thermal measurements remain a compelling focus on a climate research in the built urban areas. However, the thermal airborne and ground imagery permit definition of UHI (for the ground surface) and resolve streets, roofs and walls. The successful numerical model of the urban areas is acquired during night-time conditions,

Atmospheric correction Temperature Image

Emissivity Image

The validation of the thematic map is performed by comparing ground truth and image emissivity data. The five targets (concrete, sand lot, bitumen, tile roof and polyethylene) were measured and documented. The resulting emissivity signatures are in good agreement with ground-truth data (two examples in Figure 3A and 3B). The results presented here

LiDAR data provides precise information about the geometrical properties of the surfaces and can reflect the different shapes and formations in the complex urban environment. The point cloud (irregularly spaced points) was interpolated into the digital surface model (DSM) by applying the Kriging technique (Sacks et al. 1989). The Kriging model has its

favorably to the Planck function.

processed emissivity imagery (Figure 2).

Fig. 3. Emissivity calculated from the thermal radiance. A is a tile roof and B is a bitumen roof

origins in mining and geostatistical applications involving spatially and temporally correlated data (Cressie 1993).

The surface analysis (Figure 4) is first represented as a DEM (digital elevation model) of the scanned scene, where data are separated into on-terrain and off-terrain points (Masaharu and Ohtsubo 2002). In this study, the Kriging Gaussian correlation function was utilized to visualize and illustrate the edited DEM as a surface-response function. Note that the interpolation converts irregularly spaced LiDAR data to a self-adaptive DSM.

The DTM (digital terrain model) was created by a morphological scale-opening filter, using square structural elements (Rottensteiner et al., 2003). Then, according to the filter, the slope

Fusion of Optical and Thermal Imagery and LiDAR Data for

**Input**

Thematic Map

with no need for human intervention.

handling multi-sensor and multi-type data sets.

Fig. 5. Flow chart scheme of the input data and registration techniques

Hyperspectral / Thermal Airborne Ground Imagery

> Facades Thematic Map

Chung 2004, Fan et al., 2005, Zavorin & Moigne 2005, Xu & Chen 2007).

Application to 3-D Urban Environment and Structure Monitoring 39

**Input** LiDAR

Urban 3-D Objects

detection, and data mosaicking (Moigne et al., 2002). In manual registration, the selection of control points (CPs) is usually performed by a human operator. This has proven to be inaccurate, time-consuming, and unfeasible due to data complexity, which makes it cumbersome or even impossible for the human eye to discern the suitable CPs. Therefore, researchers have focused on automating feature detection to align two or more data sets

*AIRTop Algorithm*

*Manual Registration*

The automatic registration of data sets has generated extensive research interest in the fields of computer vision, medical imaging and remote sensing. Comprehensive reviews have been published by Brown (1992) and Zitova and Flusser (2003). Many proposed schemes for automatic registration employ a multi-resolution process (Viola and Wells 1997, Wu &

The existing automatic data-registration techniques based on spatial information fall into two categories: intensity-based and feature-based (Zitova and Flusser 2003). The featurebased technique extracts salient structures from sensed and reference data sets by accurate feature detectin and by the overlap criterion. As the relevant objects of intereset (e.g., roofs) and lines (e.g., roads) are expected to be stable in time at a fixed position, the feature-based method is more suitable for multi-sensor and multi-data set fusion, change detection and mosaicking. The method generally consists of four steps (Jensen, 2004): (1) CP extraction; (2) transformation-model determination; (3) image transformation and resampling, and (4) assessment of registration accuracy. The first step is the most complex, and its success essentially determines registration accuracy. Thus, the detection method should be able to detect the same features in all projections and in different data, regardless of the particular image/sensor/data type deformation. Despite the achieved performance, the existing methods operate directly on gray intensity values and hence they are not suited for

The suggested algorithm is an adapted version of the four stages AIRTop (Figure 6) algorithm (Brook & Ben-Dor 2011c). First, the significant features are extracted from all input data sets and converted to a vector format. Since the studied scene has a large area, regions of interest (ROI) with relatively large variations are selection. The idea of addressing the registration problem by applying a global-to-local level strategy (the whole image is now divided into regions of interest which are treated as an image) proves to be an elegant way

Fig. 4. Flow chart scheme of the LiDAR data surface analysis

map is estimated. The next stage is to fragment a surface model convolved with highly heterogeneous terrain slopes into subareas with fixed slope (Zhang et al., 2003; Shan & Sampath 2005). At this stage, the terrain is uniformly normalized and the separation between on- and off-terrain points is applicable.

The building boundary is determined by a modified convex hull algorithm (Jarvis 1973) which classifies the cluster data into boundary (contour/edge) and non-boundary (intershape) points (Jarvis 1977). Separating points located on buildings from those on trees and bushes, is a difficult task (Wang & Shan 2009). The common assumption is that the building outlines are separated from the trees in terms of size and shape. The dimensionality learning method, proposed by Wang and Shan (2009), is an efficient technique for this purpose.

In relatively flat urban areas, the roads, which have the same elevation (height) as a bare surface, can be extracted by arrangement examination. The simple geometric and topological relations between streets might be used to improve the consistency of road extraction. First, the DEM data are used to obtain candidate roads, sidewalks and parking lots. Then the road model is established, based on the continuous network of points which are used to extract information such as centerline, edge and width of the road (Akel et al. 2003; Hinz & Baumgartner 2003; Cloude et al., 2004).

## **2.4 Data registration: Automatic and manual approaches**

The optical and thermal imagery and LiDAR data have fundamentally different characteristics. The LiDAR data (monochromatic NIR laser pulse) provides terrain characteristics; hence, optical imagery (radiation reflected back from the surface at many wavelengths) provides ability for in situ, easy, rapid and accurate assessment of many materials on a spatial/spectral/temporal domain, and thermal imagery determines temperatures and radiance signature of urban materials and land covers. Since all these datasets (Figure 5) are crucial for the assessment and classification of the urban area, a novel method for automatic registration and data fusion is needed.

Data fusion techniques combine data from multiple sensors and related information from associated databases. The integrated data set achieves higher accuracy and more specific inferences that might be obtained by the use of single sensor alone. In general, data registration is a critical preprocessing procedure in all remote-sensing applications that utilizes multiple sensor inputs, including multi-sensor data fusion, temporal change

map is estimated. The next stage is to fragment a surface model convolved with highly heterogeneous terrain slopes into subareas with fixed slope (Zhang et al., 2003; Shan & Sampath 2005). At this stage, the terrain is uniformly normalized and the separation

The building boundary is determined by a modified convex hull algorithm (Jarvis 1973) which classifies the cluster data into boundary (contour/edge) and non-boundary (intershape) points (Jarvis 1977). Separating points located on buildings from those on trees and bushes, is a difficult task (Wang & Shan 2009). The common assumption is that the building outlines are separated from the trees in terms of size and shape. The dimensionality learning method, proposed by Wang and Shan (2009), is an efficient technique for this purpose.

In relatively flat urban areas, the roads, which have the same elevation (height) as a bare surface, can be extracted by arrangement examination. The simple geometric and topological relations between streets might be used to improve the consistency of road extraction. First, the DEM data are used to obtain candidate roads, sidewalks and parking lots. Then the road model is established, based on the continuous network of points which are used to extract information such as centerline, edge and width of the road (Akel et al.

The optical and thermal imagery and LiDAR data have fundamentally different characteristics. The LiDAR data (monochromatic NIR laser pulse) provides terrain characteristics; hence, optical imagery (radiation reflected back from the surface at many wavelengths) provides ability for in situ, easy, rapid and accurate assessment of many materials on a spatial/spectral/temporal domain, and thermal imagery determines temperatures and radiance signature of urban materials and land covers. Since all these datasets (Figure 5) are crucial for the assessment and classification of the urban area, a novel

Data fusion techniques combine data from multiple sensors and related information from associated databases. The integrated data set achieves higher accuracy and more specific inferences that might be obtained by the use of single sensor alone. In general, data registration is a critical preprocessing procedure in all remote-sensing applications that utilizes multiple sensor inputs, including multi-sensor data fusion, temporal change

Fig. 4. Flow chart scheme of the LiDAR data surface analysis

between on- and off-terrain points is applicable.

2003; Hinz & Baumgartner 2003; Cloude et al., 2004).

**2.4 Data registration: Automatic and manual approaches**

method for automatic registration and data fusion is needed.

Fig. 5. Flow chart scheme of the input data and registration techniques

detection, and data mosaicking (Moigne et al., 2002). In manual registration, the selection of control points (CPs) is usually performed by a human operator. This has proven to be inaccurate, time-consuming, and unfeasible due to data complexity, which makes it cumbersome or even impossible for the human eye to discern the suitable CPs. Therefore, researchers have focused on automating feature detection to align two or more data sets with no need for human intervention.

The automatic registration of data sets has generated extensive research interest in the fields of computer vision, medical imaging and remote sensing. Comprehensive reviews have been published by Brown (1992) and Zitova and Flusser (2003). Many proposed schemes for automatic registration employ a multi-resolution process (Viola and Wells 1997, Wu & Chung 2004, Fan et al., 2005, Zavorin & Moigne 2005, Xu & Chen 2007).

The existing automatic data-registration techniques based on spatial information fall into two categories: intensity-based and feature-based (Zitova and Flusser 2003). The featurebased technique extracts salient structures from sensed and reference data sets by accurate feature detectin and by the overlap criterion. As the relevant objects of intereset (e.g., roofs) and lines (e.g., roads) are expected to be stable in time at a fixed position, the feature-based method is more suitable for multi-sensor and multi-data set fusion, change detection and mosaicking. The method generally consists of four steps (Jensen, 2004): (1) CP extraction; (2) transformation-model determination; (3) image transformation and resampling, and (4) assessment of registration accuracy. The first step is the most complex, and its success essentially determines registration accuracy. Thus, the detection method should be able to detect the same features in all projections and in different data, regardless of the particular image/sensor/data type deformation. Despite the achieved performance, the existing methods operate directly on gray intensity values and hence they are not suited for handling multi-sensor and multi-type data sets.

The suggested algorithm is an adapted version of the four stages AIRTop (Figure 6) algorithm (Brook & Ben-Dor 2011c). First, the significant features are extracted from all input data sets and converted to a vector format. Since the studied scene has a large area, regions of interest (ROI) with relatively large variations are selection. The idea of addressing the registration problem by applying a global-to-local level strategy (the whole image is now divided into regions of interest which are treated as an image) proves to be an elegant way

Fusion of Optical and Thermal Imagery and LiDAR Data for

pixels size and location).

**3. 3-D urban environment model**

D scenes.

2 http://www.citygml.org/

up geometries, which in turn composes layers.

Application to 3-D Urban Environment and Structure Monitoring 41

features of the sensed goereferenced LiDAR data according to a non georeferenced imagery in order to reserve original raw geometry, dimensionality and imagery matrices (imagery

In the proposed 3-D urban application, the manual registration is used to register facades imagery and thematic mapping acquired by ground sensors and simplify buildings model extracted from LiDAR. This method is executed by a human operator, who identifies a set of corresponding CPs from the images and referenced control building model. Despite the fact that manual registration has been proven inaccurate and time-consuming due to data complexity, this method is still the most widely used technique. We found that for the

The urban database-driven 3-D model represents a realistic illustration of the environment that can be regularly updated with attribute details and sensor-based information. The spatial data model is a hierarchical structure (Figure 7), consisting of elements, which make

Level 1- "City 3D"

Level 2- "Building Model"

Level 3- "Spectral Model"

A fundamental demand in non-traditional, multi-sensors and multi-type applications is spatial indexing. A spatial index, which is a logical index, provides a mechanism to limit searches based on spatial criteria (such as intersection and containment). Due to the variation of data formats and types, it is difficult to satisfy the frequent updating and

An R-tree index is implemented on spatial data by using Oracle's extensible indexing framework (Song et al. 2009). This index approximates the geometry with a single rectangle that minimally encloses the geometry (minimum bounding rectangle MBR). A bounding volume is created around the 3-D object, which equals the bounding volume around the solid. The index is helpful in conducting very fast searches and spatial analyses over large 3-

CityGML2 is an application based on OGC's (open geospatial consortium) GML 3.1. This application not only represents the graphical appearance but in particular, it takes care of the semantic properties (Kolbe et al. 2005), such as the spectral/thematic properties, and model evaluations. The main advantage is the ability to maintain different levels of detail (Kolbe & Bacharach 2006). The underlying model differentiates three levels of detail, for

current data sets, manual registration is the easiest and most accurate solution.

Fig. 7. The 3-D urban environment application's conceptual architecture

extension requirements for developing urban environments.

which objects become more detailed as the level incise.

Fig. 6. A flow chart describing the registration algorithm. Blue box: topology map matching. Orange box: matching process. Green box: validation and accuracy.

of speeding up the whole process, while enhancing the accuracy of the registration procedure (Chantous et al. 2009). Thus, we expected this method to greatly reduce false alarms in the subsequent feature extraction and CP identification steps (Brook et al., 2011). To select the distinct areas in the vector data sets, a map of extracted features is divided into adjacent small blocks (10% × 10% of original image pixels with no overlap between blocks). Then, the significant CPs extraction has been performed by applying the SURF algorithm (Brown & Lowe, 2002). First the fast-Harris corners Detector (Lindeberg, 2004), which based on an integral image, was performed. The Hessian matrix is responsible for primary image rotation using principal points that identified as "interesting" potential CPs in the block. The local feature representing vector is made by combination of Haar wavelet response. The values of dominant directions are defined in relation to the principal point. As the number of interesting points tracked within the block is more than the predefined threshold, the block is selected and considered a suitable candidate for CPs detection.

The spatial distribution and relationship of these features are expressed by topology rules (one-to-one) and they are converted to potential CPs by determining a transformation model between sensed and reference data sets. The defined rules for a weight-based topological map-matching (tMM) algorithm manage (Velaga et al. 2009), transform and resample features of the sensed goereferenced LiDAR data according to a non georeferenced imagery in order to reserve original raw geometry, dimensionality and imagery matrices (imagery pixels size and location).

In the proposed 3-D urban application, the manual registration is used to register facades imagery and thematic mapping acquired by ground sensors and simplify buildings model extracted from LiDAR. This method is executed by a human operator, who identifies a set of corresponding CPs from the images and referenced control building model. Despite the fact that manual registration has been proven inaccurate and time-consuming due to data complexity, this method is still the most widely used technique. We found that for the current data sets, manual registration is the easiest and most accurate solution.

## **3. 3-D urban environment model**

40 Remote Sensing – Advanced Techniques and Platforms

**SURF Algorithm**

Preliminary Detection

**Loop:**

Calculation of each element in pairs

**Weights Hierarchy**

No

**Fast-Harris**  Corner detection

Yes

**Haar Wavelet Response** Region orientation

**Collect CPs**

**Test Point Error (TPE) Threshold**

**Complete Algorithm**

Yes

Integral Image

**Unpacking (**Topology match checking**):**  -Feature composition relationship

No Yes **Determine**

**CP Pair**

**RMS Threshold**

**Execute Map-Matching** Transform

Hassian Approximation

Fig. 6. A flow chart describing the registration algorithm. Blue box: topology map matching.

No

of speeding up the whole process, while enhancing the accuracy of the registration procedure (Chantous et al. 2009). Thus, we expected this method to greatly reduce false alarms in the subsequent feature extraction and CP identification steps (Brook et al., 2011). To select the distinct areas in the vector data sets, a map of extracted features is divided into adjacent small blocks (10% × 10% of original image pixels with no overlap between blocks). Then, the significant CPs extraction has been performed by applying the SURF algorithm (Brown & Lowe, 2002). First the fast-Harris corners Detector (Lindeberg, 2004), which based on an integral image, was performed. The Hessian matrix is responsible for primary image rotation using principal points that identified as "interesting" potential CPs in the block. The local feature representing vector is made by combination of Haar wavelet response. The values of dominant directions are defined in relation to the principal point. As the number of interesting points tracked within the block is more than the predefined threshold, the

The spatial distribution and relationship of these features are expressed by topology rules (one-to-one) and they are converted to potential CPs by determining a transformation model between sensed and reference data sets. The defined rules for a weight-based topological map-matching (tMM) algorithm manage (Velaga et al. 2009), transform and resample

Orange box: matching process. Green box: validation and accuracy.

**Initialization:**  -Insert features -Topology building

Primary Detection

**Automatic Registration - Control Point Detection**

> **Matching:**  -Linear network -Transfer match

**Weight-based Topological Map-matching**

**Loop:**  Calculation of lower weight pair

block is selected and considered a suitable candidate for CPs detection.

The urban database-driven 3-D model represents a realistic illustration of the environment that can be regularly updated with attribute details and sensor-based information. The spatial data model is a hierarchical structure (Figure 7), consisting of elements, which make up geometries, which in turn composes layers.

Fig. 7. The 3-D urban environment application's conceptual architecture

A fundamental demand in non-traditional, multi-sensors and multi-type applications is spatial indexing. A spatial index, which is a logical index, provides a mechanism to limit searches based on spatial criteria (such as intersection and containment). Due to the variation of data formats and types, it is difficult to satisfy the frequent updating and extension requirements for developing urban environments.

An R-tree index is implemented on spatial data by using Oracle's extensible indexing framework (Song et al. 2009). This index approximates the geometry with a single rectangle that minimally encloses the geometry (minimum bounding rectangle MBR). A bounding volume is created around the 3-D object, which equals the bounding volume around the solid. The index is helpful in conducting very fast searches and spatial analyses over large 3- D scenes.

CityGML2 is an application based on OGC's (open geospatial consortium) GML 3.1. This application not only represents the graphical appearance but in particular, it takes care of the semantic properties (Kolbe et al. 2005), such as the spectral/thematic properties, and model evaluations. The main advantage is the ability to maintain different levels of detail (Kolbe & Bacharach 2006). The underlying model differentiates three levels of detail, for which objects become more detailed as the level incise.

<sup>2</sup> http://www.citygml.org/

Fusion of Optical and Thermal Imagery and LiDAR Data for

Data Fusion *Fuzzy Logic*

*Data integration*

Data source

Component /Operator

Data processing

Layer

Data source

Component /Operator

Data processing

Layer

Interface Browser CityGML *Downloadable application*

Application to 3-D Urban Environment and Structure Monitoring 43

Thermal map Thematic

Buildings model Ground imagery

Spatial selection

Interface Browser CityGML *Downloadable application*

**Level 2 – "Building Model"**

map

*3-D thematic analysis/interaction*

**Level 3 – "Spectral Model"**

Process Map

Spatial selection

*Fuzzy Rules*

Spatial selection

*Fuzzy Rules*

Material Map

Quality Map

Process Map

Fig. 9. The 3-D urban environment application – Level 2 (detailed architecture)

Digital Picture Measurements

Spectra model

Spatial selection

Fig. 10. The 3-D urban environment application – Level 3 (detailed architecture)

*2-D thematic analysis/interaction*

The 3-D monitoring built urban environment application, up to this point, employs single processing algorithms applied on imagery or LiDAR data, without taken into account contextual information. The data fusion application must provide fully integrated information, both of the classification products and the context within the scene. In the proposed application, a complete classification and identification task consist of subtasks, which have to operate on material and object characteristic/shape levels provided by accurately registered database. Moreover, the final fused and integrated application should be operated on objects of different sizes and scales, such as a single building detected within the urban area or a selected region on a building facade. The multi-scale and multi-sensor data fusion is possible with the eCognition procedure (user guide eCognition, 2003), when

**4. 3-D urban environment application**

the substructures are archived by a hierarchical network.

The 3-D urban application is based on an integrated data set: spectral models, ground camera and airborne images, and LiDAR data. The system requirements are defined to include geo-spatial planning information and one-to-one topology. The concept architecture diagram is presented in Figure 4. As the model consist visualization and interactivity with maps and 3-D scenes, the interface includes 3-D interaction, 2-D vertical and horizontal interactions and browsers that contain spectral/thematic temporal information. The 3-D urban application provides services such as thematic mapping, and a complete quantitative review of the building and it's surrounding with respect to temporal monitoring. The design of the application shows the possibilities of delivering integrated information and thus holistic views of whole urban environments in a freeze-frame view of the spatiotemporal domain.

The self-sufficient/self-determining levels of the integrated information contribute different parts to this global urban environmental application. The first level (Figure 8), termed "City 3-D", supplies three different products: 1) integrated imagery and LiDAR data, 2) 3-D thematic map, and 3) 2-D thematic map (which includes 3-D analysis layers such as terrain properties, spatial analysis, etc.).

Fig. 8. The 3-D urban environment application – Level 1 (detailed architecture)

The second level, termed "Building Model" (Figure 9), focuses on a single building in 3-D and provides two additional products: 1) integrated imagery and building model extracted from LiDAR data set, and 2) 3-D thematic map for general materials classification, and quantitative thematic maps implemented by spectral models.

The most specific and localized level is the third level, termed "Spectral Model" (Figure 9). The area of interest in this level is a particular place (a patch) on the wall of the building in question. The spatial investigation at this level is a continuation of the previous level; yet, the data source consists of spectral models that are evaluated for spectral in-situ point measurements. This level does not provide any integrated and rectified information, but provides geo-referencing of the results of the spectral models in realistic 3-D scale. This level completes the database of the suggested 3-D urban environment application.

The 3-D urban application is based on an integrated data set: spectral models, ground camera and airborne images, and LiDAR data. The system requirements are defined to include geo-spatial planning information and one-to-one topology. The concept architecture diagram is presented in Figure 4. As the model consist visualization and interactivity with maps and 3-D scenes, the interface includes 3-D interaction, 2-D vertical and horizontal interactions and browsers that contain spectral/thematic temporal information. The 3-D urban application provides services such as thematic mapping, and a complete quantitative review of the building and it's surrounding with respect to temporal monitoring. The design of the application shows the possibilities of delivering integrated information and thus holistic views of whole urban environments in a freeze-frame view of the spatiotemporal

The self-sufficient/self-determining levels of the integrated information contribute different parts to this global urban environmental application. The first level (Figure 8), termed "City 3-D", supplies three different products: 1) integrated imagery and LiDAR data, 2) 3-D thematic map, and 3) 2-D thematic map (which includes 3-D analysis layers such as terrain

**Level 1 – "City 3D"**

Thermal map

> Spatial selection

Thematic map

Spatial selection

*Fuzzy Rules*

*2-D thematic analysis/interaction*

Fig. 8. The 3-D urban environment application – Level 1 (detailed architecture)

quantitative thematic maps implemented by spectral models.

Interface Browser CityGML *Downloadable application*

DTM Buildings model

Data Fusion *Fuzzy Logic*

*Data integration*

The second level, termed "Building Model" (Figure 9), focuses on a single building in 3-D and provides two additional products: 1) integrated imagery and building model extracted from LiDAR data set, and 2) 3-D thematic map for general materials classification, and

Roads model

*Airborne* LiDAR *Airborne* Imagery

Spatial *surface*  analysis

*3-D Thematic analysis/interaction*

The most specific and localized level is the third level, termed "Spectral Model" (Figure 9). The area of interest in this level is a particular place (a patch) on the wall of the building in question. The spatial investigation at this level is a continuation of the previous level; yet, the data source consists of spectral models that are evaluated for spectral in-situ point measurements. This level does not provide any integrated and rectified information, but provides geo-referencing of the results of the spectral models in realistic 3-D scale. This level completes the database of the suggested 3-D urban

domain.

properties, spatial analysis, etc.).

Data source

Component /Operator

Data processing

Layer

environment application.

Fig. 9. The 3-D urban environment application – Level 2 (detailed architecture)

Fig. 10. The 3-D urban environment application – Level 3 (detailed architecture)

## **4. 3-D urban environment application**

The 3-D monitoring built urban environment application, up to this point, employs single processing algorithms applied on imagery or LiDAR data, without taken into account contextual information. The data fusion application must provide fully integrated information, both of the classification products and the context within the scene. In the proposed application, a complete classification and identification task consist of subtasks, which have to operate on material and object characteristic/shape levels provided by accurately registered database. Moreover, the final fused and integrated application should be operated on objects of different sizes and scales, such as a single building detected within the urban area or a selected region on a building facade. The multi-scale and multi-sensor data fusion is possible with the eCognition procedure (user guide eCognition, 2003), when the substructures are archived by a hierarchical network.

Fusion of Optical and Thermal Imagery and LiDAR Data for

produces highly accurate and reliable merged information.

sensor-based quantitative/thermal information and models.

purpose, both registration models and data fusion techniques were used.

better the data fusion.

**5. Discussion**

for data fusion and registration.

systems with multi-sensor sources.

Application to 3-D Urban Environment and Structure Monitoring 45

introduce expert knowledge and information into the logic, the better and detailed the description of the real world environment are modeled by the membership function, the

The operational system controls that a first class hierarchy will be loaded and used in the next step for data integration. Based on this preliminary fusion, first objects of interest are created from object primitives by thematic-based fusion. The same steps are performed until the final information (spectral quantitative model) is applied. The results are registered and integrated information is followed by the reliability map, which is established by the primary accuracy and classification confidence of each input data. The reliability map is important for post-processing inspection and testing routines; objects with low reliability must be assigned manually because no decision is possible. The suggested application involves semi-automatic or even manual stages, which have proven to be time-consuming operations. Yet, due to the expert system support, it is a time efficient application that

In this chapter, we present techniques for data fusion and data registration in several levels. Our study focused on the registration and the integration of multi-sensor and multitemporal information for a 3-D urban environment monitoring application. For that

The 3-D urban application satisfies a fundamental demand for non-traditional, multi-sensor and multi-type data. The frequent updating and extension requirement is replaced by integrating the variation in data formats and types for developing an urban environment. The main benefit of 3-D modeling and simulation over traditional 2-D mapping and analysis is a realistic illustration that can be regularly updated with attribute details and remote

The proposed application offers an advanced methodology by integrating information into a 5-D data set. The ability to include an accurate and realistic 3-D position, quantitative information, thermal properties and temporal changes provide a near-real-time monitoring system for photogrammetric and urban planning purposes. The main objectives of many studies are linked to, and rely on a historical set of remotely sensed imagery for quantitative assessment and spatial evolution of an urban environment (Jensen and Cowen 1999, Donnay et al. 2001, Herold et al. 2003, 2005). The well-known methodology is pattern observation in the spatiotemporal and spectral domains. The main objective of this research is a fully controlled, near-real-time, natural and realistic monitoring system for an urban environment. This task led us first to combine the image-processing and map-matching procedures, and then incorporate remote sensing and GIS tools into an integrative method

The proposed application for data fusion proved to be able to integrate several different types of data acquired from different sensors, and which are additionally dissimilar in rotation, translation, and possible scaling. The data fusion operated by fuzzy logic is a final product of the application. This approach is an important stage for quality assurance and validation but furthermore for information fusion in current and future remote sensing

The results of spectral/thermal classification processes are by far not only a spectral/thematic aggregation of classes converted to polygons or polylines (in vector format), but also a spatial and semantic structuring of the scene content (example of roofs extraction in Figure 11). The resulting network of extracted and identified objects can be seen as a spatial/semantic network of the scene. The local contextual information describes the joint relationships and meaningful interactions between those objects in the build urban environment and linked multi-scale and multi-sensor products. This hierarchy in the rulebase design allows a well-structured incorporation of knowledge.

• Concrete

Fig. 11. Hierarchical rule-base structure in eCognition

In fact, now each object is identified not only by its spectral, thermal, textural, morphological, topological and shape properties, but also by its unique information linkage and its actual neighbors. The data is fused by mutual dependencies within and between objects that create a semantic network of the scene. To assure high level accuracy and operational efficiency the input products are inspected by the basic topological rule, which obligates that object borders overlay borders of objects on the next layer. Therefore, the multi-scale information, which is represented concurrently, can be related to each other.