Enrichment and multi-purpose visualization of building models with emphasis on thermal sensor data
Under taken by:
- Technical University of Munich
Institute of Cartography
-
Prof. Dr.-Ing. Liqiu Meng
CV (PDF)
- Technical University of Munich
Institute of Photogrammetry and Remote Sensing
-
Prof. Dr.-Ing. Uwe Stilla
CV (PDF) 
Summary
The proposal addresses the aspects of enrichment, updating, and visualization of 3D building models. It serves four principal goals: (1) to integrate the cadastral information with the 3D geometries of individual buildings by means of data matching, (2) to extract and interpret non-visible features from thermal infrared data based on image analysis coupled with context information from 3D building models, (3) to update as well as enrich the 3D building models with the information derived from non-visible features, and (4) to visualize semantic attrib-utes describing various surface elements of updated and enriched 3D building models with emphasis on time-dependent thermal information. The involved tasks will be jointly accom-plished by the Department of Cartography and the Department of Photogrammetry and Re-mote Sensing. The data sources from a selected test urban area include: (1) polyhedral de-scriptions of individual buildings, (2) a number of building attributes embedded in city plans, cadastral maps and real estate databases, (3) thermal infrared images.
The novelties of this proposal are reflected in two aspects:
- It develops an iterative strategy to value-add 3D building models by using actual sensor data from non-visible domain (i.e. bottom-up approach) and reusing the existing geoinformation (i.e. top-down approach). The building attributes from available large-scale maps and databases help to enhance the certainty and efficiency of feature detection from the thermal infrared data, whereas the detected features support the quality assurance of the existing 3D building information.
- It develops visualization techniques that are guided by users intuition and go beyond the image draping or photorealistic rendering of building appearance. Based on the case study on multi-purpose visualization of time-dependent thermal information and its analytical func-tions, cartographic design theories for 2D interactive maps are extended to cope with the arte-facts in 3D such as egocentric perspectives, occlusions, distance distortion and the associated problem of thematic overlay.
Goals
The work envisaged in this proposal is characterized by a processing chain along which scien-tific methods from cartography, GIS and image analysis interplay. It attempts to narrow two research gaps in current 3D urban GIS:
- Unavailability or inaccessibility of semantic information, especially the temporal behaviors of individual 3D urban objects;
- Little effort concerned with the visualization of meanings rather than appearances of 3D urban objects and the necessary user interactions.
In the field of enrichment of 3D building data, the previous work has shown that building contours of a 3D model can be assigned to corresponding image structures of infrared video sequences. However, the extraction of image textures rich of thermal information, has not yet been realized due to missing context information. Following the dual strategy of GIS-guided feature extraction from image data, and image-guided quality assurance of GIS data, we will make optimal use of the existing large-scale 2D maps and databases to extract maximal information from the infrared images. The available polyhedral building models from a selected urban area serve as the spatial references and carriers of the information from both 2D GIS and the infrared images. The involved research tasks will answer questions such as:
- How can the existing 2D map symbols and database objects be connected with the 3D building geometries?
- How far can the thermal behaviors of building surfaces be simulated based on the incomplete information about the building materials, fagade textures and heating conditions?
- How are the structures of thermal features influenced by the relative orientation and position of the underlying surface object to the sensor? What kind of geometric and topological relationships exist between the optically visible surface elements and thermal features?
- How can the new detected geometric features and time-dependent attribute information be modeled and embedded in the individual building surface elements?
- How can multiple non-visible attributes including the temporal behaviors of building surfaces be simultaneously mapped onto visual metaphors?
- How can thematic overlay and hiding be realized on-demand in 3D?
- How can occluded objects be directly located by pointing devices rather than text query?
- How far can abstract cartographic symbols including labels be meaningfully embed-ded in 3D perspective views?
- What kind of typical user habits from 2D GIS should be considered as design con-straints of 3D operations?
- Simulation of 3D scene composed of polyhedral descriptions of buildings, semantic at-tributes and an illumination model based on true physical conditions,
- An in-depth study on the characteristics, especially the temporal properties, of the visibil-ity based on thermal emissions in comparison to the visibility conditioned by the field of view,
- Feasible matching methods that connect IR images with the simulated scene,
- Feasible methods for the extraction and interpretation of non-visible features,
- Modeling rules for information types acquired from IR,
- Generic operations for the integration of new information with the 3D surface elements,
- Concept for the interactive 3D visualization techniques of non-visible features
The know-how to be developed in our project can be transferred to the enrichment of other urban object types with information from other non-visible domains.
Work schedule
There are eight Work Packages (WP) involved in the proposal as illustrated in Fig.1. The De-partment of Cartography (Meng) is mainly responsible for WP 1, 6, 7, 8, while the Depart-ment of Photogrammetry and Remote Sensing (Stilla) for WP 3,4,5. WP 2 will be jointly ac-complished by both departments.
Figure 1: Dataflow and involved tasks
Scene simulation (WP 1)
This WP needs the following data sources: (a) a set of polyhedral 3D building models cover-ing at least one settlement block of the test region, (b) the corresponding 2D GIS containing large-scale vector maps (city plan and cadastral maps at =1:5,000) and real estate cadastral database, and (c) the corresponding image data including sensor parameters. A generic match-ing method (some modules exist at the Department of Cartography) will be developed to inte-grate these data sources, whereby source (a) is used as reference. The building objects from different sources are linked if they indicate the approximately the same location and ground plan. With the help of the linkage, a number of relevant attributes such as building name, house number, function, the number of stories, total height, roof type including ridge lines and eave lines, building materials for roof coverage and walls, surface material on fagade, year of construction, and condition can be transferred from 2D GIS and image data to the correspond-ing polyhedral models. Further, an illumination model will be constructed and applied to the semantically enriched polyhedral buildings. Different from illumination conditions used in computer graphics for the purpose of enhancing 3D sensation, we will simulate the true physical situation with a light source derived from the altitude of the sun at the time when the image data were acquired and at some other times with the consideration of seasonal and noc-turnal influences (cf. Ramos, Siret and Musy 2004). The lighting or shading of the building surfaces and their shadow areas, however, are at this stage only dependent on the reflection and refraction characteristics of the known surface materials.
The result is a simulated 3D scene containing a part of geometric, structural, semantic and physical information of the true situation.
Visibility analysis (WP 2)
The simulated scene resulted from WP 1 will be graphically rendered with the egocentric viewing parameters (e.g. viewing time, viewing distance, viewing height, angle of inclination, and beam width) derived from the known sensor geometry, the position and orientation of the platform (e.g. from GPS and INS measurements). Visible points, edges and faces are then calculated following two different principles:
(a) Principle based on field of vision
The visibility of surface structures, e.g. points, edges and faces, of the individual buildings that are exposed to the visible electromagnetic spectrum and fall within the vision field of the sensor is detected with algorithms available in 3D graphic software such as scan-line render-ing and Z-buffering etc. In addition, the visibility degree of each visible element will be cal-culated in relation to the resolution and viewing distance of the applied sensor. The shading of visible surface structures and visible shadow boundaries are rendered as well.
(b) Principle based on material properties
Being aware of the fact that the visibility of building surfaces in an infrared image depends mainly on the thermal emissions of surface materials, we will infer the thermal behavior of 3D building models based on the available information about building materials, orientations and distances to existing thermal sources including the intensity of the incident light rays. The visible surface structures including shading and shadow boundaries associated with the ther-mal behavior will then be detected.
WP 2 results in two differently visible 3D building models for the same set of viewing pa-rameters.
Projection (WP 3)
(a) 3D to 2D projection
The visible faces of the building model has to be projected into the two-dimentional representation using the initial pose estimation obtained by GPS and INS. This results in a 2D to 2D matching problem which is less fragile to false extracted structures than a 3D to 2D matching problem. The projected model data and image data form a combined data set and has to be matched for a correct assignment.
(b) Pose estimation
In any case, the problem of 3D model to 2D image registration is not yet satisfactory solved for oblique images since the exterior orientation for the camera is only roughly known by using GPS and INS at the beginning. Furthermore, for each frame of a video sequence the position of the camera must be estimated. In general, GPS and INS information is not available for each frame. Thus, the trajectory has to be estimated by Kalman filtering and the the exterior orientation has to be interpolated. Knowing corresponding points after the matching allows to recalculate the exterior orientation and to project again the model structures into the image domain. In general this proceedure can be carried out iteratively. This problem is commonly referred to as a simultaneous position and correspondence problem.
The result of the projection is a transformation of the three-dimensional building structures into the same geometry as the image data
Matching (WP 4)
During the matching process image structures have to be analysed, extracted and matched with the projected building structures.
In practice the projected building vectors do not exactly coincide with their image location. Images which are taken with large focal lengths often suffer from severe misalignment for very small errors of the INS data. To overcome this problem, an automated matching approach of image structures and model structures is required. Some general discrepancies between the 3D model and the 2D image remain in any case: due to the limited resolution and various model simplifications the location of 3D model edges may be inaccurate.
The first step is to select suitable features and a robust feature extraction. In optical images we come across with strong edges of objects and edges by shadow. In the case of infrared data the edges of objects look smoother and areas show less details. Well-known edge detectors for optical images like Cannys edge show not the expected performance. Thus, different feature extraction methods have to be investigated in the context of infrared imagery. Additionally, to edge-based operators area-based operators should be investigated. The overall shape of a building face derived from the 3D model can be used instead of single features. This promises more robustness against clutter of the scene.
After this basic feature extraction step more complex structures (e.g. L-shaped structures, areas) have to be assembled and significant features (e.g. the intersection, center of gravity) have to be calculated. From the set of calculated features a subset has to be selected for limiting the computation time of the matching process. To cope with matching errors, existing solutions often use a probabilistic hypothesize-and-test approach such as RANSAC.
At the end of the matching and pose estimation process image segments are assigned to the model faces.
Feature Extraction (WP 5)
In this workpackage the assigned faces are investigated. Depending on the homogenity of the radiance of the face two further proceedures are foreseen. If a face is homogen, features to enrich the database are calculated for each face (a). If a face is not homogen, a search for new geometric features will be carried out (b).
(a) Assigned features
Depending on the time of the day two situations has to be distinguished which leads to different appearances of objects in thermal images. During nighttime an object appears according to its own heat emission. At daytime additionally to the emission of the object the thermal reflection of the sunlight has to be considered. This can result in a reverse contrast (cross-over). Furthermore, objects will warm up, if they are illuminated over the day. Thus, for determination of typical surface features (e.g. albedo, angular reflection, surface temperature) a set of images has to be taken at different time (day and night). This requires the development of algorithms in consideration of the time of acquisition and the variations of radiance.
For surface feature calculation first each face will be analysed with respect to possible shadow areas. This information can be derived from the simulated scene analysis (WP1) considering sun light direction, lighting duration, and shadows from adjacent objects. In case of a partial coverage by shadow the assigned face must be splitted into areas with and without shadow.
For calculation of the surface temperature the radiance of a pixel, the angular reflectance, and emission are needed. The intensity of the reflection itself depends on the orientation of the face to the sensor, which has to be taken into account.
For airborne applications it is necessary to correct the emission of the atmosphere. Everything above absoute 00K (-2730C) emits thermal energy. The emission of the optical path, the length between the ground and the airplane through the atmosphere, has to be modeled out to permit an accurate analyses of thermal energy responses from thermal infrared data.
The thermal ground reflectance has two major properties, namely, reflection albedo variations related to surface roughness and wavelength-dependent material variations. When the existing database is enriched with information about surface materials, this information will be used on the one hand to consider the material dependent angular reflectance, as different materials e.g., glass, aluminium, metal or plaster show different angular reflections. On the other hand the potential of thermal infrared data for mapping surface materials will be investigated.
(b) New geometric data
In the case that the face has inhomogeneities apart from shadow, building structures are visible which are not present in the model data. This typically occurs in ground-based images when additional structures e.g. windows, doors are present. Also, thermal emissions appar-ently change due to various factors such as material differences, different ages (therefore, dif-ferent insulation properties) of the same material, differently heated surface of the same mate-rial. Such structures can be realized because of different reflection properties. Those individual thermal building structures can be best observed by night vision where we have no influence from the sun radiation. Within the fagade structures are segmented which excess a minimum size. By an approximation of the contours with linear features we test wheather the segments are regular or not. . Image series taken over time can be used to study the cooling process and gives hints to different materials (e.g. roof slate, roofing tile).
Those building inherent features are valuable for further enrichment of the database and simulation purposes
This WP delivers thermal features and new geometric structures.
Integration (WP 6)
The thermal features and new geometric structures detected from infrared images are to be integrated into the existing database. They reveal following possible relationships with opti-cally perceivable faces: (a) they coincide with each other, (b) they overlap partly, or (c) they contain each other. Information that can be derived from these thermal features include: time-independent attributes (e.g. building materials), time-dependent behaviors (e.g. temperature distribution at a certain moment), up-to-date geometric elements (e.g. the locations of win-dows and doors), or new geometric elements that do not yet exist in the simulated scene (e.g. stairways, solar devices installed on fagade).
Modeling rules will be established with the aim to enrich the data structure of the simulated scene that should accommodate the new information. Existing 3D operations (e.g. splitting, merging, inserting, linking etc.) will be applied and refined for different integration cases concerned with (a) new surface elements, (b) up-to-date surface elements, (c) time-independent attributes with uniform values covering 1 face, and (d) time-dependent behaviors related to points, corners, edges or faces.
This WP yields new structured objects with geometric attributes between 0 and 3 dimensions, semantic attributes and linked thermal events.
Visualization (WP 7)
Unlike rendering the simulated scene for visibility study by researchers involved in this pro-ject, here visualization is treated as a value-adding process for the end users who have differ-ent knowledge profiles. Not only the geometric structures and appearances of the enriched and updated 3D buildings are concerned, but the semantic attributes and thermal behaviors of the individual surface elements will be mapped to visual variables, especially color, texture and transparency. Bearing the purpose of visual communication and exploration in mind, we will go beyond the image-draping or photorealistic rendering and focus on non-photorealistic techniques. Cartographic abstractions will be introduced into the design rules with the aim to improve the legibility, enhance the awareness of the embedded information while preserving the expressiveness of 3D graphics. Concretely, visualization widgets will be conceptualized for time-independent attributes and time-dependent behaviors associated with points, edges, faces and polyhedral elements (cf. Nienhaus and Dvllner 2003). In other words, the geometric primitives from 0-3 dimensions will be overlaid with thematic information. Main constraints for the graphic as well as info-graphic design are perceptual characteristics, knowledge pro-files and tasks of selected user groups. To facilitate the visual estimation of spatial relation-ships, parallel perspective projections (e.g. Cavalier projection, Cabinet projection) will be investigated as alternative solutions to central perspective view.
This WP yields multi-purpose presentations of spatial and temporal distribution of thermal information.
Interaction (WP 8)
This WP is envisaged to support the usability (efficiency, effectiveness and user satisfaction) of visualization results. The most challenging work here lies in making the individual surface elements with 0 to 3 spatial dimensions accessible for queries and direction manipulation. A modeling mechanism that allows the integration of structural information with building sur-faces is a prerequisite. The emphasis is put on the design of analytical and computing func-tions for the end users, not the geometric editing functions. Main design constraints are intui-tion of users, their habit gained from 2D interaction, and real-time reaction speed. A number of interactive strategies such as synchronized highlighting (e.g. blinking, brushing) in multiple windows, information hiding, selective labeling, text-based query, and information box that have been developed for 2D GIS will be adapted to 3D applications. In addition, new algo-rithms for two fundamental functions for 3D interaction are envisaged: direct selection of hidden elements (e.g. by using transparent symbols, calling depth buffer, or activating syn-chronized 2D graphics), and 3D buffering (e.g. based on the dilation of convex hulls). Exist-ing design concepts reported in literature (cf. Rahman and Khuan 2003, Shneiderman and Plaisant 2004) will be used as guidance and reference.
The result is an interactive prototype information system of 3D buildings.
