Terrain Classification with Uncertainty Suresh Lodha University of California, Santa Cruz

1
Terrain Classification with UncertaintySuresh
LodhaUniversity of California, Santa Cruz

• What is it building? trees? Are we sure?
• Where exactly is the building?

2
Aerial Lidar Data Classification and Building
Footprint Extraction

• Classification of Aerial LiDAR data into
  buildings, trees, roads, and grass using
• -- Expectation-Maximization (EM) algorithm
• -- AdaBoost algorithm
• -- Support Vector Machines (SVM)
• -- Confusion matrix/ Confidence
  computed/classified data visualized
• -- Uncertainty visualization brings out weak
  spots of algorithm/data

3
Sample Highlights 2000-2006

• Low uncertainty terrain compression/ registration
  algorithms
• Visualization of probabilistic moving Targets
  (with Varshney, Syracuse)
• Visualization of UCSC within VGIS
  (with Ribarsky, Georgia Tech)
• Pose uncertainty with lidar and camera
  (with Neumann, USC)
• Uncertainty with multiple and omnidirectional
  sensors (with Sturm, INRIA, France)
• Aerial lidar data classification with uncertainty
  (with Zakhor, UCB)

4
Low Uncertainty Compression/Registration
Algoriothms

• Topology Preserving Terrain Simplification
• Peaks/Pits/Ridges/Coastlinepreserved (2000-2001)
• Common Consistent Representation of Heterogeneous
  Geo-Spatial Temporal Data
• (Aerial LiDAR, DEMs, DOQQs, AutoCAD, Street Maps
  etc.) 2001-2002
• Hierarchical Registration of 3D Scenes

5
Uncertainty of Mobile Objects

• Probability-based modeling
• collaboration with Pramod Varshney (2000-01)
• Spatio-temporal GPS uncertainty
• 2001-2002
• Expert learning framework
• 2002-2003

6
Hierarchical Zooming into UCSC Campus
7
Multi-modal Integration of Targets and Terrains

• Integration of Santa Cruz data within VGIS
• (collaboration with Bill Ribarsky)
• Development of GPS-GIS infrastructure
• (ideas from Ulrich Neumann/Avideh Zakhor)
• Multi-modal communication (visual query/results,
  speech, geospatial database, wireless)
• Embedding of uncertain targets within terrains

8
Experiments Different cross-camera scenarios
9
Pose Estimation with Uncertainty
Aerial view
Stereo Based Registration

• Stereo based approach captures terrain undulations

LiDAR Based Registration

• LiDAR based approach seems better at turns.

10
Aerial Lidar Data Classification of UCSC Campus
using AdaBoost Classification is visualized with
and without Uncertainty
3 classes (roofs, trees, road-grass) using 3
features (height-derived features only) without
uncertainty with uncertainty
4 classes (roofs, trees, road, grass)
using 5 features (lidar and images) without
uncertainty with uncertainty





11
Aerial Lidar Data Classification and Building
Footprint Extraction

• Classification of Aerial LiDAR data into
  buildings, trees, roads, and grass using
• -- Expectation-Maximization (EM) algorithm
• -- AdaBoost algorithm
• -- Support Vector Machines (SVM)
• -- Confusion matrix/ Confidence
  computed/classified data visualized
• -- Uncertainty visualization brings out weak
  spots of algorithm/data

12
Abstract / Overview

• Goal automatically classify Lidar data using
  AdaBoost
• Use features derived from Lidar data and aerial
  imagery
• Also classify using only Lidar-based features

13
Data Acquisition

• Why Lidar?
• Cheap, accessible, good accuracy
• Collected in strips
• Fixed wing or helicopter
• .25 meter point spacing
• 15 cm vertical accuracy

14
Data Preparation and Processing
15
Overview of AdaBoost

• Uses well-known machine learning algorithm
  AdaBoost
• Combines many weak hypotheses to construct a
  general classifier
• Combining weak classifiers yields a more accurate
  general classifier

16
Data Preparation and ProcessingTraining Data
Created

• Ten subregions of 100,000 to 150,000 points used
  in training/testing
• 25 to 30 of regions manually labeled
• Presence of classes vary region to region

17
The AdaBoost Algorithm
18
The AdaBoost AlgorithmBinary AdaBoost

• Utilize probability distributions which describe
  chance of misprediction
• Adjust distributions based on previous accuracy
• Goal choose weak hypothesis with low probability
  of error as predicted by current distribution

19
The AdaBoost AlgorithmMulti-Class Extensions of
AdaBoost

• AdaBoost is limited to binary classification
• Several possible extensions toward multi-class
  classification
• AdaBoost.M1
• AdaBoost.M2 5
• Error-correcting output codes 3
• Also many methods for generating weak classifiers

20
The AdaBoost AlgorithmAdaBoost.M2

• Require training set of N examples, x1,y1
  xN, yN
• Example i is a feature vector xi paired with
  class label yi 2 Y (Y1,2,3,4)
• Weak hypothesis h(x) maps measurements xi into
  (p1,p2,p3,p4), pi 2 01
• pi represents confidence of i belonging to class
  yi

21
The AdaBoost AlgorithmAdaBoost.M2

• Dt is the distribution used by AdaBoost.M2 at
  iteration t
• Dt(xi,y) represents badness of assigning y2Y to
  xi (Dt(xi,yi) 0)
• Pseudo error now used to judge weak hypotheses,
  not absolute probability of misclassification
• Allows penalty of ½ for random guessing even when
  Y gt 2, where P(error) 1 - 1/Y
• Formally defined for weak hypothesis ht as

22
The AdaBoost AlgorithmBinary AdaBoost

• A generic iterative supervised learning algorithm
• Aggregates weak hypotheses into a master
  hypothesis using a weighted linear combination
• Requires only O(log N) weak hypothesis

23
The AdaBoost AlgorithmAdaBoost.M2
24
The AdaBoost AlgorithmGenerating Weak Hypotheses

• All-Pairs weak hypothesis generation
• Weak hypotheses discern between two classes using
  a single feature threshold
• Values threshold predict one of the pair with
  full confidence, otherwise predict remaining
  class with full confidence
• Information in Dt used to quickly find an optimal
  class pair, feature, and threshold

25
The AdaBoost AlgorithmGenerating Weak Hypotheses
26
Results and Analysis
27
Results and AnalysisMethods

• Two variations in training
• Class-weighted training
• Sample-weighted training
• Three variations in testing
• Leave-one-out
• Train-half-test-half
• Train-all-test-all
• Accuracy measured using labeled portions
• Test data sampled in same manner as training data
  (larger samples)

28
Results and AnalysisResults

• Average four-class accuracy gt 90
• Average three-class accuracy gt 93
• Type I error (four-class) highest in roads
• Type II error (four-class) highest in grass
• Most commonly confused classes
• Building-tree (four-class)
• Grass-road (four-class)
• Building-tree (three-class)

29
Results and AnalysisTabular Results
30
Results and AnalysisAnalysis of Results

• Accuracy stabilizes reasonably quickly
• Overall accuracy unaffected by sampling method
• Relative class-wise accuracy is dependent upon
  sampling
• For higher accuracy in specific class, relative
  increase in training data for that class useful

31
Results and AnalysisAccuracy vs. Iterations
Trained
32
Results and AnalysisFurther Analysis

• Key features in order of importance
• H, LRI, NV, I (four-class)
• H, NV (three-class)
• HV not critical to either
• Weak classifiers most important to four-way
  classification
• Height threshold of 3-8, 15, 70-100 for
  building-grass, building-road, and building-tree
  respectively
• Intensity threshold of 60-73 for grass-road
• Normal variation value of 50 for building-tree

33
Results and AnalysisRelative Feature Importance
34
Results and AnalysisRelative Threshold Importance
35
Results and AnalysisPrediction as a Function of
Feature Value
36
Results and AnalysisVisual Results College 8
37
Results and AnalysisVisual Results Entire
Campus
38
Results and AnalysisVisual Results Entire
Campus Close-up
39
Conclusion

• Not only achieve very accurate results
• Provide insight into which class pairs are most
  difficult to discern
• Provide insight into which features and
  thresholds are most important
• These would be imperative to
• Applying algorithm to new geographic regions
• Adding additional subclasses such as vehicles,
  telecommunication wires, etc.

40
A Bayesian Approach to Building Footprint
Extraction from Aerial Lidar Data
41
Automatic 3D Building Modeling Pipeline
42
Input
43
After AdaBoost classification and building
segmentation
44
AdaBoost Classification

• Point Based
• Combination of weak learners
• Features
• Height
• Height Variation
• Normal Variation
• Lidar return intensity

45
Segmentation Algorithm

• Region-growing segmentation
• Use Region-wide information to improve
  classification
• Multiple returns
• Edge Noise

Lidar Multiple Returns
46
Footprints

• Why?
• Used in modeling algorithms
• Give information about roof planes
• Simple modeling extend up
• General Idea
• Find boundary points
• Compute polygonal footprint approximation
• Optimize approximation based on building prior
  and observed data.
• Noisy boundaries!

47
Initial Footprint

• Edge points detected by neighborhood analysis
• Ordered using depth first search

48
Footprint Approximation

• Find possible wall locations
• Ideally one point in every corner
• Use min-e graph algorithm

49
Bayesian Optimization

• Goal Find the best footprint given building
  prior and observed data points
• P(XZ) P(X)P(ZX)
• X Current building footprint
• Z Observed boundary points
• Prior P(X) encodes belief in the likelihood of a
  footprint
• P(ZX) encodes how well the current footprint
  matches observed data

50
Bayesian Optimization

• Maximum a Posteriori (MAP) estimation
• Find footprint X from the data Z that maximizes
  P(X Z)
• P(ZX) defined in terms of the distance from the
  boundary points to the building walls

51
Prior

• P(X) contains knowledge about prior building
  footprints
• Defined in terms of local angles
• Preference for 90 (right angle) and 180 (straight
  line)

52
Bayesian Optimization

• Take negative log likelihood
• Solve by gradient descent
• Combination of two terms P(X), P(ZX)
• Simulated Annealing used during optimization
• s is a weighting on the penalty for how far from
  the walls the building points lie

53
Results
54
Tech Transfer, Transitions and Interactions

• INRIA
• Sarnoff
• NASA
• Raytheon

55
Collaborations

• Syracuse (Pramod Varshney et al.)
• probabilistic uncertain particle movement
• Georgia Tech (Bill Ribarsky et al.)
• integration of uncertainty within VGIS
• USC (Neumann et al.)
• aerial lidar data, GPS infrastructure, pose
  estimation with uncertainty
• UC, Berkeley (Zakhor et al.)
• sensor pose estimation with uncertainty
• aerial lidar data classification

56
Publications (1)

• Suresh K. Lodha, Edward J. Kreps, David P.
  Helmbold, and Darren Fitzptarick, "Aerial Lidar
  Data Classification using SVM", To appear in
  Proceedings of 3DPVT Conference, 2006.
• Suresh K. Lodha, Darren Fitzptrick, and David P.
  Helmbold,
• "Aerial Lidar Data Classification using
  AdaBoost", Manuscript, 2006.
• Oliver Wang, Suresh Lodha, and David P. Helmbold,
  A Bayesian Approach to Building Footprint
  Extraction from Aerial Lidar Data, To appear in
  Proceedings of 3DPVT Conference, 2006.
• Oliver Wang and Suresh K. Lodha, Automatic
  Segmentation of Buildings from Pre-Classified
  Aerial Lidar Data, Manuscript, 2006.

57
Publications (2)

• Srikumar Ramalingam, Suresh K. Lodha, and Peter
  Sturm, A Generic Structure-from-Motion
  Framework, submitted to Computer Vision and
  Image Understanding, January 2005.
• Kartik Venkatraman, Suresh K. Lodha, and Raghu
  Raghavan, A Kinematic-Variational Formulation
  for Animating Skin with Wrinkles", Computers and
  Graphics, Volume 29, No. 5, October 2005,
  pp.756--770.
• Lilly Spirkovska and Suresh K. Lodha,
  Context-aware intelligent assistant approach
  for decreasing pilot workload", Journal of
  Aerospace Computing, Information, and
  Communication, September 2005, Vol. 2. No. 9,
  pages 386--400.
• Peter Sturm, Srikumar Ramalingam, and Suresh K.
  Lodha, "On Calibration, Structure from Motion and
  Multi-View Geometry for Generic Camera Models",
  in Imaging Beyond the Pin-hole Model, K.
  Daniilidis, R. Klette, and A. Leonardis
  (editors), Kluwer Academic Publishers, 2005.

58
Publications (3)

• Suresh K. Lodha and Yongqin Xiao, "GSIFT
  Geometric Scale Invariant Feature Transform for
  Data Registration", Proceedings of the SPIE
  Conference on Vision Geometry XIV, Vol. 6066,
  pp. L1--L11, San Jose, CA, January 2006.
• Srikumar Ramalingam, Peter Sturm and Suresh K.
  Lodha, Theory and Calibration for Axial Cameras,
  Proceedings of the Asian Conference on Computer
  Vision (ACCV), Hyderabad, India, January 2006.
• Yongqin Xiao and Suresh K. Lodha, "Geometrically
  Invariant Feature Descriptors for Height Data
  Registration", Proceedings of the IVCNZ (Image
  and Vision Computing) Conference, pp.229--234,
  Dunedin, New Zealand, November 2005.
• Srikumar Ramalingam, Peter Sturm, and Suresh K.
  Lodha, "Towards Generic Self-Calibration of
  Cnetral Cameras", Proceedings of Omniviz workshop
  in ICCV, Beijing, China, October 2005.

59
Publications (4)

• Peter Sturm, Srikumar Ramalingam, and Suresh K.
  Lodha, "On Calibration, Structure-from-Motion and
  Multi-View Geometry for Panoramic Imaging
  Models", Proceedings of the 2nd ISPRS Panoramic
  Photogrammetry Workshop, Berlin, Germany, 2005.
• Srikumar Ramalingam, Peter Sturm, and Suresh K.
  Lodha, "Towards Complete Generic Camera
  Calibration", Proceeedings of Computer Vision and
  Pattern Recognition (CVPR), San Diego, CA, June
  2005, Vol. 1, pages 1093--1098.
• Karthik-Kumar Arun-Kumar and Suresh K. Lodha,
  Semi-Automatic Roof Reconstruction from Aerial
  Lidar Data using K-Means with Refined Seeding'',
  Proceedings of the ASPRS (American Society for
  Photogrammetry and Remote Sensing) Conference,
  Baltimore, Maryland, March 2005.
• Suresh K. Lodha, Andrew Ames, Adam Bickett, Jason
  Bane, and Hemanth Singamsetty, 3D Geospatial
  Visualization of UCSC Campus'', Proceedings of
  the ASPRS (American Society for Photogrammetry
  and Remote Sensing) Conference, Baltimore,
  Maryland, March 2005.

60
Publications (5)

• Sanjit Jhala and Suresh K. Lodha, Stereo and
  Lidar-Based Pose Estimation with Uncertainty for
  3D Reconstruction, Proceedings of Vision,
  Modeling, and Visualization Conference, Stanford,
  Palo Alto, CA, November 2004.
• Srikumar Ramalingam, Suresh K. Lodha, and Peter
  Sturm, Srikumar Ramalingam, Suresh K. Lodha, and
  Peter Sturm, A Generic Structure-from-Motion
  Algorithm for Cross-Camera Scenarios'',
  Proceedings of the OmniVis (Omnidirectional
  Vision, Camera Networks, and Non-Classical
  Cameras) Conference, Prague, Czech Republic, May
  2004.
• Amin Charaniya, Roberto Manduchi, and Suresh K.
  Lodha, Supervised Parametric Classification of
  Aerial Lidar Data, Proceedings of the IEEE
  workshop on Real-Time Sensors and Their Use,
  Washington DC, June 2004.
• Hemanth Singamsetty and Suresh K. Lodha, An
  Integrated Geospatial Data Acquisition System for
  Reconstructing 3D Environments, Proceedings of
  the IASTED Conference on Advances in Computer
  Science and Technology (ACST), St. Thomas, Virgin
  Islands, USA, November 2004.
• Sanjit Jhala and Suresh K. Lodha, On-line
  Learning of Motion Patterns using an Expert
  Learning Framework, Proceedings of the IEEE
  workshop on Learning in Computer Vision and
  Pattern Recognition, Washington DC, June 2004.

61
Publications (6)

• Suresh K. Lodha, Nikolai M. Faaland, and Jose
  Renteria, Hierarchical Toplogy Preserving
  Compression of 3D Vector Fields using Bintree and
  Triangular Quadtrees, IEEE Transactions on
  Visualization and Computer Graphics, Vol. 9, No.
  4, October 2003, pages 433442.
• Christopher Campbell, Michael M. Shafae, Suresh
  K. Lodha, and Dominic W. Massaro, Discriminating
  Visible Speech Tokens using Multi-Modality,
  Proceedings of the International Conference on
  Auditory Display (ICAD), Boston, MA, July 2003.
• Amin Charaniya and Suresh K. Lodha, Speech
  Interface for Geo-Spatial Visualization,
  Proceedings for the Conference on Computer
  Science and Technology (CST), Cancun, Mexico, May
  2003.
• Lilly Spirkovska and Suresh Lodha, Audio-Visual
  Situational Awareness for General Aviation
  Pilots'', Proceedings of the SPIE Conference on
  Visualization and Data Analysis, January 2003,
  Vol. 5009.
• Srikumar Ramalingam and Suresh K. Lodha,
  Adaptive Enhancement of 3D Scenes using
  Hierarchical Registration of Texture-Mapped
  Models, Proceedings of 3DIM 2003, October 2003.

62
Publications (7)

• Suresh K. Lodha, Nikolai M. Faaland, Grant Wong,
  Amin Charaniya, Srikumar Ramalingam, and Arthur
  Keller, "Consistent Visualization and Querying of
  Geospatial Databases by a Location-Aware Mobile
  Agent", Proceedings of the Computer Graphics
  International Conference 2003, Tokyo, Japan, July
  2003.
• Suresh K. Lodha, Krishna M. Roskin, and Jose C.
  Renteria, Hierarchical Topology Preserving
  Compression of Terrains", Visual Computer,
  September 2003.
• Amin Charaniya, Srikumar Ramalingam, Suresh
  Lodha, William Ribarsky, Nicholas Faust, Zach
  Wartell, and Tony Wasilewski, Real-Time
  Uncertainty Visualization of Mobile Objects
  within VGIS (Virtual Geographic Information
  System'', poster paper and interactive
  demonstration at the IEEE Visualization
  Conference, Boston , MA, October 2002.
• Srikumar Ramalingam, Nikolai Faaland, Amin
  Charaniya and Suresh Lodha, Visualization of
  Heterogeneous Geo-Spatial Intelligence in a
  Mobile Environment'', interactive demonstration
  at the IEEE Visualization Conference, Boston, MA,
  October 2002.
• Suresh K. Lodha, Nikolai M. Faaland, Amin P.
  Charaniya, Pramod Varshney, Kishan Mehrotra, and
  Chilukuri Mohan, "Uncertainty Visualization of
  Probabilistic Particle Movement", Proceedings of
  The IASTED Conference on KComputer Graphics and
  Imaging", August 2002, pages 226-232.

63
Publications (8)

• Lilly Spirkovska and Suresh Lodha, Audio-Visual
  Situational Awareness for General Aviation
  Pilots'', Proceedings of the SPIE Conference on
  Visualization and Data Analysis, January 2003,
  Vol. 5009.
• Suresh Lodha, A. P. Charaniya, Nikolai M.
  Faaland, and Srikumar Ramalingam,"Visualization
  of Spatio-Temporal GPS Uncertainty within a GIS
  Environment" Proceedings of SPIE Conference on
  Aerospace/Defense Sensing, Simulation, and
  Controls, April 2002.
• Suresh K. Lodha, Amin P. Charaniya, and Nikolai
  M.Faaland, "Visualization of GPS Uncertainty in
  a GIS Environment", Technical Report
  UCSC-CRP-02-22,University of California, Santa
  Cruz, April 2002, pages 1-100.
• Suresh K. Lodha, Nikolai M. Faaland, Grant Wong,
  Amin Charaniya,Srikumar Ramalingam, and Arthur
  Keller, "Consistent Visualization and Querying of
  Geospatial Databases by a Location-Aware Mobile
  Agent, ACM GIS Conference, November 2002.
• S.Lodha, N.Faaland, A.Charaniya, P.Varshney,
  K.Mehrotra and C.Mohan. Visualization of
  Uncertain particle movement. Proceedings of the
  Computer Graphics and Imaging Conference, pages
  226-232, 2001.
• Lilly Spirkovska and Suresh K. Lodha,AWE
  Aviation Weather Data Visualization
  Environment'', Computers and Graphics, Volume
  26, No.1, February 2002, pp.169--191.
  NASA/TM-2000-209617, December 2000.

64
People

• Graduate students
• Karthik-Kumar Arun-Kumar
• Amin Charaniya
• Alex DAngelo
• Darren Fitzpatrick
• Sanjit Jhala
• Edward Kreps
• Srikumar Ramalingam
• Jose Renteria
• Krishna Roskin
• Hemanth Singamsetty
• Shailaja Vats
• Oliver Wang
• Yongqin Xiao
• Undergraduate students
• Andrew Ames
• Jason Bane
• Adam Bickett
• Nikolai Faaland

65
Acknowledgements

• MURI Grant
• Airborne1 Corporation
• NSF

66
Thank You!