TREE STEM RECONSTRUCTION USING VERTICAL FISHEYE IMAGES: A PRELIMINARY STUDY A. Berveglieri a,* and A. M. G. Tommaselli a UNESP, Universidade Estadual Paulista, Department of Cartography, Rua Roberto Simonsen 305, Presidente Prudente, Brazil - adilsonberveg@gmail.com, tomaseli@fct.unesp.br Commission V, WG V/4 KEY WORDS: Dense Reconstruction, Fisheye, Image Rectification, Photogrammetry ABSTRACT: A preliminary study was conducted to assess a tree stem reconstruction technique with panoramic images taken with fisheye lenses. The concept is similar to the Structure from Motion (SfM) technique, but the acquisition and data preparation rely on fisheye cameras to generate a vertical image sequence with height variations of the camera station. Each vertical image is rectified to four vertical planes, producing horizontal lateral views. The stems in the lateral view are rectified to the same scale in the image sequence to facilitate image matching. Using bundle adjustment, the stems are reconstructed, enabling later measurement and extraction of several attributes. The 3D reconstruction was performed with the proposed technique and compared with SfM. The preliminary results showed that the stems were correctly reconstructed by using the lateral virtual images generated from the vertical fisheye images and with the advantage of using fewer images and taken from one single station. * Corresponding author 1. INTRODUCTION Tree stem reconstruction from optical images is a topic of growing interest, but usually dozen of images from different viewpoints are used, mainly when using techniques like Structure from Motion (SfM). The computation of SfM is based on image correspondences over images acquired with large overlap, being nowadays one of the most popular image-based 3D modelling algorithms. A collection of SfM applications in geosciences was reported by Westoby et al. (2012). Although stereo-photogrammetry has a long history, the automatic 3D reconstruction using point clouds extracted from multiple images was limited by the computational load and problems with image matching. In recent years, the significant advance of digital cameras and dense matching techniques has made image-based point cloud an important data source for tree measurements (Liang et al. 2015). Thus, several studies have been proposed using terrestrial optical images in forest applications. Typically, ground surveys are conducted in forest sample plots to collect variables as diameter at breast height (DBH) and tree positions, as well as to identify types of species and environmental conditions. Hapca et al. (2007) used two digital images taken from two stations with convergence defining an intersection angle of 90° to reconstruct the 3D shape of standing trees. Herrera et al. (2011) presented a fisheye stereovision method using image segmentation and image matching to separate textures of interest in forest environments and generating disparity maps of tree trunks with their approach. Liang et al. (2014) presented a work assessing the potentiality of point clouds generated using an uncalibrated hand-held camera at a forest plot. Individual trees stems were detected and modelled from the point cloud. Forsman et al. (2016) introduced a prototype of a multi-camera rig to collect images from a single station at the plot centre. Tree stem attributes were estimated by photogrammetric techniques. However, this technique used a complex system with five cameras that had to be accurately calibrated and synchronize to obtain reliable measures. The previously reported approaches used several images to extract tree data or reconstruct stems, which make them time- consuming techniques. In this paper, an alternative is proposed using fisheye cameras with vertical displacement, which reduces significantly the field work because fisheye lenses increase the possibility of collecting images with a large ground coverage area. Terrestrial laser scanning (TLS) is a technology that has been widely used for forest measurements. However, according to Forsman et al. (2016), when TLS is compared with photogrammetric techniques, camera-based methods offer potentially less expensive hardware, increase mobility and reduce time on-site. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B5, 2016 XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic This contribution has been peer-reviewed. doi:10.5194/isprsarchives-XLI-B5-627-2016 627 The main objective of this preliminary study is to investigate the feasibility of the technique proposed for stem reconstruction, which enables later measurement and extraction of several attributes from the reconstructed stem model. In this technique, the original vertical panoramic image acquired with fisheye camera is transformed to horizontal lateral images following conventional perspective geometry. Thus, photogrammetric procedures can be performed with ordinary software based on collinearity equations. 2. MATERIALS AND METHODS 2.1 Image data and study area A camera with fisheye lens is used to collect panoramic terrestrial images of tree stems inside forests. This type of image enables to acquire a large field of view, which reduces the number of images needed to generate the 3D reconstruction. The image acquisition system and the processing technique for forest areas were originally presented by Tommaselli and Berveglieri (2014) and Berveglieri et al. (2014) and it was adapted to be applied with this reconstruction technique. A telescopic pole attached to the fisheye camera in nadir viewing is used to collect vertical images among trees, as shown in Figures 1(a,b). (a) (b) Figure 1. (a) Fisheye system positioned among trees. (b) Acquisition of multiples vertical images with height variation. The camera with fisheye lenses has to be previously calibrated to estimate the Interior Orientation Parameters (IOPs): focal length (f), principal point (x0, y0) and lens distortion coefficients (K1, K2, K3, P1, P2). A mathematical model suitable for fisheye geometry (e.g., equidistant model – Schneider et al. (2009); Marcato Junior et al. (2015)) is used with addition the Conrady-Brown model (Fryer and Brown 1986). A bundle adjustment is performed using constraints imposed to the ground coordinates, observations and Exterior Orientation Parameters (EOPs) as described by Tommaselli and Berveglieri (2014) and Berveglieri et al. (2014). The study area is located in Masala (60.15°N, 24.53°E), southern Finland, where Scots pines (Pinus sylvestris L.) in mature stage are found. Such pines, common in Scandinavian regions, have features that are suitable for applying the proposed technique. 2.2 Tree stem reconstruction technique Firstly, the fisheye camera is positioned and leveled is suitable place inside the forest trying to minimize occlusions from that view point. A vertical image sequence is acquired using small variations in heights ranging from 4 m to 5 m above the ground. The images are collected in the same planimetric position (X, Y) but at different heights (Z). Small vertical displacements are used to avoid abrupt viewpoint changes among sequential images. During the image acquisition, each camera height must be recorded to be later used as constraint in the image orientation. Each fisheye image is rectified to vertical planes, generating lateral views, as exemplified in Figure 2(a). This procedure produces images of the vertical trunks at similar scales in the image sequence, facilitating image matching which is later performed only with corresponding points belonging to stems regions. As depicted in Figure 2(b), the virtual horizontal images simulate a strip sequence in a local reference system. After rectification, the images follow the perspective geometry and then the collinearity equations can be used for the image orientation with bundle adjustment. (a) (b) Figure 2. (a) Vertical fisheye image rectified to four horizontal lateral views. (b) Lateral image sequences simulating photogrammetric strips. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B5, 2016 XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic This contribution has been peer-reviewed. doi:10.5194/isprsarchives-XLI-B5-627-2016 628 An image processing technique was developed to automatically locate and extract stems from the rectified images. Initially the procedure uses a smoothing with median filter (Figure 3(a)) and then performs an image segmentation based on thresholding by the Otsu’s method (Otsu 1979). The trunk extraction is achieved by considering search for straight structures, since the trunks have vertical development (Figure 3(b)). Next, a window is opened surrounding the linear structure, as shown in Figure 3(c). Figure 3. Sequence of image processing to extract tree stems: (a) Rectified and smoothed image in gray levels; (b) detection of straight structures; (c) Stem labelling windows. Area- or feature-based techniques can be used to generate tie points, with the advantage that points will be restricted to areas close to the trunks. As ground control points are not used in this technique, a relative image orientation is performed in a local reference system. Next, a dense matching technique is applied to generate the point cloud, which allows the stem reconstruction. 3. EXPERIMENTS AND RESULTS The tree stem reconstruction was performed with the proposed technique and compared with SfM, based on the generated point clouds. 3.1 Camera calibration A Nikon D3100 digital camera with fisheye lens was used to collet vertical images. Further specifications of the camera are shown in Table 1. Elements Specification Camera model Nikon D3100 Nominal focal length 8 mm (Bower SLY 358N fisheye) Pixel size 5.0 μm Sensor dimensions CMOS APS-C (23.1 × 15.4 mm) Image dimensions 4,608 × 3,072 pixels Table 1. Features of the fisheye camera. The fisheye camera was firstly calibrated in a 3D terrestrial calibration field composed of coded targets with Aruco format (Garrido-Jurado et al. 2014), which were automatically located in the images using a software adapted by Silva et al. (2014) to identify the targets in the field. Figure 4 shows an example of fisheye image with Aruco target corners extracted. IOPs were estimated using the Calibration Multi-Camera (CMC) software (Ruy et al. 2009; Marcato Junior et al., 2015) via bundle adjustment with equidistant model for fisheye lens. Table 2 displays the estimated IOPs. Figure 4. Example of a fisheye image collected in the terrestrial calibration field showing Aruco target corners that were automatically extracted. Parameter Value Standard deviation f (mm) 8.3526 0.00306794 (±0.61 pixels) x0 (mm) 0.0701 0.00119047 (±0.24 pixels) y0 (mm) -0.1308 0.00110620 (±0.22 pixels) K1 (mm-2) 4.50×10-4 6.69×10-6 K2 (mm-4) 3.99×10-7 8.50×10-8 K3 (mm-6) 1.09×10-10 3.54×10-10 P1 (mm-1) 9.27×10-6 1.99×10-6 P2 (mm-1) -9.69×10-6 2.22×10-6 a posteriori sigma 0.98 (a priori sigma = 1 ) Table 2. IOPs estimated with equidistant model by bundle adjustment. 3.2 Image acquisition and data processing A set of nine images was collected with the fisheye camera raised to heights ranging from 4.15 m to 4.55 m with increments The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B5, 2016 XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic This contribution has been peer-reviewed. doi:10.5194/isprsarchives-XLI-B5-627-2016 629 in Z of 5 cm to perform the experiments in the study area, as shown in Figure 2. After rectification to vertical planes, a photogrammetric project in Leica Photogrammetric Suite (LPS-ERDAS) was configured to perform a relative image orientation in a local reference system as follows:  IOPs were fixed.  EOPs of the first image (the lowest height) were fixed with absolute constraints. The coordinates (X, Y) were fixed as the origin (0, 0) planimetric coordintates and Z used the height measured during the image acquisition;  EOPs of the following images were defined with weighted constraints, considering standard deviations of σ = 5 cm for XY, due to moviments when lifting the camera, and σ =1 mm for Z because this coordinate was accurately measured in field. The initial values for XY were (0, 0) and Z for each image was defined with its respective height collect in field;  Tie points were automatically extracted from the segmented stems using high point density. As the aim is to use few images in the stem reconstruction, only four images were inserted into the LPS project to generate a point cloud. In the tests, two lateral views (left and right in Figure 2(a)) were sufficient to cover all trees appearing in the scene. Then, two image strips from opposite sides were used in the bundle adjustment. After image orientation, a Digital Surface Model (DSM) was automatically generated with grid spacing of 2 cm and using the Automatic Terrain Extraction (ATE) module from LPS. Figures 5(a,b) show the point clouds generated with dense matching via ATE-LPS. Typically, many images are required to apply the SfM technique. Thus, the nine rectified and segmented images of each side (left and right) were used to generate point clouds for comparison with the results from LPS. The Agisoft Photoscan software, which implements a SfM strategy, was used with default configuration. As can be seen in the comparison between Figures 5 and 6, four imagens using LPS were sufficient to produce point clouds with more density than those with SfM, which used sequences with nine images. Only four images in the SfM were not enough to generate point clouds. Later, the produced point clouds can be used to extract measures of the stems. From the point clouds produced, a surface or mesh can be built and texturized to generate models. Figures 7(a,b) present models in tiles generated by Photoscan using a default configuration. In this case, the purpose was only to show a realistic model that can be produced with the virtual horizontal images. (a) (b) Figure 5: Point clouds generated by bundle adjustment and ATE in LPS using images from the (a) right and (b) left . (a) (b) Figure 6: Point clouds generated by the PhotoScan software using a sequence of nine images from the (a) right and (b) left sides. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B5, 2016 XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic This contribution has been peer-reviewed. doi:10.5194/isprsarchives-XLI-B5-627-2016 630 (a) (b) Figure 7: Stem models created with the 3D point clouds: (a) right side and (b) left side. 3.3 Accuracy assessment with distances An accuracy assessment with distances was performed over the reconstructed stems. During the image acquisition, distances at approximately 1.30 m of height were measured from the camera station for each tree using an electronic distance measurement device. Considering that the measurements were performed without targets over the stems, there may be an error around ~10 cm associated with the distances due to uncertainties in the field measurements, irregularities in the stem surfaces and in the point location over the point cloud. Even that, the distances were used to verify if the results were close to the observed distances. From each stem within the 3D point clouds generated with ATE-LPS, the centre point closest to 1.30 m height was selected to check the distance between the camera station and the tree. Table 3 shows the differences obtained by subtracting the estimated distances from the directly measured distances. Stem Measured distance (m) Difference (cm) 1 2.602 10.1 2 4.390 9.3 3 4.416 18.0 4 4.501 5.7 5 4.937 21.3 Table 3. Error assessment with distances. As can be seen, the differences had values in a centimetre-level. The largest and smallest values were respectively 21.3 cm and 5.7 cm, being the largest error generated with the largest distance from the tree to the camera station. The differences were 12.8 cm on average. It is important to note that the discrepancies showed in Table 3 evidenced that the point clouds have acceptable 3D geometry provided by the fisheye technique. 4. CONCLUSIONS This paper presented a feasibility study for stem reconstruction using few images taken with fisheye lenses. The purpose was to develop a straightforward technique for image acquisition in forest plots without requiring significant time demand. Furthermore, the technique was designed to use ordinary photogrammetric software for data processing with perspective geometry, enabling the accurate data extraction of tree trunks. The experiments showed that it was possible to reconstruct stems using horizontal virtual images created from the vertical fisheye images, with the additional advantage of using fewer images and taken from one single station. When comparing point clouds generated by ATE-LPS and SfM, the bundle adjustment via LPS produced more dense 3D point clouds. The preliminary results were only checked with distances measured between the trees and the camera station. However, the achieved accuracy (12.8 cm) make feasible to conclude about the technical feasibility, which can be improved with some further refinements. For future studies, the results achieved with the fisheye technique should be assessed to verify the accuracy level of tree attributes, for example, errors of DBH measures and tree locations for forest inventory. In addition, point clouds generated with fisheye images should be compared with point clouds collected with TLS, verifying whether or not the data provided by photogrammetry can present similar accuracy. ACKNOWLEDGEMENTS The authors would like to thank the São Paulo Research Foundation (FAPESP) – grants 2013/50426-4 and 2014/05533-7 and the Finnish Geodetic Institute (FGI) for cooperation in the field work. REFERENCES Berveglieri, A., Oliveira, R. A. and Tommaselli, A.M.G., 2014. A feasibility study on the measurement of tree trunks in forests using multi-scale vertical images. In: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Riva del Garda, Italy, Vol. XL-5, pp.87–92. Forsman, M., Börlin, N. and Holmgren, J., 2016. Estimation of Tree Stem Attributes Using Terrestrial Photogrammetry with a Camera Rig. Forests, 7(3), pp. 1-20. Fryer, J.G. and Brown, D.C., 1986. Lens distortion for close- range photogrammetry. Photogrammetric Engineering & Remote Sensing, 52(1), pp. 51–58. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B5, 2016 XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic This contribution has been peer-reviewed. doi:10.5194/isprsarchives-XLI-B5-627-2016 631 Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F.J. and Marín-Jiménez, M.J., 2014. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition, 47(6), pp. 2280 – 2292. Hapca, A. I., Mothe, F. and Leban, J. M., 2007. A digital photographic method for 3D reconstruction of standing tree shape. Annals of forest science, 64(6), pp. 631-637. Herrera P. J., Pajares, G., Guijarro, M., Ruz J.J. and Cruz, J. M., 2011. A stereovision matching strategy for images captured with fish-eye lenses in forest environments. Sensors, 11(2), pp. 1756-1783. Liang, X., Jaakkola, A., Wang, Y., Hyyppä, J., Honkavaara, E., Liu, J. and Kaartinen, H., 2014. The use of a hand-held camera for individual tree 3d mapping in forest sample plots. Remote Sensing, 6(7), pp. 6587–6603. Liang, X., Yunsheng, W., Jaakkola, A., Kukko, A., Kaartinen, H., Hyyppä, J., Honkavaara, E. and Liu, J., 2015. Forest data collection using terrestrial image-based point clouds from a handheld camera compared to terrestrial and personal laser scanning. IEEE Transactions on Geoscience and Remote Sensing, 53(9), pp. 5117–5132. Marcato Junior, J., Moraes, M. V. A. and Tommaselli, A. M. G., 2015. Experimental assessment of techniques for fisheye camera calibration. Boletim de Ciências Geodésicas, 21(3), pp. 637–51. Otsu, N., 1979. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1), pp. 62–66. Ruy, R., Tommaselli, A.M.G., Galo, M., M., Hasegawa, J.K. and Reis, T.T., 2009. Evaluation of bundle block adjustment with additional parameters using images acquired by SAAPI system In: Proceedings of 6th International Symposium on Mobile Mapping Technology. Presidente Prudente, Brazil. Schneider, D., Schwalbe, E. and Maas, H.G., 2009. Validation of geometric models for fisheye lenses. ISPRS Journal of Photogrammetry and Remote Sensing, 64(3), pp. 259–266. Silva, S.L.A., Tommaselli, A.M.G. and Artero, A.O., 2014. Utilização de alvos codificados na automação do processo de calibração de câmaras. Boletim de Ciências Geodésicas, 20(3), pp. 636–656. Tommaselli, A.M.G. and Berveglieri, A., 2014. Automatic orientation of multi-scale terrestrial images for 3D reconstruction. Remote Sensing, 6(4), pp. 3020-3040. Westoby, M. J., Brasington, J., Glasser, N. F., Hambrey, M. J. and Reynolds, J. M., 2012. ‘Structure-from-Motion’ photogrammetry: a low-cost, effective tool for geoscience applications. Geomorphology, 179, pp. 300–314. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLI-B5, 2016 XXIII ISPRS Congress, 12–19 July 2016, Prague, Czech Republic This contribution has been peer-reviewed. doi:10.5194/isprsarchives-XLI-B5-627-2016 632