The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W18, 2019

GeoSpatial Conference 2019 – Joint Conferences of SMPR and GI Research, 12–14 October 2019, Karaj, Iran

BAND TO BAND REGISTRATION OF MULTI-SPECTRAL AERIAL IMAGERY –

RELIEF DISPLACEMENT AND MISS-REGISTRATION ERROR

M. Hassanpour 1, F. Dadras Javan 1, *, A. Azizi 1

1 School of Surveying and Geospatial Engineering, University College of Engineering, University of Tehran, Tehran, Iran –
(fdadrasjavan, mhd.hassanpour)@ut.ac.ir

Commission VI, WG VI/4

KEY WORDS: Band to Band Registration, Multi-Spectral Imagery, Relief Displacement, Miss-registration Error, Aerial Imagery

ABSTRACT:

With the development of UAVs in agricultural applications, multi-spectral cameras are designed to be installed on these platforms in
small size and weight to help users for applications. These cameras are designed in multi lenses structure each lens considered for
each spectral band. Band miss-registration of low altitude multispectral images acquired via UAVs is challenging due to multi view
image sensors and optics structure of the multispectral cameras. In this study, an image based registration method is proposed to
reduce the effect of relief displacement on miss-registration error. For this purpose, three different data sets with different level of
relief displacement are considered and three different strategies are proposed. In the proposed method, by dividing the image into
patches, selecting the appropriate local window, and local matching in each window, an attempt has been made to obtain the proper
distribution of points in the image. Also, by limiting the search space, the matching speed is also increased. To evaluate the proposed
method, the comparison with the ENVI and ArcGIS software has been made where results clearly present the superiority of the
proposed method by average of 0.28 pixel in accuracy.

1. INTRODUCTION steps of registration are done in image space. Xiaoyang Zhao

(2018), registered bands of Red-Edge’s camera through local
In image processing applications, image registration is one of the window, Fang Xu (2018) for multi temporal image registration
most important demands (Aicardi, et al, 2016). Alignment of used two steps of coarse registration and fine registration, Yanjia
images taken from UAVs, airplanes, or satellites, image chen (2017) for visible and infrared images registration applied
registration is highly needed (Pritt, et al, 2014). The visible two steps of coarse and fine registration. But in the object based
cameras usually have 3 bands including Red, Green, and Blue strategy, images are registered in the object space or the model
bands, where they are not much applicable in the field of space. Hao Gan (2018), for visible and thermal camera
agriculture and vegetation based application (Zhang, et al, 2016). registration, and Jyun-Ping Jhan (2016-2017-2018) for band to
For this propose, multi-spectral cameras that have more than 3 band registration of mini-MCA’s camera used object based
visible bands are used, which can be noted the near infrared or approaches.
red edge bands beside the visible bands. Multi-spectral images When multi-spectral cameras are installed on UAVs and image
are usually captured by satellites, which later evolves with the are captured at low altitudes, miss registration error between
development of technology and UAVs, multi-spectral cameras bands becomes more distorted. Also, in the situation of high
are built in small size to be installed and operated on the UAVs elevation differences, the error resulting from the relief
(Grant, 2017). For example, Red-Edge camera (MicaSense, Inc., displacement appears in the images which presented in figure 2.
USA), which has five bands of Red, Green, Blue, Near Infrared,
and Red-edge which structure is presented in figure 1.

Figure1. Red-Edge camera structure Figure2. Relief displacement

Accordingly, the following points can be considered:
In multi-spectral images that captured by UAVs and airplanes,  The imaging height is low in many applications, such
because of the frame based geometry and the multi lens structure as agriculture, with the difference in the height of the
of these sensors, band to band registration is critical before using areas, and the relief displacement is unavoidable.
these images in further processing application (Zhao, et al, 2018).
 This difference has been made in applications such as
In this context, there are two approaches for image registration as
diagnosis of pests, plant diseases, and so on where even
object level and image level. In the image based strategy, all the
a pixel displacement may

* Corresponding author

This contribution has been peer-reviewed.

https://doi.org/10.5194/isprs-archives-XLII-4-W18-467-2019 | © Authors 2019. CC BY 4.0 License. 467
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W18, 2019
GeoSpatial Conference 2019 – Joint Conferences of SMPR and GI Research, 12–14 October 2019, Karaj, Iran


 lead to false conclusion. It is necessary to correct this
error.
 Multi-spectral cameras include multi lenses. The 5-lens
Red-Edge camera is one of the most common cameras
used in agricultural applications.


The purpose of this study is to develop a new method for band to (a) (b)
band registration of aerial multi-spectral images which is
composed of choosing a suitable feature points extractor,
selecting the optimal solution for data with different features and
different height displacement, and if necessary, selecting a local
window to detect feature points with proper distribution
throughout the images. Finally, for accuracy assessment of the (c) (d) (e)
study, results are compared to those achieved by ENVI and Figure 5. Data set 3: the image of an orchard in five bands with
ArcGIS software. high altitudes differences, up to 6 meters: (a) red, (b) green, (c)
blue, (d) NIR, (e) red-edge.
2. MATERIALS AND METHODS
2.2 Proposed Band to Bands Registration’s Methods
2.1 Test Images
Due to the optics and lenses structure of the multi-spectral
Test images are taken by Red-Edge camera (MicaSense, Inc., camera, the images of this type of camera do not align with each
USA), which has five spectral bands including Red, Green, Blue, other, and if the images are taken at low altitude, an error due to
Near Infrared (NIR), and Red-Edge by a small multirotor the lack of uniformity of the height difference displacement in
platform, over 3 different height relief fields. These areas cover the images is created. For a better understanding, this is shown in
a flat area (dataset 1) (micasense.com), the cultivating area with Figures 6, 7 and 8.
bushes with moderate height difference (dataset 2) As shown in figure 7, each lens is in a different position than the
(micasense.com), and a Tree Garden area with trees of varying other lenses. for this reason, the images taken by these lenses
heights (dataset3). For dataset 3, some of flight planning aren’t aligned with each other. Also, due to the different viewing
information as follows: angles of each lens relative to a feature, the error due to the height
Flight altitude is 25 (m), Ground resolution is 12.7 (mm/pix), difference in each image will be different from other images,
Footprint is 16.25×12.45 (m2). which would result in a high-altitude features error in low altitude
imaging, after registration.

(a) (b)

(c) (d) (e)

Figure 3. Data set 1: the image of a calibration panel in five
bands with almost flat contents: (a) red, (b) green, (c) blue, (d) Figure 6. Miss registration in multi-spectral images
NIR, (e) red-edge.

(a) (b)

(c) (d) (e)

Figure 4. Data set 2: the image of an agricultural area in five
bands with contents of mid height relief: (a) red, (b) green, (c) Figure 7. Lens structure and not alignement
blue, (d) NIR, (e) red-edge.

This contribution has been peer-reviewed.

https://doi.org/10.5194/isprs-archives-XLII-4-W18-467-2019 | © Authors 2019. CC BY 4.0 License. 468
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W18, 2019
GeoSpatial Conference 2019 – Joint Conferences of SMPR and GI Research, 12–14 October 2019, Karaj, Iran

Since the effects of varying heights cause problems in the

distribution of points, they cause the occurrence of large errors in
high difference features compared to low difference features. For where dh = features height
example, the miss-registration error occurred in a high relief area dp = difference of parallax
is up to 10 pixels (Fig. 8). b = Distance between shooting stations
on the photo
H = flight height

If the multi-spectral camera bands registered correctly to each

other, the height difference displacement value can be equal to
the parallax in equation 1. Thus, with a constant flight height, the
higher the difference of height of the features, the greater the
displacement. Because of the need to fly at low altitude in plant
degradation and plant disease applications, the data can be
divided into the following three categories.
 Data with almost flat features that can be neglected due
to relief displacement.
Figure 8. miss-registration error vecor in data set 3, with 20x  Data with moderate heights of 1 to 3 meters, with a lack
magnification and the points are presented in blue and the error of relief displacement.
vector in red lines.  Data with features of over 3 meters, which is important
due to the relief displacement.
In figure 8, which is a slice of the data set 3, it is well presented So, first, the data is analyzed in terms of elevation features. Then
that the error vector for the points on top of the trees is larger than three registration methods will be discussed. In the first method,
the error vector for the points on the ground. a global registration is made on the whole image. In the second
Since height difference displacement is one of the important issue method, images are divided into several patches, and in each
in the photogrammetry and the main challenge in multi-spectral patch feature points are extracted. According to Table 1, which
cameras installed on UAVs, then an explanation about the height shows the number of extracted and matched points, along with
difference displacement will be initially given. The height running time in 6, 12, and 20 patches. Therefore, if the number
difference displacement is the radial distance between a feature of patches get larger, then the more corresponding points will be
in the image and its correct position in the planer coordinate added. But by according to running time, 6 patches have better
system relative to the center of the image (Sunita Devi, Veena, results than 12 or 20. So in this study to avoid creating gaps in
2014). Figure 9 shows the height displacement. the image, 6 patches have been chosen.

Num.
Matched
of Bands Points Time (s)
points
patches
Red 475 425 18/34
Green 2942 2888 18/22
6
Blue 691 645 17/51
NIR 1888 1779 15/10
Red 759 490 32/05
Green 3167 2896 31/36
12
Blue 819 710 31/21
NIR 1937 1791 29/49
Red 920 564 65/09
Green 3432 2924 69/50
20
Blue 1002 806 70/39
Figure 9. Relief displacement image (Sunita Devi, Veena, 2014) NIR 2432 1926 62/48
This displacement is different for different heights. On the other Table 1. Time, points, and matched points in different patches
hand, the relationship between parallax and the height of features of data with high altitudes
can be expressed in Equation 1.
In the third method, the images are divided into several patches,
and after that points are extracted in each patch. Moreover, in the
dp form of windows to the center of each point, new points are
dh  H extracted, and then registration is done. This helps in distributing
dp  b (1) points appropriately throughout the image. Since the features in
each area are different in height, distribution of points will be

This contribution has been peer-reviewed.

https://doi.org/10.5194/isprs-archives-XLII-4-W18-467-2019 | © Authors 2019. CC BY 4.0 License. 469
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W18, 2019
GeoSpatial Conference 2019 – Joint Conferences of SMPR and GI Research, 12–14 October 2019, Karaj, Iran

accompanied by problems that aggravate relief displacement. ratio (NNR) or the sum of square differences (SSD) are used in
The processing steps of the method are shown in Figure 10. order to find matched pairs. Extracted matched points are
. struggling with mismatches. So, points that have been mistakenly
matched are eliminated using RANSAC. Finally, using a transfer
function such as affine, geometric transformation between the
images are performed, and a number of check points are
considered and compared to control the work.

2.2.2 Patch-Wise Registration’s Method

In this method, at first the images are divided into six overlapping
patches due to the reason given in table 1. This overlapping is due
to avoid creating gaps between the patches. In each patch of the
master (reference) image and other bands, the points are xtracted.
In this study, 6 patches were used. After the extraction of points
in each patch, the points are put together in the original image
and the matching process is done. Corresponding points may be
wrong, to delete the wrong points RANSAC is applied. Finally,
a geometric transformation is performed using affine transfer
function. Check points are also used to estimate accuracy.

2.2.3 Patch-Wise and Local Window Registration’

Method

This method includes the following steps:

Image Patch Generation. Images are divided into six patches

that are overlapped with each other, so that there is no gap
between them. The number of patches is selected according to the
size of the image and based on practical tests. Whatever the
differences between the altitudes is high, more patches must be
selected and in smaller size, but they should not be so small to
complicate the computational process.
Patch based matching. The corresponding patches are
considered in the reference and slave image and the feature points
are extracted. Then, the matching operation is performed for each
corresponding patches.
Local window selection. In the master image and the slave
image, extracted points are selected as window centers. The size
of these new windows should be the same everywhere.
Feature points extraction in sub-images. In this section, in each
of the extracted windows (sub-images), feature points are
extracted.
Local matching. In each corresponding sub-images (windows),
the matching operations are performed for extracted features.
This not only improves the search speed, but also results in more
uniform distribution and higher density of extracted
corresponding points in all parts of the image.
Blunder removal. Despite the matching process of the points,
there will still be miss match points in the image. Therefore, by
estimating an affine transformation model by RANSAC, the
elimination of miss match points is considered. This algorithm
first estimates an approximate Affine model with three random
points. Then, the transformation model is fitted to all points by a
fitness function.
By default, the number of maximum random trails to find inlier
matches of 1000 and the confidence of finding the maximum
number of inlier matches is set to 0.99. Therefore, the maximum
distance in pixels from a point is set to 1.5 for the estimation of
transformation (Zhao, et al, 2018).
Geometric transformation. Considering the extracted and
Figure 10. Diagram of band to band registration’s methods. matched points as corresponding points, and using affine
transformation model, slave images are transferred to the master
2.2.1 Global Registration’s Method image and resampled.
The transformation function defined for the coordinates of the
By using a points extraction algorithm, some points are extracted corresponding control points in the two master images and the
in both reference and slave images. Then the nearest neighbor other image. Ttransfer, rotation, scale, or a combination of these,

This contribution has been peer-reviewed.

https://doi.org/10.5194/isprs-archives-XLII-4-W18-467-2019 | © Authors 2019. CC BY 4.0 License. 470
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W18, 2019
GeoSpatial Conference 2019 – Joint Conferences of SMPR and GI Research, 12–14 October 2019, Karaj, Iran

have been considered for Affine transformation. In this process,

the matrix of the corresponding points in the master image is Correct
Time's Speed of Match
presented as p  f ( x, y, z ) and matrix of points in slave Name
extract-ion
Points
extract-ion points
match
points
images is presented as q  f ( x, y, z) . The relation between
SURF 0.535 28265 18.926 1066 243
two images is considered as p  H .q where H is a 3 3
FAST 0.01 1423 70.27 361 55
matrix (Yang, et al, 2016). This final relation is presented in
equation 2. SIFT 0.623 20465 30.44 967 198
BRISK 0.171 2084 82.05 95 53
 x   h11 h12 h13   x 
p   y   h21 h23 . y 
Harris 0.157 825 19.03 544 58
h22 Table 3. Comparison of point extraction algorithms, data set 3
 z  h31 h32 h33   z  
According to Tables 2-4, the SURF algorithm has higher speed
and is more precise in matching. However, the number of
corresponding points must also be considered and increased.
h1T   x 
 T 
h2 . y   H .q (2) On all the three different data, orchards contents with high
difference in altitudes, agriculture with medium height
h3T   z  
  differences and flat area, proposed registration strategy is
performed.
In figure 11 to 13, the distribution of extracted points for three
Where hij (i  1,2,3; j  1,2,3) is elements of H and data set based on proposed method is presented.

hi (i  1,2,3) means (hi1 , hi 2 , hi 3 ) . In an Affine

transformation h31  h32  0 and h33  1 (Zhao, et al,
2018).

3. RESULTS

In the first step, comparing SIFT, SURF, Harris, FAST, and

BRISK feature points extractors, the SURF operator has been
selected for being applied in the proposed method due to the a) Matched points before b) Registered images
speed and the larger number of extracted corresponding points in registration
comparision with other operators. Figure 11. Matching and registration, data set1

Speed of Correct
Time's Match
Name Points extract- match
extraction Points
ion points
SURF 0.864 96189 89.82 21568 10425
FAST 0.02 7598 26.32 981 491
SIFT 0.823 90884 90.55 19125 956
BRISK 0.267 5197 51.37 189 75
Harris 0.259 2548 10.16 909 83
Table 2. Comparison of point extraction algorithms, data set 1 a) Matched points in global b) Matched points in path-
method before registration wise method before
registration
Time's Speed of Correct
Match
Name extract- Points extract- match
points
ion ion points
SURF 0.752 54593 13.77 2643 967
FAST 0.02 4925 40.60 739 247
SIFT 0.617 41265 14.95 1946 762
BRISK 0.249 3641 68.38 106 73
Harris 0.251 1374 18.26 742 79
Table 3. Comparison of point extraction algorithms, data set 2 c) Registered images
Figure 12. Matching and registration, data set2

This contribution has been peer-reviewed.

https://doi.org/10.5194/isprs-archives-XLII-4-W18-467-2019 | © Authors 2019. CC BY 4.0 License. 471
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W18, 2019
GeoSpatial Conference 2019 – Joint Conferences of SMPR and GI Research, 12–14 October 2019, Karaj, Iran

c) Registered images
a) Matched points in global b) Matched points in path-
method before registration wise and local window
method before registration

Figure 15. Final accuracy of band to band registration of the 4

bands with the reference of the red-edge band, data set 2

c) Registered images
Figure 12. Matching and registration, data set3

Also in figures 14 to 16, accuracy assessment of proposed

method for three data sets is displayed based on RMSE. As
shown in Figures 14 to16, the accuracy of the Patch-Wise and
local window method is better than the other two methods, but in
the Patch-Wise and local window method, due to the higher
number of extracted points in the first and second data, the
processing time is considerably high. Also, for the first data, the
Patch-Wise method extracts more points and increases the
processing time, but this method will have an acceptable Figure 16. Final accuracy of band to band registration of the 4
accuracy by extracting a suitable number of points and spending bands with the reference of the red-edge band, data set 3
a suitable time for the second data. for data set 1, the Global
method is very acceptable. because the number of extraction For Patch-Wise and local window based methods, the selection
points and spending time for this data is acceptable. But for the of window size is one of the important challenges. Figure 17
third data which is high in height difference, the Global and shows the number of extracted points for different window sizes.
Patch-Wise methods are not suitable because of the lack of As the window size increases, the number of extracted points also
uniform distribution throughout the image, and the proposed increases, but duplicate points may be extracted, which itself
method has shown much better results. increases the processing time. The larger the size of the window,
the lower the correct matching rate, and the speed of the run with
a sharp drop. Therefore, the window with small dimensions has
more precision in matching (Zhao, et al, 2018).

Figure 14. Final accuracy of band to band registration of the 4

bands with the reference of the red-edge band, data set 1
Figure 17. The number of correct matched points based on the
window size

This contribution has been peer-reviewed.

https://doi.org/10.5194/isprs-archives-XLII-4-W18-467-2019 | © Authors 2019. CC BY 4.0 License. 472
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W18, 2019
GeoSpatial Conference 2019 – Joint Conferences of SMPR and GI Research, 12–14 October 2019, Karaj, Iran

4. DISCUSSION AND CONCLUSION

In this paper, effect of relief displacement on band to band

registration error is investigated and a three view strategy is
proposed. For evaluation of the proposed method, three different
data sets from Red-Edge’s multi spectral camera (MicaSense,
Inc., USA) is applied.
According to achieved results, it can be concluded that miss-
registration error in data sets where the relief displacement with
respect to flight height is high, is prominent. In these situation
and based on the lower level of textural pattern in UAV based
multi spectral images, common feature extractor and matching
operator cannot do well. This is why the number and distribution
of extracted points and consequently the accuracy of registration
Figure 19. Compare the precision of the proposed method with
would not be acceptable.
the ENVI and ArcGIS software.
Achieved results obviously present that using the proposed
Patch–Wise and local window based strategy, not only the feature
extraction stage but also the accuracy of mapping process is
REFERENCES
improved up to the average 0.28 pixel.
To evaluate results, in ENVI and ArcGIS software, registration Aicardi, I., Nex, F., Gerke, M. and Lingua, A., 2016. An image-
process is also performed with reference of red-edge band. Also based approach for the co-registration of multi-temporal UAV
in ENVI software, it requires at least 3 points for automatic image datasets. Remote sensing, 8(9), p.779.
registration. The number of correct matched points for different
methods are presented in figure 18. Chen, Y., Zhang, X., Zhang, Y., Maybank, S.J. and Fu, Z., 2018.
Results also presented that the accuracy of registration process in Visible and infrared image registration based on region features
the proposed method is better than those from ENVI and ArcGIS and edginess. Machine Vision and Applications, 29(1), pp.113-
software. 123.

Devi, S., 2014. Veena. Measurement of Relief Displacement

from Vertical Photograph. Int. J. Sci. Eng. Technol Res, 3,
pp.2800-2805.

Grant, B.G., 2017, May. UAV imagery analysis: challenges and

opportunities. In Long-Range Imaging II (Vol. 10204, p.
1020406). International Society for Optics and Photonics.

Gan, H., Lee, W.S. and Alchanatis, V., 2018. A photogrammetry-

based image registration method for multi-camera systems–With
applications in images of a tree crop. Biosystems engineering,
174, pp.89-106.

Jhan, J.P., Rau, J.Y. and Huang, C.Y., 2016. Band-to-band

registration and ortho-rectification of multilens/multispectral
imagery: A case study of MiniMCA-12 acquired by a fixed-wing
Figure 18. Comparison of the number of correct matched points UAS. ISPRS Journal of Photogrammetry and Remote Sensing,
between Patch-Wise and local window strategy with ENVI and 114, pp.66-77.
ArcGIS software
Jhan, J.P., Rau, J.Y., Haala, N. and Cramer, M., 2017.
As shown in figure 18, the corresponding points in the Patch- Investigation of parallax issues for multi-lens multispectral
Wise and local window method are much higher than ENVI and camera band co-registration. The International Archives of
ArcGIS. The corresponding points for all three categories of data Photogrammetry, Remote Sensing and Spatial Information
have occurred in ENVI and ArcGIS software in homogeneous Sciences, 42, p.157.
regions that may not be accurate. The accuracy of Patch-Wise
and local window method is good and acceptable. According to Jhan, J.P., Rau, J.Y. and Haala, N., 2018. Robust and adaptive
figure 19, the accuracy of the proposed method is better than band-to-band image transform of UAS miniature multi-lens
ENVI and ArcGIS. multispectral camera. ISPRS Journal of Photogrammetry and
Remote Sensing, 137, pp.47-60.

https://www.micasense.com/rededge-mx-sample-data

Pritt, M. and Gribbons, M.A., Lockheed Martin Corp, 2014.

Automated registration of synthetic aperture radar imagery with
high resolution digital elevation models. U.S. Patent 8,842,036.

Xu, F., Yu, H., Wang, J. and Yang, W., 2018, July. Accurate
Registration of Multitemporal UAV Images Based on Detection

This contribution has been peer-reviewed.

https://doi.org/10.5194/isprs-archives-XLII-4-W18-467-2019 | © Authors 2019. CC BY 4.0 License. 473
 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-4/W18, 2019
GeoSpatial Conference 2019 – Joint Conferences of SMPR and GI Research, 12–14 October 2019, Karaj, Iran

of Major Changes. In 2018 21st International Conference on

Information Fusion (FUSION) (pp. 1480-1485). IEEE.

Yang, K., Tang, L., Liu, X., Dingxiang, W.U., Bian, Y.,
Zhenglong, L.I., 2016. Differen source image registration
method based on texture common factor. Comput. Eng., Vol. 42,
pp. 233-237.

Zhang, J., Yang, C., Song, H., Hoffmann, W., Zhang, D. and
Zhang, G., 2016. Evaluation of an airborne remote sensing
platform consisting of two consumer-grade cameras for crop
identification. Remote Sensing, 8(3), p.257.

Zhao, X., Zhang, J., Yang, C., Song, H., Shi, Y., Zhou, X., Zhang,
D. and Zhang, G., 2018. Registration for Optical Multimodal
Remote Sensing Images Based on FAST Detection, Window
Selection, and Histogram Specification. Remote Sensing, 10(5),
p.663.

This contribution has been peer-reviewed.

https://doi.org/10.5194/isprs-archives-XLII-4-W18-467-2019 | © Authors 2019. CC BY 4.0 License. 474