2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

October 25-29, 2020, Las Vegas, NV, USA (Virtual)

CUHK-AHU Dataset: Promoting Practical Self-Driving Applications in

the Complex Airport Logistics, Hill and Urban Environments
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) | 978-1-7281-6212-6/20/$31.00 ©2020 IEEE | DOI: 10.1109/IROS45743.2020.9341317

Wen Chen, Zhe Liu, Hongchao Zhao, Shunbo Zhou, Haoang Li, Yun-Hui Liu

Abstract— This paper presents a novel dataset targeting three

types of challenging environments for autonomous driving, Data collection
platform Data collection

i.e., the industrial logistics environment, the undulating hill

platform

environment and the mixed complex urban environment. To

the best of the author’s knowledge, similar dataset has not
been published in the existing public datasets, especially for
the logistics environment collected in the functioning Hong
Kong Air Cargo Terminal (HACT). Structural changes always
suddenly appeared in the airport logistics environment due to
the frequent movement of goods in and out. In the structureless (a) (b)
and noisy hill environment, the non-flat plane movement is
usual. In the mixed complex urban environment, the highly VLP-16
z x
z
dynamic residence blocks, sloped roads and highways are z x Camera 0
y Camera 1z x y y
included in a single collection. The presented dataset includes z Camera 5
60° z x
LiDAR, image, IMU and GPS data by repeatedly driving along x y 60°
y
several paths to capture the structural changes, the illumination x
60°
60° y x
changes and the different degrees of undulation of the roads. z 60°
IMU
60°
The baseline trajectories are provided which are estimated by Camera 2
y
Simultaneous Localization and Mapping (SLAM). x z
x y
Camera 4
I. INTRODUCTION z y
z y
In the field of autonomous driving, public datasets provide z
Camera 3 y
good data sources for research and verification of various
x
algorithms. They are generally collected from real environ-
ments with different characteristics using data acquisition x
vehicles equipped with different combinations of sensors. Dual-antenna GPS
Among them, cameras or LiDARs as essential parts are com-
monly combined with other sensors to obtain information (c)
about the surroundings and the vehicles’ motion. Cameras get
rich texture information from the environment, and LiDARs
provide 3D structure information of the environment.
Many research groups have opened up datasets with dif-
ferent features. The most representative and influential one
is the KITTI dataset proposed in [1]. It provides LiDAR and
image data in lowly dynamic rural areas and highways for
different research topics, such as state estimation, 3D object
detection and tracking. The dataset presented in [2] consists
of two levels of sensor data collected from the complex urban
environments. Focusing on the long-term autonomy, the
NCLT dataset in [3] consists of omnidirectional image and
LiDAR data collected along various trajectories in the cam-
pus environment. Similarly, the Oxford RobotCar Dataset
[4] provides LiDAR and image data along the same route
This work was supported in part by the Natural Science Foundation
of China under Grant U1613218, in part by the Hong Kong ITC under (d)
Grant ITS/448/16FP, in part by the National Key Research and Development
Program of China under Grant 2018YFB1309300, in part by the Hong Kong Fig. 1. Two types of data collection vehicles with the same platform. (a)
Centre for Logistics Robotics, and in part by the VC Fund 4930745 of the Platform installed on the industrial tractor in airport. (b) Platform installed
CUHK T Stone Robotics Institute. on the passenger car. (c) The layout of the sensors on data collection
W. Chen, Z. Liu, H. Zhao, S. Zhou, H. Li and Y.-H. Liu are with the T platform. (d) The typical scene in the industrial logistics environment. The
Stone Robotics Institute and Department of Mechanical and Automation six-camera unit provides synchronized 360 visual measurements. The 3D
Engineering, The Chinese University of Hong Kong, HKSAR, China. LiDAR (VLP-16) provides 360 range measurements.
([email protected])

978-1-7281-6212-6/20/$31.00 ©2020 IEEE 4283

Authorized licensed use limited to: Rutgers University. Downloaded on May 15,2021 at 12:08:17 UTC from IEEE Xplore. Restrictions apply.
in dynamic urban environments. Both datasets are collected independent mechanical structure which is attached on the
at different time over one year, suffering from environment top of vehicles. The main objective of the modular design
structural changes and various weather conditions. Other is to acquire ominidirectional visual and range informa-
relevant datasets released in [5], [6], [7] are collected from tion from totally different application environments, such
urban environments using different sensor combinations. as airport cargo terminal and undulating hill roads. The
These datasets have played a very good role in promoting configuration of the sensors is summarized in Table. I.
the development of autonomous driving in urban or suburban A 3D LiDAR (“VLP16”) is installed at the center of
environments. However, the designed collection routes are the platform to provide 360-degree and 16-channel range
relatively simple such that few datasets can cover different information of the surrounding scene at a frequency of 10Hz.
environments in a single collection. What’s more, to the For the inertial measurements, a consumer-level MTi IMU
best of our knowledge, there is not dataset that targets sensor is mounted under the LiDAR by coaxial installation. It
the challenges in the industrial logistics environment and can provide nine-axis measurements: linear acceleration, ro-
undulating hill environment. In these two environments, tational angular velocity and geomagnetism orientation. The
autonomous driving faces different challenges compared with platform also has a dual-antenna GPS with RTK correction
existing datasets. Structural changes are always taking place signals to provide 2.5Hz position measurement.
in the logistics environment. The environment may become The platform includes six color cameras on a flat plane:
completely different as goods are moved in and out at any two Point Grey Grasshopper3 cameras and four Point Grey
time which makes robust localization and mapping very Blackfly cameras. The two Grasshopper3 cameras provide
difficult. In the undulating hill environment, the movement the front and rear view for better imaging quality, and the
of vehicles is not on a flat plane, and the surroundings are remaining four Blackfly cameras are evenly distributed in the
structureless and noisy. other view directions around the central axis of the LiDAR to
In this paper, we present a LiDAR and image dataset form an ominidirectional visual sensing unit. To perform the
that focuses on three types of environments, i.e., air cargo time synchronization of the six-camera unit, synchronized
terminal environment, the undulating hill environment and capture technology provided by the Point Grey is applied to
the mixed complex urban environment using two different ensure the same frame rates. One “primary” camera is used
vehicles shown in Fig. 1. All sensors are mounted on a de- to trigger the other “secondary” cameras by linking their
tachable data collection platform shown in Fig. 1(c), which is GPIO pins. In our application, the “Camera 2” is treated as
then installed on the top of the vehicles. The industrial tractor “primary” camera, yielding 20Hz image capture rate.
in Fig. 1(a) is used to collect data in the HACT which is the All sensors are logged using an industrial computer run-
second busiest air cargo terminal in the world. The passenger ning Ubuntu Linux with an i7 processor, 32GB DDR4 mem-
car in Fig. 1(b) is used to collect data around the Chinese ory and two 2TB SSDs. All the sensor drivers are developed
University of Hong Kong (CUHK) campus which is located on the ROS Kinetic. And the logger are based on the rosbag
on a hill. As for the mixed complex urban environment, we package. The timestamp for each sensor measurement is
also use the passenger car to collect data along the route created by the related driver running on the computer.
through the highly dynamic residence blocks, sloped roads
and highways. We repeatedly collect LiDAR, camera, IMU, B. Sensor Calibration
and GPS data along several paths in these environments to 1) Camera Calibration: The camera calibration includes
capture the structural changes, the illumination changes and the estimation of the intrinsic parameters of each camera
the different degrees of undulation of the road. In summary, and the extrinsic transformation among six cameras. The
the key contributions of this paper are listed in the following: intrinsic calibration is performed separately using the Single
• provides a novel dataset which firstly covers three Camera Calibrator App developed by MATLAB. To improve
types of challenging environments: the highly dynamic the intrinsic calibration, we remove the high-error images to
industrial logistics environment, the undulating hill en- make the mean reprojection error lower than 0.3 pixels.
vironment and the mixed complex urban environment. For the extrinsic calibration among cameras, the feature-
• provides baseline trajectories generated by the proposed matching based method is hard to be applied because of
SLAM approach, which combines the state-of-art Li- the limited overlapping view field between any two adjacent
DAR odometry, graph-based optimization and point cameras. Hence, we choose to recover the transformation
cloud based place recognition. matrix between any two cameras from the calibration result
The first batch of the presented dataset is available of the corresponding LiDAR-camera pairs.
at: http://ri.cuhk.edu.hk/research/public_ 2) Joint LiDAR-Camera Calibration: The calibration
datasets/CUHK_AHU_Dataset. More data and devel- problem between LiDAR and camera is a typical Perspective-
opment tools will be released periodically. n-Point (PnP) problem. One common strategy is to transfer
the 3D-to-2D problem to the 3D-to-3D problem by estimat-
II. C OLLECTION P LATFORM C ONFIGURATION ing the depth of each 2D image feature point with the help
A. Hardware Platform of checker board. However, in practice, the placement of the
Fig. 1(c) shows the sensors’ locations on the data col- checker board affects the final calibration result since the
lection platform. All sensors are mounted on a carrier- working distance of the LiDAR ranges from 1m to 100m.

4284

Authorized licensed use limited to: Rutgers University. Downloaded on May 15,2021 at 12:08:17 UTC from IEEE Xplore. Restrictions apply.
 TABLE I
D ETAILED CONFIGURATION INFORMATION OF THE SENSORS USED ON THE PLATFORM

Sensor item Manufacturer Model Hz Description

3D LiDAR Velodyne VLP-16 10 16-channel and 360-degree 3D LiDAR
IMU Xsens MTi-300 100 Full gyro-enhanced AHRS
Dual-antenna GPS Samsung OEM718D 2.5 Commercial grade GPS
1920 ⇥ 1200, 1/1.2” Sony IMX174 CMOS, global shutter,
Main camera 0 Point Grey GS3-U3-23S6C-C 20
8mm anti-shock & vibration lens (Kowa 1” LM8HC-V), 79.4 HFov
1920 ⇥ 1200, 1/1.2” Sony IMX174 CMOS, global shutter,
Slave camera 3 Point Grey GS3-U3-23S6C-C 20
8mm lens (Kowa 1” LM8HC), 79.4 HFoV
1920 ⇥ 1200, 1/1.2” Sony IMX249 CMOS, global shutter,
Slave camera 1&2&4&5 Point Grey BFLY-U3-23S6C-C 20
8mm lens (Kowa 1” LM8HC), 79.4 HFoV

In order to reduce the influence of this factor, we manually In addition, the GPS signals become inevitably unstable or
selected 3D-2D point pairs from different distance zone. For unavailable as the tractor frequently switched between inside
each LiDAR-camera pair, we adopt the same method to find and outside areas. Thus, the GPS measurements are not
the 3D-to-2D correspondences which were initially solved provided in this group.
by the non-iterative EPnP algorithm [8]. In addition, we also 2) Undulating hill environment: Fig. 2(b) shows three
select the image points associated with the same 3D LiDAR paths of collection in the CUHK campus which is situated on
point from the overlapping area of two adjacent images to an undulating hill. The elevation difference of the paths can
enhance the connection between cameras. Finally, all corre- exceed one hundred meters. During the acquisition process,
sponding pairs are put into a jointly non-liner optimization the average driving speed of the car was about 30km/h. The
problem solved by a Gauss-Newton scheme. More details samples of the data group are shown in Fig. 3(b).
can be found in [9]. 3) Mixed complex urban environment: Fig. 2(c) presents
3) GPS/IMU-LiDAR Calibration: The fusion of GPS and the path in mixed complex urban environments, including
IMU provides estimation of the vehicle states [10]. And also, the non-flat mountain roads, highly dynamic living blocks,
the trajectory of the vehicle can be estimated from the match- structured industrial zone and highways. The driving speed
ing of the edge and planar features points extracted from varied from 0km/h to 90km/h. In the process of collection,
LiDAR measurements [11]. The calibration of GPS/IMU and we experienced many situations in manual driving, such
LiDAR becomes a well-known hand-eye calibration problem as long-time traffic jam, overtaking and being overtaken.
[12]. However, the accuracy of the INS and the LiDAR Fig. 3(c) illustrates the typical scenes in the living block
odometry highly depends on the surrounding scenarios: the and the highways.
former needs to work in open and unsheltered environments, B. Data Formats
and the latter needs environments full of structural informa- 1) Synchronized Images: All datasets are distributed by
tion. Thus, we choose an outdoor parking lot to calibrate the the paths of collection into groups (“HACT”, “CUHK”,
extrinsic parameters between the GPS/IMU and LiDAR. “Taipo”). In order to reduce unnecessary trouble in the
III. DATASET D ESCRIPTION process of downloading, each set is further divided into
The datasets collected using our two types of manually several sub-sets whose size dose not exceeds 5GB. Such
driving vehicles cover different features such as industrial a sub-set is compressed in a tar file which is named
logistics environment, undulating hill area and mixed com- as hGroupi hP athi hN o.setsi hN o.subsetsi .tar (e.g.,
plex urban environment. All data are grouped by the path of HACT f orward 1 01.tar). The file tree of the sub-set is
collection, and each group contains at least two sets of data presented in Fig. 4. And the timestamps of measurements
acquired at different time from day to night. The collection from the six cameras and velodyne LiDAR are saved re-
paths of each group are shown in Fig. 2. And the statistics spectively in the hsensori timestamps.csv. The formats
of the size, frame counts, complexity and elevation for each for other measurements are as shown in the following:
path are listed in Table. II. All six cameras synchronize their time using a primary-
secondary trigger mechanism. The timestamps of all images
A. Acquisition Environments are also recorded in case there is a need for higher accuracy
1) Functioning Air Cargo terminal: The HACT group in time processing. The synchronous frequency of these
of datasets is collected in a normally functioning air cargo cameras is 20Hz. The images are saved in the lossless PNG
terminal with the ability of handling 3.5 million tonnes of format whose bayer mode is RGGB8. To convert the bayer
cargo every year. It contains an indoor multi-level storage images to RGB images, the demosaic function in MATLAB
system and several outdoor box storage zones, as shown in or the cvtColor function in OpenCV can be applied.
Fig. 2(a). In order to record the most real logistics scenario, 2) 3D LiDAR scans: The 3D LiDAR scans assembled
we collect the data of the whole terminal environment along from the packets returned by the VLP-16 are saved as
forward and backward paths, without special traffic control. binary files at a frequency of 10Hz. The timestamp of
Fig. 3(a) shows the dramatic changes in the terminal. the last packet during a rotation are recorded as the times-
During data collection, goods, trailers or cargo planes parked tamp of the scan which is used to name the saved file:
at every location may be removed or replaced by others. hT imestampi .bin. Users should pay some attention on the

4285

Authorized licensed use limited to: Rutgers University. Downloaded on May 15,2021 at 12:08:17 UTC from IEEE Xplore. Restrictions apply.
 Outdoor box storage zone
7 10 11
4 3 2

Indoor Indoor
multi- multi-
level level
storage storage
system system

5 6
8 9 12

(a) (b) (c)

Fig. 2. The paths of data collection. (a) The path of data collection in HACT using the industrial tractor. And the numbers are used to visualize the sequence
of the forward and backward paths. The forward sequence is 1 2 3 4 5 6 2 3 5 6 1 2 7 8 9 10 7 8 9 12 11 10 7 2 1,
and the backward sequence is 1 2 7 10 11 12 9 8 7 10 9 8 7 2 1 6 5 3 2 1 6 5 4 3 2 1. (b) The
path of data collection in CUHK campus using the passenger car. (Top): CUHK Path 1; (Middle): CUHK Path 2; (Bottom): CUHK Path 3; (c) The path
of data collection in Taipo using the passenger car. The paths are represented by GPS positions.

TABLE II
S TATISTICS OF THE WHOLE DATASET

Group Path No. Sets Size Duration(s) No.of images/clouds Complexity Elevation (Min/Avg/Max)
8 2922.0 GB 10173 205200/101689 ***** 0 meters
HACTL Forward
8 2771.5 GB 9653 192709/96461 ***** 0 meters
6 1558.2 GB 9671 193577/96373 *** 5/37/123 meters
CUHK Path 1 6 1541.3 GB 9852 198092/98226 *** 6/61/133 meters
4 2591.1 GB 9003 181576/90010 *** 4/23/128 meters
Taipo Path 1 2 2098.7 GB 10734 214428/107295 ***** 1/18/127 meters

recovering of the cut angle since it is not specified during acquisition environments, the GPS-based positioning meth-
collection. All the points are represented by four-dimension ods are not applicable because they are easily affected by
floating tuples: (X, Y, Z, I). I is the intensity of the point. the surrounding buildings and trees. Some paths are even
(X, Y, Z) is the coordinate value of the point represented in indoors. Similar to [2], we provide baseline trajectories
the local Cartesian coordinate defined w.r.t moving LiDAR, estimated by the graph-based SLAM technology [13].
at the time corresponding to the recorded timestamp. So the Our SLAM system fuses the sensor information from
motion distortion problem of the 3D pointclouds is always LiDAR, IMU and GPS. The measurements from IMU are
present, and we haven’t done any process to compensate initially applied to compensate the motion distortion of the
them in the raw data. LiDAR points. Similar to the state-of-art LiDAR odometry
3) GPS/IMU: The GPS measurements are recorded in the [11], the planar and edge features extracted from the undis-
gps.csv at 2.5Hz. They are formated as seven-dimension tu- torted pointclouds are used to establish the constrains with
ple: (timestamp, latitude, longitude, altitude, latitude std, lon- adjacent frames or local map clouds. The integral of IMU
gitude std, altitude std). The imu.csv stores the IMU mea- data are used as the initial guess of the optimization problem
surements at 100Hz. IMU provides the nine-axis measure- constructed from the constrains.
ments from the accelerator, gyroscope and magnetometer. The back-end pose graph optimization is then performed to
They are formated as (timestamp, acc x, acc y, acc z, gyr x, get the globally consistent map. The key to this optimization
gyr y, gyr z, ori qua w, ori qua x, ori qua y, ori qua z). We problem is not only to obtain the estimation of the relative
provide the two-minute data for each dataset when the transformations between pose nodes, but also to obtain the
vehicle is stationary before moving. The initial value for the closed-loop constraints by place recognition. In our previous
estimation of the bias and gravity can be computed. The work, a deep neural network LPN-Net [14] was proposed
fusion results of GPS and IMU are recorded in the ins.csv. to achieve the point cloud based place recognition in large
scale environments. In this paper, we encode the point cloud
IV. BASELINE T RAJECTORIES of each key frame to a global descriptor using the LPN-Net.
The ground-truth trajectories provided by high-precision The re-passed places can be recognized by comparing the
sensors are always needed for evaluating the performance Euclidean distance between the newly-added frame and the
of different algorithms in public datasets. However, in our old key frames. Then the relative transformation between the

4286

Authorized licensed use limited to: Rutgers University. Downloaded on May 15,2021 at 12:08:17 UTC from IEEE Xplore. Restrictions apply.
 (a)

(b)

(c)

Fig. 3. The image samples of each group. The columns from left to right represent the images captured by cameras 0 to camera 5. (a) The samples of
the HACT group, presenting the dramatic changes happened every day in the air cargo terminal. (b) The samples of the CUHK group, illustrating the ups
and downs of the road in the hill environment. (c) The samples of the Taipo group, showing the crowded roads and the high-speed roads in the mixed
complex urban environment.

4287

Authorized licensed use limited to: Rutgers University. Downloaded on May 15,2021 at 12:08:17 UTC from IEEE Xplore. Restrictions apply.
 <Group>_<Path>_<No. sets>_<No. subsets>.tar
camera_0 … camera_5
<Time_stamp>.png <Time_stamp>.png
velodyne

<Time_stamp>.bin
camera_0_timestamps.csv … camera_5_timestamps.csv

velodyne_timestamps.csv

gps.csv

ins.csv

imu.csv

Fig. 4. File tree of each sub-set. (a)

matched clouds are estimated by GICP [15] with the initial
guess from the GPS measurements if they are available. The
reconstructed point-cloud maps in two typical scenarios are
presented in Fig. 5. And the optimized trajectories treated as
baseline are presented by the black points in the map.
V. C ONCLUSION AND F UTURE WORK
We presented a new dataset focusing on autonomous
driving in the industrial logistics environment, the undulating
hill environment and the mixed complex urban environment.
Due to the unavailability of the GPS in some collection paths,
we provide baseline trajectories estimated by the fusion of
LiDAR, IMU and GPS using SLAM. In the future, we will
improve the accuracy of the baseline trajectories by adding
image data. The extrinsic calibration between the LiDAR
and six cameras will be improved with the consideration
of the influence of the partially overlapping among cameras.
The annoation works, like the joint objects annotation on the
ominidirectional 2D visual and 3D range measurements, will
be added in the dataset. In order to accelerate the application
of autonomous driving technology in the industry, the dataset
will continue to be updated, and more typical industrial
environments will be added.
R EFERENCES
[1] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous (b)
driving? the kitti vision benchmark suite,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2012, pp. Fig. 5. The reconstructed maps of part of CUHK (a) and HACT (b). The
3354–3361. baseline trajectories are represented by black points in the maps.
[2] J. Jeong, Y. Cho, Y.-S. Shin, H. Roh, and A. Kim, “Complex urban
dataset with multi-level sensors from highly diverse urban environ- [9] Y. Liu, C. Suo, Z. Liu, and Y. H. Liu, “A multi-sensor fusion based
ments,” International Journal of Robotics Research, vol. 38, no. 6, 2D-Driven 3D object detection approach for large scene applications,”
pp. 642–657, 2019. in Proceedings of the IEEE International Conference on Robotics and
[3] N. Carlevaris-Bianco, A. K. Ushani, and R. M. Eustice, “University Biomimetics, 2019, pp. 2181–2188.
of Michigan North Campus long-term vision and lidar dataset,” [10] T. Moore and D. Stouch, “A generalized extended Kalman filter im-
International Journal of Robotics Research, vol. 35, no. 9, pp. 1023– plementation for the robot operating system,” Advances in Intelligent
1035, 2016. Systems and Computing, vol. 302, pp. 335–348, 2016.
[4] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 year, 1000 [11] J. Zhang and S. Singh, “LOAM: Lidar Odometry and Mapping in
km: The Oxford RobotCar dataset,” International Journal of Robotics Real-time,” in Proceedings of Robotics:Science and Systems, 2014,
Research, vol. 36, no. 1, pp. 3–15, 2017. pp. 9–17.
[5] X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, [12] F. Furrer, M. Fehr, T. Novkovic, H. Sommer, I. Gilitschenski, and
and R. Yang, “The apolloscape dataset for autonomous driving,” in R. Siegwart, Evaluation of combined time-offset estimation and hand-
IEEE Computer Society Conference on Computer Vision and Pattern eye calibration on robotic datasets. Cham: Springer International
Recognition Workshops, 2018, pp. 1067–1073. Publishing, 2017.
[6] A. S. Huang, M. Antone, E. Olson, L. Fletcher, D. Moore, S. Teller, [13] G. Grisetti, R. Kummerle, C. Stachniss, and W. Burgard, “A tutorial
and J. Leonard, “A high-rate, heterogeneous data set from the DARPA on graph-based SLAM,” IEEE Intelligent Transportation Systems
urban challenge,” International Journal of Robotics Research, vol. 29, Magazine, vol. 2, no. 4, pp. 31–43, 2010.
no. 13, pp. 1595–1601, 2010. [14] Z. Liu, S. Zhou, C. Suo, P. Yin, W. Chen, H. Wang, and Y.-H. Liu,
[7] M. Smith, I. Baldwin, W. Churchill, R. Paul, and P. Newman, “The “LPD-Net: 3D point cloud learning for large-scale place recognition
new college vision and laser data set,” International Journal of and environment analysis,” in Proceedings of the IEEE International
Robotics Research, vol. 28, no. 5, pp. 595–599, 2009. Conference on Computer Vision, 2019, pp. 2831–2840.
[8] V. Lepetit, F. Moreno-Noguer, and P. Fua, “Epnp: An accurate o [15] A. V. Segal, D. Haehnel, and S. Thrun, “Generalized-ICP (probailistic
(n) solution to the pnp problem,” International Journal of Computer ICP tutorial),” in Proceedings of Robotics: science and systems, 2009.
Vision, vol. 81, no. 2, p. 155, 2009.

4288

Authorized licensed use limited to: Rutgers University. Downloaded on May 15,2021 at 12:08:17 UTC from IEEE Xplore. Restrictions apply.