See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/374661400

Experimental study on controlling suspended cable-driven parallel robots for

autonomous video-capturing of football games and obtaining the statistics of
the games ✩

Article in Mechatronics · September 2023

DOI: 10.1016/j.mechatronics.2023.103058

CITATIONS READS

0 35

3 authors:

Mohammad Ghanatian Mohammad Reza Hairi Yazdi

University of Tehran University of Tehran
7 PUBLICATIONS 17 CITATIONS 119 PUBLICATIONS 1,642 CITATIONS

SEE PROFILE SEE PROFILE

Mehdi Tale Masouleh

University of Tehran, School of Electrical and Computer Engineering
237 PUBLICATIONS 2,093 CITATIONS

SEE PROFILE

All content following this page was uploaded by Mohammad Ghanatian on 13 December 2023.

The user has requested enhancement of the downloaded file.

 Mechatronics 95 (2023) 103058

Contents lists available at ScienceDirect

Mechatronics
journal homepage: www.elsevier.com/locate/mechatronics

Experimental study on controlling suspended cable-driven parallel robots for

autonomous video-capturing of football games and obtaining the statistics of
the games✩
Mohammad Ghanatian a,b , Mohammad Reza Hairi Yazdi a ,∗, Mehdi Tale Masouleh b
a
School of Mechanical Engineering, University of Tehran, Tehran, Iran
b Human and Robot Interaction Laboratory, School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran

ARTICLE INFO ABSTRACT

Keywords: This research explores the practical application of suspended cable-driven parallel robots for autonomous
Cable-driven parallel robot video-capturing of football games, as well as obtaining the statistics of the games using machine vision and
Autonomous robot deep learning techniques. On this regard, an algorithm for autonomous video-capturing of football games
Football
and extracting the statistics of the games is presented, which is practically implemented on a suspended
Machine vision
cable-driven parallel robot available at the Human and Robot Interaction Laboratory, University of Tehran.
Football statistics
To achieve this, a computer network is established, comprising a Server and a Client connected via Wi-Fi.
The Client computer employs YOLOv5s and machine vision techniques to localize the ball and players on
the football field, which facilitates the real-time calculation required for obtaining the statistics of the games.
These statistics encompass various aspects such as goal count, outs, corners, percentage of ball presence in
each field half, and heatmaps of the players of each team and the ball. Furthermore, the algorithm calculates
the optimal camera position for capturing game footage and transmits this information to the Server system,
which is responsible for controlling the robot. On the Server system, the current and desired positions of the
robot are taken into consideration, and a smooth trajectory is planned between these coordinates using a
septic polynomial, ensuring precise and fluid movements. To facilitate this, a digital controller optimized by
the Simulated Annealing algorithm is designed. Through practical experimentation, this study demonstrates
the viability of introducing autonomy to suspended cable-driven parallel robots for video-capturing of football
games and extracting the statistics of the games.

1. Introduction the desired footage [2]. Leveraging their cable-based structure, CDPRs
offer exceptional speed and acceleration capabilities, surpassing those
Football, widely recognized as the world’s most captivating sport [1], of rigid-link systems [3]. Furthermore, the utilization of long cables
has spurred the development of a wide array of technologies cater- empowers CDPRs to cover large-scale workspaces, even spanning the
ing to the needs of players, spectators, coaches, and leagues. These size of an entire stadium. Moreover, their simple design facilitates
cutting-edge devices and algorithms serve diverse purposes, including straightforward assembly and disassembly procedures [4]. However,
video-capturing, analysis, and live streaming of football games. Within it is worth noting that CDPRs are subject to certain limitations, such
this realm, Suspended Cable-Driven Parallel Robots (CDPRs) have as potential drawbacks in terms of accuracy and the occurrence of
gained prominence as instrumental tools deployed in stadiums world- vibrations [5]. From a theoretical standpoint, the kinematic analysis [6,
wide to capture football games professionally. These robots, equipped 7] and dynamic analysis [8,9] of CDPRs pose significant challenges,
with an end-effector connected to multiple motors via cables, exhibit primarily due to the deformation of the cables and the redundancy
graceful and spider-like maneuvers above the football field, enabling
involved in their structures.
dynamic and agile camera positioning. By commanding the robot’s
SkyCam1 and Spidercam2 have emerged as leading examples of
end-effector, personnel can precisely position the camera to capture
CDPRs that have gained widespread acclaim in stadiums for capturing

✩ This paper was recommended for publication by Associate Editor Chris Manzie.
∗ Corresponding author.
E-mail address: [email protected] (M.R. Hairi Yazdi).
1
http://skycam.tv/.
2
https://www.spidercam.tv/.

https://doi.org/10.1016/j.mechatronics.2023.103058
Received 18 February 2023; Received in revised form 27 August 2023; Accepted 30 August 2023
Available online 19 September 2023
0957-4158/© 2023 Elsevier Ltd. All rights reserved.
M. Ghanatian et al. Mechatronics 95 (2023) 103058

captivating videos of field sports, including football, Super Bowl, and joint actuators than that required to provide the desired end-effector
cricket [10,11]. These state-of-the-art systems offer viewers a unique motion. These advancements in redundancy offer potential solutions for
camera perspective reminiscent of video games, elevating the broadcast the limitations faced by nonredundant parallel manipulators [24–26].
experience of sporting events. The camera is securely mounted on the An example of a study harnessing the benefits of redundantly actuated
end-effector of these robots, which is connected to a camera stabilizer. parallel manipulators is presented in [27], which proposes a novel
The CDPR operates in conjunction with a dedicated controller computer torque optimization method for a 3-DOF redundantly actuated parallel
and relies on the collaborative efforts of two key individuals: the ‘‘Pilot’’ manipulator specifically designed for friction stir welding. This method
and the ‘‘Cameraperson’’. The Pilot assumes control of the end-effector’s considers the minimum actuator torque range, aiming to enhance the
motion within the large-space three-dimensional space of the stadium, control and performance of the manipulator in the welding process.
while the Cameraperson expertly adjusts various parameters such as For CDPRs to be fully constrained, a minimum of 𝑓 + 1 cables is
pan, tilt, zoom, and focus to ensure optimal camera footage. Through required, where 𝑓 represents the number of degrees of freedom of
this synchronized collaboration, professionally crafted videos of foot- the end-effector [28,29]. This redundancy in cable count is crucial to
ball games come to life, enhancing the overall viewing experience for prevent slackness, as CDPR cables can only exert tensile axial forces and
audiences worldwide. no compression [28]. On the other hand, SkyCam exemplifies under-
While human intervention and supervision are currently required constrained robots, as it has fewer cables than its degrees of freedom.
for robot-assisted video-capturing of football games using suspended Although this results in a simpler structure, it introduces unwanted
CDPRs, the field of robotics has seen significant advancements in oscillations due to the extra degrees of freedom in the end-effector [30].
the development of fully autonomous systems for various industrial, When using CDPRs for video recording purposes, the camera should
service, and field applications. Therefore, one of the main concerns be attached to a stabilizer to compensate for the vibrations caused by
of this research is to enhance CDPRs by introducing autonomy into cable movements. The effectiveness of compensation depends on the
their video-capturing capabilities, thereby expanding their potential. performance quality of the installed camera stabilizer, and efforts have
Autonomous robotic systems have found widespread application in been made to reduce these swaying motions. In fact, vibration analysis
different domains; for instance, in the agricultural sector, researchers is a challenge addressed in CDPR research. In [31], the study focuses
have developed a robotic system for harvesting cucumbers in green- on the vibration analysis of general 6-DOF CDPRs. By analyzing the
houses, demonstrating an impressive 80% success rate in autonomous system’s natural frequencies, the research demonstrates the possibility
cucumber picking [12]. In warehouse environments, an autonomous of designing a cable manipulator with sufficient stiffness for specialized
robotic system for inventory management has been devised, eliminating applications, such as hardware-in-the-loop simulation of contact dy-
the need for external infrastructure [13]. The cleaning of photovoltaic namics. Additionally, [32] tackles vibration-related challenges in large
panels in desert zones has been addressed through the proposal of an CDPRs by developing dynamic models which consider the distributed
autonomous robotic system [14]. Furthermore, autonomous robots for mass and potential length variation of each cable.
parcel delivery in urban areas have been explored and studied [15], On the other hand, [33] presents a geometrically exact analytical
while autonomous mobile robots for firefighting purposes in industrial model for CDPRs operating in three-dimensional space. This model
settings have also been developed [16]. accurately determines the positioning of a rigid body by consider-
As it is evident from the above examples, the field of autonomy in ing factors such as cable mass, elasticity, and sag. In [34], a para-
robotics has witnessed successful implementations across various do- metric nonlinear model of cable-driven parallel manipulators with
mains, presenting an opportunity to integrate autonomous features into a three-dimensional end-effector is formulated, considering the dis-
CDPRs for video-capturing of football games. While CDPRs have a long tributed stiffness, inertia, and damping of time-varying length cables.
history, the exploration of adding autonomy specifically for capturing This model allows for the study and characterization of the dynamic
football games remains unexplored, to the best of the authors’ knowl- response of manipulators equipped with an arbitrary number of cables,
edge. The inception of CDPRs dates back to the late 1980s with the de- addressing both the direct and inverse dynamic problems in the case
velopment of the parallel link ROBOCRANE by the National Institute of of minimally-actuated and over-actuated CDPRs. Another challenge
Standards and Technology (NIST). This pioneering CDPR was designed faced by CDPRs is the limited accuracy in position estimation, both
for tasks such as cutting, excavating, lifting, and positioning [17]. in joint space and task space. In order to address this issue, a vision-
Over time, as technologies advanced, novel applications for CDPRs based method is proposed in [35] which enables cable-driven robots to
have emerged. For example, CDPRs are deployed in the Five-Hundred simultaneously measure their manipulator configuration and the pose
Meter Aperture Spherical Radio Telescope (FAST),3 the world’s largest of the target object by detecting Apriltag 2D codes and leveraging the
single-dish radio telescope. FAST utilizes a CDPR with six cables to kinematic model of cable-driven robots.
maneuver its feed support system over the telescope’s reflector [18]. The analysis and statistics of football games have become increas-
CDPRs have found notable applications in the field of rehabilitation, ingly important in recent years. They provide valuable insights for
including waist rehabilitation [19], and in the construction industry team managers in areas such as strategy planning, team selection, and
for large-scale 3D printing [20]. They have also demonstrated their player lineup on the field [36]. Companies like Opta,4 Wyscout,5 and
potential in reconstructing 3D graphical models of objects [21,22] and StatsBomb6 have emerged as prominent vendors, offering game statis-
proposed as continuous backbone robots for dexterous manipulation in tics including data on passes, shots, and cards [37]. These statistics,
manufacturing and space environments [23]. often annotated manually in football videos, are highly beneficial for
CDPRs belong to the family of parallel manipulators, which offer football spectators, coaches, leagues, and investors. However, many
various benefits such as high payload capability and low inertia. How- of the existing technologies used for game analysis and statistical
ever, parallel robots often face limitations such as small workspace and gathering, such as multiple cameras, wearable sensors, and GPS de-
singularities, which hinder their industrial application. To overcome vices, can be prohibitively expensive and inaccessible for smaller stadi-
these challenges, researchers have focused on redundantly actuated ums [38]. Additionally, there are controversial instances like offside
parallel manipulators, which can be achieved through kinematic re- calls where the use of technology for reliable information detection
dundancy or actuation redundancy. Kinematic redundancy involves becomes crucial [39].
manipulators having more degrees of freedom than necessary for per-
forming a specified task, while actuation redundancy introduces extra
4
https://www.statsperform.com/opta/.
5
https://wyscout.com/.
3 6
https://fast.bao.ac.cn/. https://statsbomb.com/.

2
M. Ghanatian et al. Mechatronics 95 (2023) 103058

Given these needs, the availability of affordable and widely accessi-

ble devices and algorithms for football game analysis holds significant
value, particularly for smaller stadiums and leagues with limited re-
sources. Fortunately, there is a wealth of research on machine vision
and machine learning algorithms focused on ball and player detection
in football fields which can be further explored and expanded upon
to develop cost-effective and autonomous solutions for extracting the
statistics of football games. These algorithms offer promising avenues
for detecting the ball and players on football fields, forming the basis
for future research endeavors. For example, a study by [40] introduces
a support vector machine-based algorithm for goal detection using
image data, showcasing its potential to assist referees in accurately
identifying goals. In another research endeavor, [41] proposes a deep
convolutional network-based model specifically designed for ball de-
tection in long-shot football videos. Furthermore, [42] presents an
integrated player and ball detection algorithm utilizing deep neural
networks, which surpasses the processing speed of the Faster-RCNN
algorithm. By building upon these existing algorithms and expanding
their capabilities, this research aims to develop novel and efficient ap-
proaches for autonomous and cost-effective extraction of the statistics
of the games. Leveraging the advancements in machine vision and deep Fig. 1. Overview of the system proposed for video-capturing of football games and
learning, the goal is to democratize access to game analysis tools and obtaining the statistics of the games autonomously.
empower stadiums and leagues of all sizes with invaluable insights
derived from comprehensive and affordable statistical data.
In summary, this research endeavors to pioneer an innovative al-
gorithm as a fundamental step towards achieving autonomous video-
capturing of football games using suspended CDPRs. The primary objec-
tive is to enable these robots to autonomously capture football games,
leading to significant reductions in operational costs and establishing a
well-defined framework for the video-capturing process. However, this
study goes beyond conventional video-capturing methods by automat-
ing the extraction of the statistics of the games. This statistical analysis
includes metrics such as goal counts, corner kicks, and outs, along with
the generation of heatmaps that visualize the positioning of the ball and
players of each team. Notably, this statistical analysis is accomplished Fig. 2. The fixed camera and the moving camera utilized in this research.

autonomously, eliminating the need for manual annotation by human

operators.
The rest of the paper is structured as follows. Section 2 provides an shown in Fig. 2, is positioned beside the football field and serves the
overview of the proposed system, including the experimental setup and purpose of localizing the ball and players on the field. On the other
hardware utilized. In Section 3, the machine vision and deep learning hand, the Moving Camera, mounted on the robot’s end-effector, is
algorithms developed for this research are presented. Section 4 intro- responsible for capturing video footage of football games. The Fixed
duces the algorithm developed for obtaining the statistics of football Camera utilized in this research is the RGB camera of a Microsoft Xbox
games. The control and trajectory planning algorithms utilized in this 360 Kinect Sensor. It records videos with a resolution of 640 × 480
research are discussed in Section 5. The algorithm for autonomous pixels at a frame rate of 30 Frames Per Second (FPS). In contrast, the
video-capturing of football games, as presented in this work, is detailed Moving Camera offers a higher resolution of 2336 × 1080 pixels and
in Section 6. The obtained results from the research are reported in operates at the same frame rate of 30 FPS. Mounted on the end-effector
Section 7. Finally, Section 8 presents the most important conclusions of the robot, it enables dynamic video recording during football games.
derived from this study and provides some hints for ongoing works. To overcome the potential issue of vibrations and oscillations that may
affect the quality of the captured videos, the moving camera is mounted
2. System overview and experimental setup on a DJI Osmo Mobile 2 camera stabilizer, which is attached to the
bottom of the end-effector. This setup, illustrated in Fig. 2, ensures
This section provides an overview of the proposed system and intro- smoother and more stable video recording.
duces the hardware utilized for implementing the concept. The football The localization of the ball and players on the football field is
game is simulated in a scaled miniature 340 × 286 cm2 football field at performed by the fixed camera, which is connected to a laptop system.
the Human and Robot Interaction Laboratory, University of Tehran, as Deep Learning and Machine Vision techniques are employed in this
depicted in Fig. 1. The players of each team are represented by colored system for localization purposes. The laptop used for this research is
balls, which are manipulated by one or a couple of individuals standing an ASUS N550JK device, featuring a CORE i7 [email protected] GHz CPU,
beside the football field and holding a long stick. The red and blue balls an NVIDIA GEFORCE GTX 850M GPU, and 8 GB of RAM. Through this
symbolize the players of the RED and BLUE teams, respectively. Each system, all the players and the ball on the football field are localized,
team comprises two players, and a yellow ball represents the football in and the obtained data is processed to extract the statistics of the game.
play. The objective is to capture videos of the football game conducted Furthermore, this data is utilized to plan the motions of the suspended
on the field using the suspended CDPR constructed at this laboratory CDPR, enabling the moving camera to autonomously capture video
and also obtain the statistics of the football games automatically using footage of the football games.
machine vision and deep learning techniques. The robot utilized for autonomous video-capturing in this research
In the proposed setup, as depicted in Fig. 1, two cameras are is the suspended CDPR, developed at the Human and Robot Interac-
employed: a ‘Fixed Camera’ and a ‘Moving Camera’. The Fixed Camera, tion Laboratory, University of Tehran [43]. The robot’s dimensions

3
M. Ghanatian et al. Mechatronics 95 (2023) 103058

data processing to derive the statistics of the games. The calculated

data for autonomous video-capturing of the game are then transmitted
to the Server system. The Server system, responsible for controlling
the robot, receives the data from the Client system. It uses this in-
formation to orchestrate the movements of the robot’s end-effector
camera during the game. By following the prescribed instructions, the
end-effector camera autonomously captures the desired video footage
of the game. This collaborative setup between the laptop system and
the controller computer ensures that image processing, data analysis,
and robot control are seamlessly integrated, enabling the autonomous
video-capturing of football games.

3. Machine vision
Fig. 3. The dimensions of the robot and the scaled miniature football field at the
human and robot interaction laboratory, University of Tehran.
All machine vision and deep learning algorithms in this research
are executed on the client system. The fundamental machine vision
algorithms are implemented using the Python OpenCV library, while
the deep learning algorithms are implemented using PyTorch. Initially,
the Kinect sensor is connected to the client system, which operates on
the Windows 10 operating system. The necessary register files of the
Microsoft Xbox 360 Kinect Sensor are installed on the laptop, enabling
the sensor to function as a webcam for the Windows software.
However, utilizing the Kinect sensor with the OpenCV library in
Python requires additional configuration settings during the library
installation process. In this research, an alternative method is employed
to access the Kinect sensor’s RGB camera in OpenCV. To achieve this, a
third-party software called ManyCam7 is installed on the client system,
creating a virtual camera from the Kinect sensor. This virtual camera
can then be accessed through the OpenCV library as well.
For the detection of the ball and players on the field, a YOLOv5s
deep learning model is employed in this study. This model accurately
localizes the ball and players in real-time on each frame of the captured
videos. YOLO models are chosen over R-CNN or Fast R-CNN algorithms
due to their improved speed and accuracy, as well as the advantage
of performing the localization process within a unified architecture
and single network [45]. Among the various YOLO versions available,
YOLOv5s [46] demonstrates superior real-time performance compared
to previous iterations, aligning with the specifications of the GPU
utilized in the client system.
Fig. 4. The hardware of the suspended cable-driven robot used in this research.

3.1. Preparing training data

are illustrated in Fig. 3, and its kinematic and dynamic models are To utilize the YOLOv5s model for detecting the ball and players
described in detail in [44]. The robot, as depicted in Fig. 4, consists on football fields, it is essential to train the model with appropriate
of an end-effector and four motors, each connected to a winch through training data. In this study, two videos were recorded from the football
a coupling mechanism. Cables are used to link the motors to the end- field using the Fixed Camera while changing the positions of the
effector, and by rotating the motors, the length of the cables can be ball and players. These videos can be accessed through VIDEO1 and
adjusted, enabling controlled movements of the end-effector within VIDEO2. Using the Python OpenCV library, these videos were read, and
the robot’s workspace. The suspended CDPR is equipped with four frames with a resolution of 640 × 480 pixels were saved on the client
FD422 Kinco AC Servo SMH80S-0075 servo motors, with each motor computer at intervals of 5 s and 4 s, respectively.
connected to an FD422-CA-000 Kinco driver. The drivers are connected Consequently, a total of 197 and 47 images were obtained from
to a Personal Computer system using RS232 cables through a CP- these two videos, resulting in a collection of 244 images exhibiting
114 UL MOXA PCI Express Board. The computer, equipped with an different ball and player positions. In the subsequent stage, these 244
Intel Core [email protected] GHz CPU and 6 GB of RAM, is responsible for the images were labeled manually using the labelImg8 software. Each ball
robot’s control. For a comprehensive overview of the robot’s hardware was labeled with one of the following color labels: Blue, Red, or Yellow,
components and their interconnections, please refer to Fig. 4. corresponding to the players of the BLUE team, the players of the RED
The autonomous video-capturing of football games involves the team, and the ball, respectively. An example of the labeling process for
coordination of two computers: the laptop system and the controller one of these 244 images is illustrated in Fig. 5.
computer. These computers are interconnected through a Wi-Fi net- Following the labeling process, the labeled images were uploaded
work, with the laptop system acting as the Client computer and the to the Roboflow9 website, where the data was further processed. The
controller computer serving as the Server system. The establishment images were divided into Train, Test, and Validation sets according
of this computer network is achieved using Socket programming in
Python, enabling seamless communication between the two systems
over Wi-Fi. On the laptop computer, which functions as the Client sys- 7
https://manycam.com/.
tem, all image processing and deep learning algorithms are executed. 8
https://github.com/heartexlabs/labelImg.
9
This includes tasks such as ball and player detection, localization, and https://roboflow.com.

4
M. Ghanatian et al. Mechatronics 95 (2023) 103058

Table 1 Table 2
The distribution of Train, Test, and Validation data. The localized objects in an example frame shown as a DataFrame.
Type of data Number Percentage 𝑥𝑚𝑖𝑛 𝑦𝑚𝑖𝑛 𝑥𝑚𝑎𝑥 𝑦𝑚𝑎𝑥 Confidence Class Name
Training 191 78% 0 196.701355 121.083549 217.520294 141.997421 0.899140 2 Yellow
Test 24 10% 1 286.637756 173.246170 312.285400 199.200882 0.876054 1 Red
Validation 29 12% 2 105.749420 104.382332 126.628708 125.007935 0.873267 0 Blue
Total 244 100% 3 415.950928 136.781555 438.657318 159.927994 0.867834 1 Red
4 322.408051 74.365807 339.842712 92.662987 0.850683 0 Blue

Fig. 5. The labels of different objects in one of the frames in labelImg software.

Fig. 7. An example of one of the frames given to the YOLO model and the localized
objects.

this research, a confidence score threshold of 75% is applied during the

detection process to ensure that only objects with a confidence level
above 75% for one of the considered classes are detected.
In the subsequent step, the obtained data from the YOLO model,
including the bounding boxes, classes, and confidence scores, are stored
Fig. 6. Performance of the training algorithm of YOLOv5s used in this research.
in a Pandas DataFrame for each frame. An example frame is depicted in
Fig. 7, while Table 2 illustrates the corresponding sample DataFrame.
In this study, to simplify the object location estimation, the center
to the following distribution: 78% (191 samples) were allocated as of each detected object’s bounding box is considered as its location.
Training data, 10% (24 samples) were set aside as Validation data, Specifically, the 𝑋 and 𝑌 coordinates of each object are determined as
and the remaining 12% (29 samples) were designated as Test data. The follows:
distribution of the data sets can be seen in Table 1.
𝑥min + 𝑥max 𝑦min + 𝑦max
3.2. Training the YOLO model 𝑋= , 𝑌 = (1)
2 2
To calculate the statistics of the games, it is necessary to convert
In the subsequent phase, the YOLOv5s model is trained using the
the coordinates obtained from the YOLO model and Eq. (1) from the
PyTorch library in Python within the Google Collaboratory environ-
image coordinate system (denoted as {A} in Fig. 8) to the football field
ment. The training procedure is conducted for 300 epochs with a
coordinate system (denoted as {B}). This coordinate transformation can
batch size of 16. The employed YOLO model in this study consists of
be achieved using the following relation, where 𝑥𝑖 and 𝑦𝑖 represent the
213 layers and 7,018,216 parameters, which are optimized during the
object coordinates in the image coordinate system, 𝑥𝑖 ′ and 𝑦𝑖 ′ denote
training process. Fig. 6 illustrates the performance plots of the training
the object coordinates in the football field coordinate system, and 𝑡𝑖 is
algorithm, demonstrating that the 𝑚𝐴𝑃 _0.5 metric reaches 0.995 and
a free parameter:
the 𝑚𝐴𝑃 _0.5 ∶ 0.95 metric reaches 0.702 after 300 epochs. These indices
validate the effectiveness of the trained object detection algorithm, and
consequently, the trained model is saved as a .pt file to be utilized in ⎡𝑡𝑖 𝑥′𝑖 ⎤ ⎡𝑥𝑖 ⎤
⎢ ′⎥
subsequent sections of this research, where real-time localization of the ⎢𝑡𝑖 𝑦𝑖 ⎥ = 𝐌m ⎢⎢𝑦𝑖 ⎥⎥ (2)
ball and players in each frame of the streaming videos is required. ⎢ ⎥ ⎣1⎦
⎣ 𝑡𝑖 ⎦
3.3. Localizing objects in football field The transformation matrix 𝐌𝑚 is a 3 × 3 matrix calculated using the
getPerspectiveTransform() function from the OpenCV library
The localization of the ball and players on the football field is in Python. This function takes the coordinates of four points in the {A}
achieved using the trained YOLOv5s model. The Kinect sensor videos coordinate system from an image and their corresponding positions in
are streamed to the Client system, and each frame is fed as input to the the {B} coordinate system as input. It then calculates 𝐌𝑚 as the output,
YOLOv5s model. The model then outputs the bounding boxes, classes ensuring that these four points satisfy the relation defined by Eq. (2).
(Blue, Red, Yellow), and confidence scores for all detected objects. In The element at position (3,3) in 𝐌𝑚 is always 1, resulting in a matrix

5
M. Ghanatian et al. Mechatronics 95 (2023) 103058

Fig. 8. The image coordinate system {A} and the football field coordinate system {B}.
Fig. 10. The goal area in the football field, which are shown as the red and blue areas.

the boundaries of the goal area, it can be determined that a goal has
occurred. In the experimental setup utilized in this research, as shown
in Fig. 9, a challenge was encountered where the goal area appeared
small compared to the size of the ball in the miniature football field.
This size ratio made goal detection a difficult process. To address this
challenge, it was assumed that the entire penalty area represents the
goal area, as depicted in Fig. 10. In this configuration, whenever the
yellow ball enters the penalty area, a goal is recognized for the team
in the front. It is important to note that the goal area size was adjusted
for this specific experimental setup, and this challenge will not be
encountered in real-size football fields.
The goal detection algorithm developed in this research works as
follows: A goal is detected when the ball is recognized to be outside
the goal area for 2 consecutive frames and subsequently inside the goal
area for 5 consecutive frames. This approach ensures the robustness of
the goal detection algorithm, especially when the ball is near the goal
Fig. 9. The number of goals, outs, and corners of each team are shown on the screen area but has not yet crossed the goal boundaries. The algorithm for
in real-time. detecting goals is presented in Algorithm 1.

with eight unknown elements. With the known 𝑥 and 𝑦 coordinates of

the four points in both the {A} and {B} coordinate systems, there are 4.2. Detection of outs
a total of eight equations. By solving this system of linear equations,
the 𝐌𝑚 matrix can be easily calculated, enabling the transformation of In order to detect outs in football games, it is important to determine
object locations from the image coordinate system to the football field which team’s player kicked the ball out of the field. However, in this
coordinate system. research, since the ball and the players are manually moved by a
4. Obtaining statistics person beside the football field, it is not possible to directly detect
the possession of the ball. To address this limitation, an assumption is
This research involves two processes for obtaining the statistics of made: when the ball passes the longitudinal boundaries of the football
the games. In the first process, real-time detection of goals, outs, and field from the RED team’s half-field, it is considered that a player
corners is performed. These occurrences are displayed on the screen, as from the RED team kicked the ball out, resulting in an out in favor
shown in Fig. 9, and a voice message is played to announce each event of the BLUE team. Similarly, when the ball goes out from the BLUE
to the match spectators. In the second process, the location of the ball team’s half-field, it is assumed that a player from the BLUE team kicked
and the players is stored in a Pandas DataFrame. This information is the ball out, resulting in an out in favor of the RED team. The out
utilized at the end of each game to calculate the heatmap of the ball and boundaries are depicted in Fig. 11.
players for each team on the football field. Additionally, the percentage
In the developed algorithm, the occurrence of an out is detected
of ball presence in each half of the field is determined.
by checking if the ball is inside the field in two consecutive frames and
4.1. Detection of goals subsequently outside the field in five consecutive frames. This approach
ensures the accurate and robust detection of outs. The pseudocode
The detection of goals relies on the accurate identification of the for detecting outs, combined with the corner detection algorithm, is
ball’s position on the football field. When the ball is found within presented in the following subsection.

6
M. Ghanatian et al. Mechatronics 95 (2023) 103058

Algorithm 1 Goal Detection

1: 𝑔𝑜𝑎𝑙𝐵𝐿𝑈 𝐸𝑐𝑜𝑢𝑛𝑡 ← 0
2: 𝑔𝑜𝑎𝑙𝑅𝐸𝐷𝑐𝑜𝑢𝑛𝑡 ← 0
3: 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 ← 0
4: for all incoming video frames do
5: increment 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟
6: for all objects in each frame do
7: if object is ball then
8: if object is inside the BLUE goal boundaries
from 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 to 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 4 frames then
9: if object was outside of BLUE goal boundaries from
𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 5 to 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 6 frames then
10: increment 𝑔𝑜𝑎𝑙𝐵𝐿𝑈 𝐸𝑐𝑜𝑢𝑛𝑡
11: end if
12: end if
13: if object is inside the RED goal boundaries
from 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 to 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 4 frames then
14: if object was outside of RED goal boundaries from
𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 5 to 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 6 frames then Fig. 11. The boundaries of outs in the football field in this research.

15: increment 𝑔𝑜𝑎𝑙𝑅𝐸𝐷𝑐𝑜𝑢𝑛𝑡

16: end if
17: end if
18: end if
19: end for
20: end for

4.3. Detection of corners

Corners occur when players kick the ball out from the transverse
boundaries of their own half-field. It is important to note that if the
ball is kicked out by the players of the opposing team, it results in an
out, not a corner. Although possession of the ball is typically considered
in detecting corners, in this research, where the ball and players are
manually moved by individuals using a long stick, the concept of
possession is disregarded. Therefore, the assumption is made that when
the ball crosses the transverse boundaries of the field, it was kicked by
the players of the same field. Consequently, if the ball goes out from the
Fig. 12. The defined boundaries of corners of each team.
BLUE team side, it is assumed that the players of the BLUE team kicked
the ball out, resulting in a corner in favor of the RED team. Similarly,
when the ball goes out from the RED team’s half-field, it is assumed
that the corner is awarded to the BLUE team. The corner boundaries from the BLUE team. Similarly, ⟨R1x⟩⟨R1y⟩ and ⟨R2x⟩⟨R2y⟩ indicate the
for each team can be observed in Fig. 12. coordinates of the two players from the RED team.
The detection of corners involves checking the ball’s position rel-
ative to the field boundaries. In this research, a corner is detected In order to visualize the heatmaps of the ball and players, a Graph-
when the ball remains inside the field for two consecutive frames and ical User Interface (GUI) is developed using the matplotlib library in
then, right after that, goes outside the transverse boundaries for five Python. The seaborn library is used to create the heatmaps, employing
consecutive frames. This pattern signifies that the ball has crossed the Kernel Density Estimation (KDE) [47] to estimate the Probability Den-
transverse boundaries of the field, indicating a corner occurrence. The sity Function of the ball and player locations stored in the DataFrame.
algorithm developed for detecting outs and corners in this research is As an example, the heatmaps for a sample football game, which can be
outlined in Algorithm 2. viewed through THIS LINK, are calculated and plotted in Fig. 13 (ball),
Fig. 14 (BLUE team players), and Fig. 15 (RED team players).
4.4. Heatmaps of the ball and the players
5. Control and trajectory planning

To gather the necessary data for calculating the heatmaps of the ball In this section, the control and trajectory planning algorithms imple-
and players, as well as the percentage of ball presence in each half of mented on the robot are discussed. The control algorithm is responsible
the field, a DataFrame is utilized which includes the following columns: for ensuring precise robot movement with minimal error, while the
trajectory planning algorithm generates the motion profile that serves
⟨Time⟩⟨Yx⟩⟨Yy⟩⟨B1x⟩⟨B1y⟩⟨B2x⟩⟨B2y⟩⟨R1x⟩⟨R1y⟩⟨R2x⟩⟨R2y⟩ as the reference signal for the controller. In a previous study [48],
it was demonstrated that the trajectory planning algorithm plays a
The ⟨Time⟩ column stores the elapsed time, while ⟨Yx⟩ and ⟨Yy⟩ crucial role in reducing oscillations in the end-effector of suspended
represent the 𝑥 and 𝑦 coordinates of the ball. The columns ⟨B1x⟩ and CDPRs. This reduction is particularly important when the robot is
⟨B1y⟩ correspond to the coordinates of one player from the BLUE used for video-capturing purposes, as oscillations can negatively impact
team, and ⟨B2x⟩ and ⟨B2y⟩ represent the location of the second player the smoothness of captured videos. Although camera stabilizers can

7
M. Ghanatian et al. Mechatronics 95 (2023) 103058

Algorithm 2 Out and Corner Detection

1: 𝑜𝑢𝑡𝑅𝐸𝐷𝑐𝑜𝑢𝑛𝑡 ← 0
2: 𝑜𝑢𝑡𝐵𝐿𝑈 𝐸𝑐𝑜𝑢𝑛𝑡 ← 0
3: 𝑐𝑜𝑟𝑛𝑒𝑟𝑅𝐸𝐷𝑐𝑜𝑢𝑛𝑡 ← 0
4: 𝑐𝑜𝑟𝑛𝑒𝑟𝐵𝐿𝑈 𝐸𝑐𝑜𝑢𝑛𝑡 ← 0
5: 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 ← 0
6: for all incoming video frames do
7: increment 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟
8: for all objects in each frame do
9: if object is ball then
10: if Y of object is outside the field boundaries
from 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 to 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 4 frames then
11: if object was inside the field boundaries from
𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 5 to 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 6 frames then
12: if X of object is on RED field then
13: increment 𝑜𝑢𝑡𝑅𝐸𝐷𝑐𝑜𝑢𝑛𝑡
14: else
15: increment 𝑜𝑢𝑡𝐵𝐿𝑈 𝐸𝑐𝑜𝑢𝑛𝑡
16: end if
17: end if
18: end if Fig. 13. Heatmap of the ball in a sample game.
19: if X of object is outside the field boundaries
from 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 to 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 4 frames then
20: if object was inside the field boundaries from
𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 5 to 𝑓 𝑟𝑎𝑚𝑒𝑁𝑢𝑚𝑏𝑒𝑟 − 6 frames then
21: if X of object is less than RED field boundaries then
22: increment 𝑐𝑜𝑟𝑛𝑒𝑟𝑅𝐸𝐷𝑐𝑜𝑢𝑛𝑡
23: else
24: increment 𝑐𝑜𝑟𝑛𝑒𝑟𝐵𝐿𝑈 𝐸𝑐𝑜𝑢𝑛𝑡
25: end if
26: end if
27: end if
28: end if
29: end for
30: end for

help mitigate oscillations, their effectiveness may vary, and residual

oscillations can still affect video quality. Therefore, designing control
and trajectory planning algorithms which effectively reduce oscillations
can complement the camera stabilizer’s efforts in minimizing vibrations
affecting the camera.
Fig. 14. Heatmap of the players of the team BLUE in a sample game.
In this research, the autonomous video-capturing of football games
using suspended CDPRs incorporates control and trajectory planning al-
gorithms outlined in [48]. The robot’s servo motors operate in velocity
mode, with control feedback provided by the internal encoders of each
motor. Consequently, the motors receive velocity commands as inputs
from the computer, while the output is the angular shaft angle of each
motor; therefore, mathematically, each motor can be modeled as an
integrator. In [48], a PID controller was utilized in the continuous time
domain, and the 𝐾𝑃 , 𝐾𝐼 , and 𝐾𝐷 gains of the controller were optimized
using the Simulated Annealing algorithm. The optimization aimed to
minimize the summation of the Integral Time Absolute Error (ITAE) of
the four motors. The PID controller and ITAE criteria are as follows:

𝑡
𝜃̇ = 𝐾𝑃 𝑒 + 𝐾𝐷 𝑒̇ + 𝐾𝐼 𝑒𝑑𝑡 (3)
∫0
𝑡
ITAE = 𝑡|𝑒|d𝑡 (4)
∫0
When implementing the controller on the robot hardware, it is
important to consider its digital representation. The closed-loop con-
tinuous system bandwidth should be kept below 1/20 of the hardware
sampling frequency to ensure stability of the discretized controller [49]. Fig. 15. Heatmap of the players of the team RED in a sample game.

8
M. Ghanatian et al. Mechatronics 95 (2023) 103058

Fig. 16. The trajectory planning and the control algorithm utilized to control the robot
for video-capturing purposes.

However, practical tests conducted in [48] revealed that the digital con-
troller remains stable even when the closed-loop bandwidth is within
the range of 1/20 to 1/15 of the sampling frequency. Therefore, the
PID controller gains are optimized to ensure that the continuous closed-
loop system bandwidth is less than 1/15 of the digital system sampling
frequency. Another crucial aspect is the placement of the closed-loop Fig. 17. An overview of the system developed in this research for autonomous
system poles, which should be positioned in the left-half plane, far away video-capturing of the game.
from the origin. To satisfy this requirement, an additional constraint is
imposed on the optimization problem, stipulating that the real part of
the closed-loop poles should be less than -2. In this research, the robot in this section, as depicted in Fig. 16, the motors exhibit smooth angular
hardware has a maximum sampling frequency of 20 Hz. Taking into position profiles, characterized by zero jerk, acceleration, and velocity
account the objective function and the aforementioned constraints, the at the start and end of motions. These two algorithms, briefly described
optimization problem can be formulated as follows: in this section, will be employed in the Server computer utilized in this
research to govern the robot’s movements.
( )
min ITAE1 + ITAE2 + ITAE3 + ITAE4 It is noteworthy that the authors have conducted comprehensive
{ research on the tracking performance of the robot through the in-depth
Bandwidth(closed − loop) < 20
15
(5)
analysis of the combination of the employed control and trajectory
s.t.
Real(Poles(closed − loop) < −2 planning algorithm, as documented in [48]. Referring to this publica-
tion is encouraged for more detailed insights into the robot’s tracking
This optimization problem is solved using the Simulated Annealing
capabilities and performance.
algorithm for an arbitrary reference signal, and after 1300 iterations,
the optimal PID gains are found to be 𝐾𝑃 = 6.13, 𝐾𝐼 = 19.78, 𝐾𝐷 = 6. Autonomous video-capturing
0.1 [48]
As mentioned earlier, the trajectory planning algorithm plays a In this section, an algorithm for autonomous video-capturing of
crucial role in minimizing the oscillations experienced by the end- football games is presented based on the localization of the ball and
effector of the suspended CDPR. In [48], a practical implementation of players on the field. The ball’s location is of utmost importance to
a trajectory planning algorithm is conducted at the Human and Robot football spectators and for capturing game footage [51]. The reason
Interaction Laboratory, University of Tehran, where its effectiveness is that the ball’s movement plays a crucial role in determining the
in reducing end-effector oscillations is investigated. This trajectory game’s flow, and the players around the ball draw significant attention.
planning algorithm, which ensures zero jerk, acceleration, and velocity Therefore, in this research, the focus is on video-capturing the ball and
at the start and end of motions, is also utilized in this research for its surrounding environment using the Moving Camera. To achieve this,
autonomous video-capturing of football games. The algorithm involves the football field is divided into multiple grids, as depicted in Fig. 18.
the conversion of the initial and final positions of the robot in task space In each frame, the grid containing the ball is determined, allowing to
into the joint space by solving the Inverse Kinematics Problem (IKP). calculate the optimal position for the end-effector camera. Utilizing the
Additionally, a Septic polynomial is fitted to the joint space between trajectory planning and control algorithms described in Section 5, it is
the initial and final desired positions, subject to certain boundary ensured that the robot smoothly moves from its initial position to the
conditions described as follows [50]: desired location, enabling autonomous video-capturing of the games.
By prioritizing the ball’s location and incorporating precise camera po-
sitioning, the proposed algorithm guarantees that the captured videos
∑
8
encompass the essential aspects of the game. This approach enhances
𝑃 (𝑡) = 𝐴𝑖 𝑡𝑖
𝑖=1 the viewing experience for spectators and provides valuable footage for
analysis and review.
𝑃 (0) = 𝜃initial , 𝑃 (𝑡𝑓 ) = 𝜃f inal As mentioned in Section 2, the real-time calculations are executed
′
on a Client–Server network, establishing a connection via Wi-Fi us-
𝑃 (0) = 0 , 𝑃 ′ (𝑡𝑓 ) = 0 ing Socket programming in Python. The image processing and deep
′′
(6)
𝑃 (0) = 0 , 𝑃 ′′ (𝑡𝑓 ) = 0 learning algorithms are exclusively conducted on the Client system,
𝑃 ′′′ (0) = 0 , 𝑃 ′′′ (𝑡𝑓 ) = 0 enabling the prompt computation of the statistics of the games and
determining the optimal position for the Moving Camera. Simultane-
The coefficients 𝐴1 , 𝐴2 , 𝐴3 , … , 𝐴8 in this algorithm can be deter- ously, the Client system transmits the desired camera position to the
mined by solving a system of seven linear equations with seven un- server system, which is the computer responsible for controlling the
knowns. This computation is performed after converting the initial and robot. The server system processes the received data, plans the robot’s
final positions in task space to the joint space through the solution of movements, and dispatches the corresponding control signals utilizing
the Inverse Kinematics Problem (IKP). Subsequently, the joint space ref- the digital controller outlined in Section 5. The overall architecture of
erence trajectory is provided to the optimized digital PID controller. By the proposed system is illustrated in Fig. 17, providing a comprehensive
implementing the control and trajectory planning algorithms outlined overview of its functionality and interaction between the components.

9
M. Ghanatian et al. Mechatronics 95 (2023) 103058

Fig. 18. The detected position of the ball and the calculated position of the camera,
indicated in the user interface designed in this research.

6.1. Processes performed on the client system

Fig. 19. The practical experiments performed to assess the proposed algorithm.

The miniature football field at the Human and Robot Interaction

Laboratory, University of Tehran is virtually divided into 5 equal parts Table 3
Performance of the trained YOLOv5s on the validation data.
in both length and width, resulting in a total of 25 grids within the
Class Images Instances Precision Recall [email protected] [email protected]:.95
field area. This division can be visualized in Fig. 18, which depicts the
Graphical User Interface (GUI) developed in this research and displayed All 29 125 0.997 1 0.995 0.725
on the Client computer. In the GUI, the ball’s location is represented Blue 29 51 0.996 1 0.995 0.739
Red 29 50 0.999 1 0.995 0.714
by a yellow circle, while the desired camera position is indicated by
Yellow 29 24 0.996 1 0.995 0.723
a black square. Based on the ball’s location on the football field, the
corresponding grid is determined, and the algorithm calculates the
desired camera position to ensure that the ball and its surrounding Table 4
area remain visible in the frames captured by the Moving Camera. This The statistics of the game extracted from the game in THIS LINK.
enables the robot to autonomously adjust its position to capture the Item RED team Blue team
game footage effectively. Goal 0 1
In this research, a strategy is adopted to determine the desired Out 1 0
location of the Moving Camera. It is decided that the camera should Corner 1 0
Presence 0% 100%
be positioned at the center of the adjacent grid on the right side of
Duration 00:01:22
the grid where the ball is located. Fig. 18 provides an example where
the ball is detected in the (3,1) grid cell, leading to the calculation
of the camera’s location in the center of the (4,1) grid. This approach
allows the algorithm to capture football game videos effectively for all 7. Results and discussion
grids except the five grids located in the rightmost column. For these
particular grids, the camera can be positioned on the adjacent left grid. This research focused on enhancing the capabilities of Suspended
Whenever the ball transitions from one grid to another, if it remains CDPRs for autonomous video-capturing of football games and extract-
on the new grid for more than 10 frames, a new message is sent to ing the statistics of the games. To achieve this, a YOLOv5s model was
the Server computer containing the coordinates of the desired location trained with a performance detailed in Table 3, to detect the ball and
for the Moving Camera. This ensures continuous tracking of the ball’s players on the football field. Through the integration of deep learning
movement and timely adjustment of the camera’s position to capture and machine vision techniques, the positions of the ball and players
the game footage accurately. were localized in the captured frames.
The extracted data was then analyzed to obtain the statistics of
6.2. Processes performed on the server system the games, with particular emphasis on the ball’s location, which is a
key focal point in football games. This processing took place on the
Once the Client computer establishes a connection with the Server
Client computer, and the resulting end-effector camera position was
system, the Server continuously listens for incoming messages. Upon
transmitted to the Server system responsible for controlling the robot.
receiving a new message containing the updated camera position, a
Leveraging the control and trajectory planning algorithms presented
new thread is initiated using Multi-Threading principles. This thread
earlier, the Server system orchestrated smooth and precise movements
is responsible for analyzing the message and determining the required
of the end-effector, seamlessly transitioning it from its current position
time for the robot to move from its current position to the desired
position. In this research, considering the robot’s performance, it is to the desired location at a desired speed. Experimental results, exem-
decided that the robot moves with a velocity of 14 cm/s. plified by the snapshot depicted in Fig. 19, showcased the feasibility
After obtaining the current and desired positions of the robot, the and effectiveness of the proposed methodology in utilizing suspended
thread calculates the time required based on the distance between these CDPRs to autonomously capture videos of football games. Furthermore,
positions. Using the septic trajectory planning algorithm described in this research successfully demonstrated the algorithm’s capability in
Section 5, a trajectory is planned to ensure smooth movements with obtaining the statistics of the games. An illustrative example of the
zero jerk, acceleration, and velocity at the start and end of the motion. game statistics can be viewed via THIS LINK, and Table 4 provides a
Additionally, the optimized digital controller presented in Section 5 summary of the captured game’s statistics. Overall, the experimental
is utilized to facilitate the robot’s movement from its current position results obtained from the algorithms presented in this research affirm
to the desired position. This algorithm has been implemented and the CDPRs’ potential in autonomously capturing football game videos
tested on the suspended CDPR at the Human and Robot Interaction and gathering valuable game statistics.
Laboratory, University of Tehran. A video showcasing the algorithm’s Throughout this research, deep learning and image processing tech-
performance can be viewed via THIS LINK. The video provides a visual niques were employed to process the data at a speed of 5 to 10 frames
demonstration of the robot’s smooth and precise movements achieved per second (FPS), considering the utilized hardware. However, it is
through the algorithm’s coordination. worth noting that this processing speed can be further improved to

10
M. Ghanatian et al. Mechatronics 95 (2023) 103058

achieve real-time performance (30 FPS) by upgrading the GPU and CPU Data availability
components of the Client system. On the other hand, in this study, the
players were represented as colored balls, and the possession of the ball No data was used for the research described in the article.
was not considered when calculating the statistics of the games. Future
research endeavors could expand upon this work by incorporating an Appendix A. Supplementary data
experimental setup that takes ball possession into account. By doing so,
a more comprehensive analysis of the game dynamics can be achieved. Supplementary material related to this article can be found online
The practical experiments conducted on the miniature football field at https://doi.org/10.1016/j.mechatronics.2023.103058.
at the Human and Robot Interaction Laboratory, University of Tehran,
validated the feasibility of autonomously capturing football games
References
using suspended CDPRs and obtaining the statistics of the games. A
notable advantage of the proposed method is the separation of the com- [1] Jiri Dvorak F-MFMA, Astrid Junge Dr, Graf-Baumann Toni, Peterson Lars,
puter responsible for image processing algorithms (Client computer) Center R. Football is the most popular sport worldwide. Am J Sports Med
from the computer in charge of controlling the robot. This allows for 2004;32(1). http://dx.doi.org/10.1177/0363546503262283.
the execution of machine vision and deep learning algorithms on a [2] Cone LL. Skycam: An aerial robotic camera system. Byte 1985;10(10):122–32.
[3] Hwang SW, Bak JH, Yoon J, Park JH. Oscillation reduction and frequency
separate system, which can be connected to the existing stadium sys-
analysis of under-constrained cable-driven parallel robot with three cables.
tems that control suspended CDPRs (referred to as the Server system in Robotica 2020;38(3):375–95. http://dx.doi.org/10.1017/S0263574719000687.
this research). Consequently, this algorithm’s development serves as a [4] Hwang SW, Bak JH, Yoon J, Park JH, Park JO. Trajectory generation to suppress
foundation for more sophisticated algorithms in the future, facilitating oscillations in under-constrained cable-driven parallel robots. J Mech Sci Technol
autonomous video-capturing of football games and statistical analysis 2016;30(12):5689–97. http://dx.doi.org/10.1007/s12206-016-1139-9.
in larger-scale stadiums. Overall, this research paves the way for further [5] Qi R, Rushton M, Khajepour A, Melek WW. Decoupled modeling and model
predictive control of a hybrid cable-driven robot (HCDR). Robot Auton Syst
advancements in autonomous video-capturing and game analysis, open- 2019;118:1–12. http://dx.doi.org/10.1016/j.robot.2019.04.013.
ing up opportunities for more comprehensive and advanced algorithms [6] Merlet J-P, Alexandre-dit Sandretto J. The forward kinematics of cable-driven
in the field of football game automation. parallel robots with sagging cables. In: Pott A, Bruckmann T, editors. Cable-
driven parallel robots. Cham: Springer International Publishing; 2015, p.
8. Conclusions 3–15.
[7] Pott A. An algorithm for real-time forward kinematics of cable-driven parallel
robots. In: Lenarcic J, Stanisic MM, editors. Advances in robot kinematics: Motion
This research focused on introducing autonomy to suspended CDPRs in man and machine. Dordrecht: Springer Netherlands; 2010, p. 529–38.
for the purpose of video-capturing football games, as well as obtaining [8] Ferravante V, Riva E, Taghavi M, Braghin F, Bock T. Dynamic analysis
the statistics of the games. Both algorithms presented in this study of high precision construction cable-driven parallel robots. Mech Mach The-
operated autonomously, eliminating the need for human intervention ory 2019;135:54–64. http://dx.doi.org/10.1016/j.mechmachtheory.2019.01.023,
URL https://www.sciencedirect.com/science/article/pii/S0094114X18307134.
or input. Computation was distributed across a computer network con-
[9] Miermeister P, Pott A. Modelling and real-time dynamic simulation of the cable-
sisting of a Server and a Client, connected via Wi-Fi. While the Client driven parallel robot ipanema. In: Pisla D, Ceccarelli M, Husty M, Corves B,
computer performed the machine vision and deep learning algorithms, editors. New trends in mechanism science. Dordrecht: Springer Netherlands;
the Server computer controlled the robot to capture video footage of 2010, p. 353–60.
the football games on the field. The experimental results showcased [10] Picard E. Modeling and robust control of cable-driven parallel robots for
industrial applications (Ph.D. thesis), École centrale de Nantes; 2020.
the feasibility of incorporating autonomy into CDPRs for autonomous
[11] Majumdar B, Naha S. Live sport during the COVID-19 crisis : fans as cre-
video-capturing of football games, along with the effectiveness of the ative broadcasters. Sport Soc 2020;23(7):1091–9. http://dx.doi.org/10.1080/
proposed algorithm in extracting the statistics of the games. However, 17430437.2020.1776972.
it is important to acknowledge that the experiments conducted in this [12] van Henten EJ, Hemming J, van Tuijl BAJ, Kornet JG, Meuleman J, Bontsema J,
research were carried out at a slower speed compared to real-time van Os EA. An autonomous robot for harvesting cucumbers in greenhouses.
football games. The impact of higher operational speeds on detection Autonomous Robots 2002;13:241–58.
[13] Beul M, Droeschel D, Nieuwenhuisen M, Quenzel J, Houben S, Behnke S. Fast
capabilities and the dynamic behavior of the CDPR system should
autonomous flight in warehouses for inventory applications. IEEE Robot Autom
be taken into account for practical applications. Future extensions Lett 2018;PP(c):1. http://dx.doi.org/10.1109/LRA.2018.2849833.
of this research should address these challenges and ensure robust [14] Michele Gabrio Antonelli EP, Zobel Pierluigi Beomonte, Marcellis Andrea De.
performance in real-world, high-speed scenarios. Despite these consid- Autonomous robot for cleaning photovoltaic panels in desert zones. Mechatron-
erations, the presented algorithms offer a cost-effective solution that ics 2020;68(February):102372. http://dx.doi.org/10.1016/j.mechatronics.2020.
102372.
does not rely on expensive hardware. The vision for this research is
[15] Buchegger A, Lassnig K, Loigge S, Clemens M, Steinbauer G. An autonomous
that with further development and refinement, the resulting algorithms vehicle for parcel delivery in urban areas. In: 2018 21st International conference
could potentially be implemented in stadiums worldwide. This would on intelligent transportation systems (ITSC). Maui, Hawaii, USA; 2018, p.
greatly enhance the autonomous video-capturing capabilities within the 2961–7.
context of football games, providing valuable insights and statistics to [16] Raj PA, Srivani M. Internet of robotic things based autonomous fire fighting mo-
bile robot. In: 2018 IEEE international conference on computational intelligence
spectators, coaches, and analysts alike.
and computing research (ICCIC). Madurai, India: IEEE; 2018, p. 1–4.
[17] Albus J, Bostelman R, Dagalakis N. The NIST ROBOCRANE. J Robot Syst
CRediT authorship contribution statement 1993;10(5):709–24.
[18] Tang X, Yao R. Dimensional design on the six-cable driven parallel manipulator
Mohammad Ghanatian: Conceptualization, Investigation, Method- of FAST. J Mech Des 2011;133:1–12. http://dx.doi.org/10.1115/1.4004988.
ology, Software, Validation, Writing – original draft. Mohammad Reza [19] Chen Q, Zi B, Sun Z, Li Y, Xu Q, Member S. Design and development
of a new cable-driven parallel robot for waist rehabilitation. IEEE/ASME
Hairi Yazdi: Supervision, Validation, Writing – review & editing.
Trans Mechatronics 2019;24(4):1497–507. http://dx.doi.org/10.1109/TMECH.
Mehdi Tale Masouleh: Supervision, Validation, Writing – review & 2019.2917294.
editing. [20] Cabay E, Culla D, Rodriguez M, Barrado M. Large-scale 3D printing with
cable-driven parallel robots. Construct Robot 2017;1:69–76. http://dx.doi.org/
Declaration of competing interest 10.1007/s41693-017-0008-0.
[21] Zare S, Ghanatian M, Yazdi MRH, Masouleh MT. Reconstructing 3-D Graphi-
cal Model Using an Under-Constrained Cable-Driven Parallel Robot. In: 2020
The authors declare that they have no known competing finan- IEEE 6th iranian conference on signal processing and intelligent systems (IC-
cial interests or personal relationships that could have appeared to SPIS). Mashhad, Iran: IEEE; 2020, http://dx.doi.org/10.1109/ICSPIS51611.2020.
influence the work reported in this paper. 9349584.

11
M. Ghanatian et al. Mechatronics 95 (2023) 103058

[22] Deschênes JD, Lambert P, Perreault S, Martel-Brisson N, Zoso N, Zaccarin A, [44] Ghanatian M, Hairi Yazdi MR, Tale Masouleh M. Dynamic modeling, parameter
Hébert P, Bouchard S, Gosselin CM. A cable-driven parallel mechanism for cap- identification and nonlinear model predictive control of a cable-suspended
turing object appearance from multiple viewpoints. In: 3DIM 2007 - Proceedings parallel robot with four cables. In: 2022 8th International conference on control,
6th international conference on 3-d digital imaging and modeling. 2007, p. instrumentation and automation, ICCIA 2022. 2022, http://dx.doi.org/10.1109/
367–74. http://dx.doi.org/10.1109/3DIM.2007.4. ICCIA54998.2022.9737169.
[23] Li C, Rahn CD. Design of continuous backbone, cable-driven robots. Trans ASME, [45] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: Unified,
J Mech Des 2002;124(2):265–71. http://dx.doi.org/10.1115/1.1447546. real-time object detection. 2016, arXiv arXiv:arXiv:1506.02640v5.
[24] Wang L. A measure for evaluation of maximum acceleration of redundant and [46] YOLOv5. http://dx.doi.org/10.5281/zenodo.7347926. URL https://github.com/
nonredundant parallel manipulators 2016;8:1–8. (2017), http://dx.doi.org/10. ultralytics/yolov5.
1115/1.4031500. [47] Chen YC. A tutorial on kernel density estimation and recent advances.
[25] Wu J, Wang J, Wang L, Li T. Dynamics and control of a planar 3-DOF parallel Biostat Epidemiol 2017;1(1):161–87. http://dx.doi.org/10.1080/24709360.2017.
manipulator with actuation redundancy. Mech Mach Theory 2009;44(4):835–49. 1396742, arXiv:1704.03924.
http://dx.doi.org/10.1016/j.mechmachtheory.2008.04.002. [48] Ghanatian M, Hairi Yazdi MR, Tale Masouleh M. Experimental study on reducing
[26] Merlet J-P. Parallel robots. 2006. the oscillations of a cable-suspended parallel robot for video capturing purposes
[27] Wu J, Qiu J, Ye H. Torque optimization method of a 3-DOF using simulated annealing and path planning. In: 7th International conference
Redundant Parallel Manipulator Based on Actuator Torque Range. J on signal processing and intelligent systems (ICSPIS). Tehran, Iran: IEEE; 2021.
Mech Robot 2022;15(2):021005. http://dx.doi.org/10.1115/1.4054618, [49] Gene F. Franklin AE-N, David Powell J. Feedback control of dynamic systems.
arXiv:https://asmedigitalcollection.asme.org/mechanismsrobotics/article- 1938.
pdf/15/2/021005/6890325/jmr_15_2_021005.pdf. [50] Angeles J. Fundamentals of robotic mechanical systems. 4th ed.. Montreal, QC,
[28] Roberts RG, Graham T, Lippitt T. On the inverse kinematics, statics, and fault Canada: Springer; 2003, http://dx.doi.org/10.1108/ir.2003.04930fae.003, no. 6.
tolerance of cable-suspended robots. J Robot Syst 1998;15(10):581–97. http://dx. [51] Shahidul Islam M. Introducing drone technology to soccer coaching. Int J Sports
doi.org/10.1002/(SICI)1097-4563(199810)15:10<581::AID-ROB4>3.0.CO;2-P. Sci Phys Educ 2020;5(1):1–4. http://dx.doi.org/10.11648/j.ijsspe.20200501.11.
[29] Carricato M, Merlet J-P. Stability analysis of underconstrained cable-driven
parallel robots. IEEE Trans Robot 2013;29(1):288–96. http://dx.doi.org/10.1007/
978-3-7091-1379-0_3.
[30] Hwang SW, Bak JH, Yoon J, Park JH, Park JO. Trajectory generation to suppress Mohammad Ghanatian received his B.Sc. in Mechanical
oscillations in under-constrained cable-driven parallel robots. J Mech Sci Technol Engineering from Shiraz University, Iran in 2019 and then
2016;30(12):5689–97. http://dx.doi.org/10.1007/s12206-016-1139-9. received his M.Sc. in Mechanical Engineering from the
[31] Diao X, Ma O. Vibration analysis of cable-driven parallel manipulators. Multibody University of Tehran, Iran in 2022. He has been a Graduate
Syst Dyn 2009;21(4):347–60. http://dx.doi.org/10.1007/s11044-008-9144-0. Research Assistant at the Human and Robot Interaction Lab-
[32] Du J, Agrawal SK. Dynamic modeling of cable-driven parallel manipulators with oratory (TaarLab) since 2019 and has performed research
distributed mass flexible cables. J Vib Acoust, Trans ASME 2015;137(2):1–8. on Dynamic Modeling, System Identification, and Control of
http://dx.doi.org/10.1115/1.4029486. Suspended Cable-Driven Parallel Robots. He has performed
[33] Ottaviano E, Arena A, Gattulli V. Geometrically exact three-dimensional mod- experimental studies on controlling Suspended Cable-Driven
eling of cable-driven parallel manipulators for end-effector positioning. Mech Parallel Robots for autonomous video-capturing of football
Mach Theory 2021;155:104102. http://dx.doi.org/10.1016/j.mechmachtheory. games and obtaining the statistics of the games using
2020.104102. Machine Vision and Deep Learning techniques. His main
[34] Arena A, Ottaviano E, Gattulli V. Dynamics of cable-driven parallel manipulators research interests include Robotics, Mechatronics, Control,
with variable length vibrating cables. Int J Non-Linear Mech 2023;151(Febru- Machine Learning, and Machine Vision.
ary):104382. http://dx.doi.org/10.1016/j.ijnonlinmec.2023.104382.
[35] Xu W, Yan P, Wang F, Yuan H, Liang B. Vision-based simultaneous measurement
of manipulator configuration and target pose for an intelligent cable-driven robot. Mohammad Reza Hairi Yazdi received his B.Sc. and
Mech Syst Signal Process 2022;165(2021):108347. http://dx.doi.org/10.1016/j. M.Sc. degrees in Mechanical Engineering from Amir Kabir
ymssp.2021.108347. University of Technology, Tehran, Iran in 1985 and 1987
[36] Sports analytics market size, share & trends analysis report by component respectively. He received his Ph.D. degree from Imperial
(software, service), by analysis type (on-field, off-field), by sports (football, College London in 1992, and since then, he has been at
cricket, basketball, baseball), and segment forecasts, 2022-2030. URL https: the University of Tehran, Tehran, Iran where he is a Full
//www.gr{and}viewresearch.com/industry-analysis/sports-analytics-market. Professor at the School of Mechanical Engineering. His main
[37] Decroos T, Bransen L, Van Haaren J, Davis J. VAEP: An objective approach to research interests include design, simulation, manufacturing
valuing on-the-ball actions in soccer (extended abstract). In: Proceedings of the and control of dynamic systems.
twenty-ninth international joint conference on artificial intelligence (IJCAI-20)
Sister. 2020, p. 4696–700. http://dx.doi.org/10.24963/ijcai.2020/648.
[38] Öberg Filip. Football analysis using machine learning and computer vision (Ph.D.
thesis), Luleå University of Technolog; 2021.
[39] Maruenda FB. Can the human eye detect an offside position during a football
match? Bmj 2004;329(7480):1470–2. http://dx.doi.org/10.1136/bmj.329.7480. Mehdi Tale Masouleh received the B.Eng. M.Sc. and Ph.D.
1470. degrees in Mechanical engineering (Robotic) from the Laval
[40] Ancona N, Cicirelli G, Branca A, Distante A. Goal detection in football by using University, Québec, Canada, in 2006, 2007 and 2010, re-
support vector machines for classification. In: Proceedings of the international spectively. He is currently a faculty member of the School
joint conference on neural networks, vol. 1. 2001, p. 611–6. http://dx.doi.org/ of Electrical and Computer Engineering at the University of
10.1109/ijcnn.2001.939092. Tehran. He is also the director of the Human and Robot
[41] Komorowski Jacek, Kurzejamski Grzegorz, Sarwas Grzegorz. Deepball: deep Interaction Laboratory, known as TaarLab for its Persian
neural-network ball detector. VISIGRAPP 2019 - Proceedings of the 14th abbreviation. His research interests are kinematics, dynamic
International Joint Conference on Computer Vision, Imaging and Computer and design of serial and parallel robotic systems, Humanoid,
Graphics Theory and Applications 2019;5:297–304. http://dx.doi.org/10.5220/ mobile robots and optimization techniques (interval analysis
0007348902970304. and convex optimization...) for robotic applications. He is
[42] Komorowski J, Kurzejamski G, Sarwas G. FootAndBall: Integrated player and ball supervising several undergraduate and graduate students
detector 2020;5:47–56. http://dx.doi.org/10.5220/0008916000470056, arXiv: and published several papers in different fields of robotic
1912.05445, 2019 - arxiv.org, arXiv:1912.05445. mechanical systems. Some of the industrial projects under
[43] Mersi R, Vali S, Haghighi MS, Abbasnejad G, Tale Masouleh M. Design and the supervision of Mehdi Tale Masouleh are the develop-
control of a suspended cable-driven parallel robot with four cables. Proceedings ment of 3-DOF parallel mechanisms, a Haptic device for
of the 6th RSI International Conference on Robotics and Mechatronics, IcRoM dental education simulation as a national-level project, and
2018 2019;470–5. http://dx.doi.org/10.1109/ICRoM.2018.8657534. the development of a 4DOF Delta robot.

12

View publication stats