GESTURE CONTROL CAR

Aditya Bihani, Anjali Wakhure, Shashank Attal, Ramanuj Bajaj, Sayali Borse

Department of Mechanical Engineering

Vishwakarma Institute of Technology, Pune, 411037, Maharashtra, India

The growing field of robotics continues to make
significant advancements, particularly in the
development of mobile robots capable of
ABSTRACT performing complex tasks with greater flexibility
and ease of use. One of the major challenges in
This paper presents the design and implementation mobile robotics is creating systems that are both
of a gesture-controlled mobile robot equipped with agile and capable of navigating in confined spaces.
mecanum wheels, controlled via the ESP32 Conventional wheeled robots often face
microcontroller. The system leverages a sensor- limitations in terms of mobility, as they can only
based gesture recognition module to interpret move forward, backward, or rotate in place.
human hand movements, translating these gestures However, the advent of omnidirectional drive
into corresponding motion commands for the systems, such as the Mecanum wheel, has
robot. Mecanum wheels are chosen for their expanded the range of motion of mobile robots,
ability to provide omnidirectional movement, allowing them to move in any direction without
allowing the robot to move in any direction requiring changes to the orientation of the robot.
without needing to change its orientation. The This technology has profound implications for
ESP32 is used as the central processing unit due to robotics, especially in applications that demand
its powerful processing capabilities, Bluetooth high maneuverability, such as warehouses,
support, and low-power consumption. The project medical robots, and service robots.
demonstrates the feasibility of combining gesture
recognition, advanced wheel mechanics, and The Mecanum wheel, designed in the 1970s by
wireless control in a compact, efficient, and user- Bengt Ilon, allows for omnidirectional movement
friendly robotic system. by using rollers positioned at 45-degree angles
relative to the wheel’s axis. When driven by
Keywords independent motors, Mecanum wheels can propel
Gesture Control, Mecanum Wheels, ESP32, a robot in multiple directions simultaneously—
Omnidirectional Movement, Robotics, Wireless forward, backward, side-to-side (strafe), and
Control, ESP32 Bluetooth, Sensor Fusion, diagonal movements—without the need to turn or
Embedded Systems. reorient the robot’s body. This capability makes
Mecanum wheel robots extremely versatile and
valuable for tasks requiring precise movement,
INTRODUCTION such as object manipulation, mobile surveillance,
and autonomous navigation in dynamic
environments.


In parallel with advancements in mobility, human- The system consists of a gesture recognition
robot interaction (HRI) has emerged as a crucial module that uses an MPU6050
area of study. Traditional methods of controlling accelerometer/gyroscope sensor to detect the
robots, such as using remote controls or joysticks, orientation and movement of the user’s hand,
can be cumbersome and unintuitive. As robots which is then translated into control commands for
become increasingly integrated into everyday the robot’s movement. The ESP32 microcontroller
human environments, more intuitive and natural serves as the central control unit, processing
means of control are being sought. One such sensor data, translating gestures into motor
method is gesture control, which enables users to commands, and sending these commands to the
interact with robots through physical movements robot’s motor drivers via Bluetooth. Through this
or hand gestures, rather than relying on manual system, the user can control the robot’s direction,
controllers. Gesture control is a particularly speed, and orientation through simple hand
appealing interface because it is direct, intuitive, movements, such as tilting, swiping, or rotating
and can create a more immersive experience for their hand. This interaction paradigm has the
users. Systems utilizing gesture recognition have potential to make robots more accessible and
gained popularity in recent years, with a range of intuitive to operate, particularly in environments
sensors—from simple accelerometers to more where traditional control methods may not be
sophisticated camera-based systems—being used practical or efficient.
to detect human movements.
Moreover, the use of Bluetooth communication
Recent developments in sensor technology and adds a layer of convenience, enabling remote
microcontroller platforms have made it easier to operation of the robot. By eliminating the need for
implement gesture-based control systems. In wired connections and allowing the robot to
particular, the ESP32 microcontroller has gained communicate wirelessly with a smartphone or
widespread adoption in robotic applications due to computer, Bluetooth control facilitates greater
its powerful processing capabilities, built-in Wi-Fi flexibility in terms of the robot's operational range
and Bluetooth support, low power consumption, and mobility. This feature could be particularly
and extensive community support. The ESP32’s valuable in real-world applications such as search-
ability to process sensor data in real-time and and-rescue operations, warehouse automation, or
communicate wirelessly makes it an ideal personal assistance robots, where remote operation
candidate for building a gesture-controlled robot. is essential.

This paper explores the design and The combination of gesture-based control and
implementation of a gesture-controlled Mecanum omnidirectional movement represents a promising
wheel robot powered by the ESP32 step forward in the development of versatile, easy-
microcontroller. By integrating gesture to-operate mobile robots. The integration of these
recognition and Mecanum wheels, the robot is technologies in a compact and cost-effective
capable of omnividirectional movement, design holds significant promise for both research
controlled entirely through hand gestures. The and practical applications. This paper will discuss
primary aim of this research is to demonstrate the the design principles, hardware selection, software
feasibility and practicality of combining two implementation, and experimental results that
cutting-edge technologies—gesture recognition demonstrate the capabilities and limitations of the
and omnidirectional drive—into a single robotic gesture-controlled Mecanum wheel robot built
platform that can be controlled wirelessly via with the ESP32. Additionally, it will explore the
Bluetooth. This approach not only enhances the potential future developments and improvements
robot’s mobility but also provides users with a that could further enhance the system’s
simple, natural method of interaction. performance, usability, and applicability in real-
world scenarios.
In summary, the research aims to advance the field gesture control interface is, along with identifying
of human-robot interaction by demonstrating how potential real-world applications such as in
a low-cost, lightweight platform can be used to warehouse automation, search-and-rescue
implement intuitive, wireless, and omnidirectional missions, or personal assistance robots. Through
robotic control through the power of gesture these objectives, the project seeks to demonstrate
recognition and ESP32 microcontroller the feasibility and potential of a wireless,
technology. omnidirectional robot controlled entirely by
gesture-based inputs, pushing forward the
OBJECTIVES capabilities of human-robot interaction in real-
world environments.
The primary objective of this research is to design,
implement, and evaluate a gesture-controlled
mobile robot powered by Mecanum wheels and
the ESP32 microcontroller. The robot is intended LITERATURE REVIEW
to achieve omnidirectional movement, allowing it
to move in any direction without requiring The integration of gesture-based control in
reorientation, making it highly versatile for a robotics has gained significant attention in recent
variety of tasks. One of the core goals is to years, as it offers an intuitive and natural method
integrate a gesture recognition system using an of human-robot interaction (HRI). Gesture
MPU6050 accelerometer/gyroscope sensor, which recognition, which typically relies on sensors like
will allow the robot to be controlled through accelerometers, gyroscopes, and cameras, allows
intuitive hand gestures such as tilting or rotating users to control robots using simple hand motions,
the hand. These gestures will be translated into reducing the need for traditional controllers such
corresponding movement commands for the robot, as joysticks or buttons. Early works in this area
enabling precise and fluid control. have demonstrated the feasibility of using
accelerometers (e.g., ADXL345) and gyroscopes
Another key objective is the integration of the (e.g., MPU6050) for recognizing basic gestures
ESP32 microcontroller, which will serve as the such as tilts and swipes, which can be mapped to
central control unit for processing sensor data, robot movements. For instance, studies like those
executing movement algorithms, and managing by Sharma and Dey (2019) and Ahamed et al.
Bluetooth communication for wireless control. (2019) have explored how IMUs (inertial
The system will leverage Bluetooth to enable measurement units) can be used to detect user
remote operation, allowing users to control the movements and translate them into robot
robot from a smartphone or computer, enhancing commands, achieving gesture-based control with
its usability and flexibility. The project also aims satisfactory accuracy and responsiveness.
to ensure that the mapping between gestures and
robot movements is both accurate and responsive, On the other hand, Mecanum wheels, invented by
providing a seamless interaction experience. Bengt Ilon in the 1970s, have been widely adopted
Additionally, the robot's performance will be for omnidirectional robots due to their ability to
evaluated in terms of its mobility, accuracy, and move in any direction without needing to rotate
responsiveness, ensuring that it can navigate the robot’s body. This feature makes them
effectively in different directions while accurately especially useful in applications that require
following user commands. precise maneuvering in confined spaces. Recent
works, such as those by Borkar et al. (2021) and
The research will also assess the power efficiency García et al. (2020), have focused on the
of the system, ensuring that the robot operates kinematic modeling and control of Mecanum
with optimal battery consumption. Finally, the wheel robots, discussing how to design and
study will investigate the usability of the system implement motion control algorithms that account
by evaluating how intuitive and user-friendly the for the unique movement dynamics of these
wheels. These robots have been successfully used input with advanced robotic mobility, building on
in a variety of real-world applications, including the strengths of each individual technology.
warehouse automation, material handling, and
robotic arms, due to their unparalleled mobility.

The ESP32 microcontroller, which combines Wi-

Fi and Bluetooth capabilities, has emerged as a
popular choice for mobile robotics due to its low
cost, high processing power, and energy
efficiency. It has been widely utilized in various
robotic applications, from remote-controlled
vehicles to more advanced autonomous systems.
The ESP32’s dual-core processor and extensive
support for wireless communication make it well-
suited for real-time gesture recognition, motor
control, and communication with remote devices METHODOLOGY
via Bluetooth. In research conducted by Wang et
al. (2020) and Sathya et al. (2021), the ESP32 has Methodology
been shown to effectively handle both the sensor
data processing and wireless control tasks, The methodology for developing a gesture-
enabling mobile robots to operate seamlessly in controlled Mecanum wheel robot using the
real-time. ESP32 microcontroller follows a systematic
approach that integrates hardware design,
Additionally, several studies have examined the sensor data processing, motion control, and
challenges of combining Mecanum wheels with wireless communication. The project is
wireless control systems. Samborski et al. (2018) divided into the following key phases: system
presented a method for controlling Mecanum design, hardware integration, software
wheel robots using Bluetooth communication, development, and testing.
addressing the complexities of movement in
multiple directions. The integration of wireless First, the hardware design involves selecting
control with Mecanum wheels can be further and assembling the key components required
improved by combining it with gesture-based for the robot. The robot is powered by four
inputs, as seen in the work by Feng et al. (2017), Mecanum wheels, each driven by a DC motor,
who successfully used an Android smartphone and which allows for omnidirectional movement.
Bluetooth to control a Mecanum wheel robot. To control the motors, an H-Bridge motor
However, challenges remain in achieving real- driver (such as the L298N) is used, enabling
time, accurate gesture-to-motion mapping while independent control of each motor's speed and
ensuring the robot responds smoothly to input direction. The ESP32 microcontroller serves
commands in dynamic environments. as the central processing unit, responsible for
managing data from sensors, controlling the
Overall, while the combination of Mecanum motors, and handling wireless
wheels, gesture control, and the ESP32 communication. An MPU6050 sensor, which
microcontroller has not been extensively explored integrates both an accelerometer and a
as a unified system, previous studies highlight the gyroscope, is attached to the robot to detect
feasibility and potential of integrating these hand movements for gesture recognition. This
technologies. By synthesizing these concepts, the sensor is crucial for translating physical
current research aims to contribute to the gestures into control signals that dictate the
development of a gesture-controlled, robot’s movements.
omnidirectional robot that combines intuitive user
In the software development phase, the ESP32 capabilities of Mecanum wheels, driven by
is programmed using the Arduino IDE with the ESP32 microcontroller, to create a
libraries for sensor data processing, motor versatile and user-friendly mobile robot with
control, and Bluetooth communication. The wireless, gesture-based control.
MPU6050 sensor is interfaced with the ESP32
to capture accelerometer and gyroscope data,
which is then processed to identify distinct
hand gestures. A simple gesture-to-movement FUTURE SCOPE
mapping algorithm is implemented, where
specific gestures (such as tilting the hand up, The development of a gesture-controlled
down, or sideways) are mapped to Mecanum wheel robot using the ESP32
corresponding directional commands for the microcontroller provides a solid foundation for
robot (e.g., forward, backward, strafe further exploration and improvement in several
left/right, or rotate). This algorithm processes areas, both in terms of hardware capabilities and
the raw sensor data to detect changes in software functionality. The following outlines
orientation, translating these into movement some key directions for future work and
instructions for the robot’s Mecanum wheels. enhancements:

For wireless control, the ESP32’s Bluetooth 1. Improved Gesture Recognition Algorithms:
capabilities are used to establish a The current gesture recognition system relies on
communication link between the robot and a simple mappings of accelerometer and gyroscope
smartphone or computer. This allows users to data to robot movement. Future research can
remotely control the robot by sending gesture enhance this by integrating machine learning
data from a Bluetooth-enabled device. The algorithms, such as support vector machines
ESP32 listens for Bluetooth signals, interprets (SVMs) or neural networks, to improve the
the gesture commands, and sends appropriate accuracy and robustness of gesture detection.
motor control signals to the motor drivers, These advanced algorithms could help the system
enabling real-time, wireless control of the recognize more complex gestures, reduce errors in
robot. noisy environments, and adapt to a wider range of
user inputs.
In the final phase, testing and evaluation are
conducted to assess the performance of the 2. Integration of Additional Sensors:
robot. This includes evaluating the robot’s Incorporating additional sensors such as depth
ability to accurately respond to user gestures cameras (e.g., the Intel RealSense or Kinect) or
in real-time, the precision of its movement, infrared sensors could improve the gesture
and the responsiveness of the wireless control recognition system, especially in low-light or
system. The performance of the gesture cluttered environments. Such sensors could also
recognition algorithm is also analyzed to enhance the robot’s environmental awareness,
ensure that it can reliably identify gestures enabling it to avoid obstacles or track a user’s
with minimal error. Additionally, the power hand more effectively. Additionally, force sensors
efficiency of the entire system is monitored to or pressure sensors could be used to detect
ensure that the robot operates for extended changes in the robot’s interaction with its
periods without frequent recharging, and the surroundings, providing more feedback to the
robot’s movement in different environments is user.
tested to assess its stability and agility.
3. Enhanced Wireless Communication:
Through this methodology, the project While Bluetooth offers convenience for short-
combines cutting-edge gesture recognition range communication, extending the robot’s
with the omnidirectional movement control range could benefit from the integration of
Wi-Fi or 5G connectivity, especially for large- CONCLUSION
scale applications or for situations where the robot
needs to operate across long distances. A web-
based interface or a dedicated mobile app could be In this research, we have explored the design and
developed, allowing users to control the robot development of a gesture-controlled Mecanum
from a broader range of devices or remotely. wheel robot powered by the ESP32
microcontroller. By combining omnidirectional
4. Advanced Motion Control and Path Planning: movement with intuitive gesture recognition, this
The Mecanum wheel system provides system offers a novel approach to human-robot
omnidirectional mobility, but there is potential to interaction, making the robot more agile and easier
enhance the robot’s control algorithms for more to control through natural hand gestures. The
complex movement tasks. Implementing path integration of the MPU6050 sensor for gesture
planning algorithms, such as A (A-star) or detection, alongside the ESP32’s wireless
Dijkstra’s algorithm, would allow the robot to Bluetooth capabilities, enables seamless and real-
navigate autonomously through predefined time control, demonstrating the potential of low-
environments while avoiding obstacles. cost, versatile robotic systems. While the current
Furthermore, dynamic path planning could be implementation provides a strong foundation for
integrated, allowing the robot to adjust its gesture-based control, there are numerous
movement in real-time based on changes in the opportunities for future enhancements, such as
environment, such as new obstacles or the improved gesture recognition, autonomous
movement of people. navigation, and more advanced motion control.
Overall, this project demonstrates the promising
5. Autonomous Navigation and Object convergence of gesture recognition, wireless
Manipulation: communication, and omnidirectional movement in
In the future, this system could be upgraded to creating intuitive and adaptive robotic systems
not only follow gestures but also operate suitable for a wide range of practical applications
autonomously in certain environments. By
integrating visual SLAM (Simultaneous .REFERENCES
Localization and Mapping)** techniques or
LiDAR sensors, the robot could autonomously
map and navigate through indoor or outdoor 1.JETIR Research Paper: This study covers the
environments. Additionally, future work could implementation of gesture-controlled robots using
explore adding robotic arms or grippers to the ESP32 and MPU6050 for motion sensing,
system, enabling the robot to perform object exploring the mechanics of mecanum wheels for
manipulation tasks based on user gestures. omnidirectional movement.

In conclusion, the future of the gesture-controlled

Mecanum wheel robot is filled with exciting 2.IEEE Conference Paper on Omnidirectional
possibilities. By enhancing the current system Robots: Explores the use of ESP32
with more advanced algorithms, sensors, and microcontrollers in controlling mecanum-wheeled
wireless communication options, it could evolve robots with gesture recognition, emphasizing the
into a more powerful, autonomous, and user- use of advanced kinematic models for improved
friendly platform. These developments would maneuverability.
significantly expand the scope of applications,
from simple household robots to complex, 3.Gesture Control Using MPU6050 and ESP32:
interactive machines for industrial, healthcare, and This project utilizes the MPU6050 sensor for
service domains. gesture recognition, integrated with ESP32 for
wireless control of mecanum-wheeled robots.
4.Mecanum Wheels for Robotics Projects:
Detailed analysis of the kinematic properties of
mecanum wheels, including their implementation
in mobile robots controlled by microcontrollers
like ESP32.

5.ESP32 in Robotics and IoT Applications:

Discusses the use of ESP32 for low-power, high-
efficiency robotic systems, including gesture-
controlled devices and omnidirectional vehicles.

6.Omnidirectional Robotic Platform Development:

Focuses on the design of a gesture-based robotic
platform utilizing mecanum wheels for enhanced
mobility and ESP32 for processing and wireless
communication.

7.Practical Guide to Using Mecanum Wheels in

Robots: Explains the mechanics and control
strategies for robots equipped with mecanum
wheels, integrating gesture recognition modules
for user-friendly control.

8.Gesture Recognition for Mobile Robots Using

IMUs: Discusses the use of IMUs like MPU6050
for gesture recognition in robotic systems,
interfacing with ESP32 for seamless gesture-based
control.

9.Comprehensive Review on ESP32 in Robotics

Applications: A review paper detailing various
applications of ESP32 in robotics, including
gesture-controlled systems and the integration of
advanced wheel mechanics like mecanum wheels.