• AR: Selection of AR Platform,

• Integrating Hardware and Software

• Optical & Inertial Calibration
• Tracking
• AR Computer Vision
 Selection of AR Platform
• According to the latest report from Statista, the global augmented reality (AR) and virtual reality (VR)
markets are exploding and are forecast to reach a total of 72.8 billion U.S. dollars by 2024.
• Wikitude was the first mobile app to use augmented reality (AR) and was launched in 2008.
• There are three main types of augmented reality platforms available to you:
▪ social media apps
▪ browser-based AR (WebAR)
▪ native mobile apps
• Each of these comes with its own set of benefits and will be best suited to different types of AR
campaigns.
• Selecting an AR platform depends on several factors, including the specific use case, the target
audience, the devices you intend to support, and your development expertise.
Selection of AR Platform
 Selection of AR Platform

• ARCore is Google’s augmented reality SDK offering cross-platform

APIs to build new immersive experiences on Android, iOS, Unity,
and Web.
• Transform the way people play, shop, learn, create, and experience
the world together through contextual understanding about people,
places, and things.
• The fundamentals of the platform include
•Motion tracking, which shows positions relative to the world
•Anchors, which ensures tracking of an object’s position over
time
•Environmental understanding, which detects the size and
location of all types of surfaces
•Depth understanding, which measures the distance between
surfaces from a given point •Use Cases: Retail and marketing
•Light estimation, which provides information about the applications, location-based
average intensity and color correction of the environment experiences, AR social experiences.
Geospatial AR Content in Google Maps
https://storage.googleapis.com/gweb-developer-goog-blog-assets/original_videos/3-reencode.mp4
 Selection of AR Platform

• Apple's AR platform is optimized for iOS devices and is well-suited for

creating interactive, immersive experiences.
• ARKit combines device motion tracking, world tracking, scene
understanding, and display conveniences to simplify building an AR
experience.
• ARKit 6 introduces 4K video, so you can capture stunning high-resolution
videos of AR experiences — perfect for professional video editing, film
production, social media apps, and more.

•Use Cases: Educational apps, product visualization,

gaming (think Pokemon Go).
 Selection of AR Platform

• Unity’s AR Foundation is a cross-platform framework that allows the user to

write AR experiences and build for either Android or iOS devices.
• Unity provides powerful tools to make rich and deeply engaging AR
experiences that intelligently interact with the real world.
• It has custom resources and an industry-leading software development
platform that pairs tools purpose-built for AR creators.
• It is used across a range of industries, including automotive, manufacturing,
government, architecture, energy and retail.
• Unity is one of the most popular game development tools out there and has been
used to create popular games such as Pokémon Go, Hearthstone and
Rimworld Unity is one of the top choices for game developers, designers and artists.
Use Cases: AR games, interactive experiences, product
prototypes
 Selection of AR Platform

• The MetaSpark Studio platform offers the user flexible tooling to easily build
and share AR experiences at scale. Users can start developing from templates
or build their own custom AR experiences with code.
• It is easy to build interactive AR experiences with the platform, allowing for
target tracking, depth mapping, occlusion and audio and haptic integrations.
 Selection of AR Platform

• A popular AR SDK, Vuforia deploys computer vision technology to recognize

and track 3D objects in real time. It allows developers the ability to create
both marker-based and markerless AR experiences and one of its highlight
features is its support of persistent objects. Vuforia provides an API for
multiple programming languages and supports native development for iOS,
Android, UPW, and Unity.

•Use Cases: Industrial applications, maintenance and

repair guides, marketing campaigns with image
recognition.
 Selection of AR Platform

• Wikitude is a mobile AR technology provider based in Salzburg, Austria. Founded in

2008, the platform initially focused on providing location-based augmented reality
experiences through the Wikitude World Browser App. Its AR offers iOS, Android
and Smart Glasses apps with image and object tracking, instant tracking (SLAM) and
Geo AR, along with other capabilities.
• Wikitude is a flexible AR development tool with which developers can customize
solutions and retrieve 3D objects from third-party SDKs through its own platform.
Some of its features include navigation support, distance-based scaling, and
localization. Wikitude can be used for Android, iOS, and Windows for tablets, and
certain smart glasses.
se Cases: Geolocation based AR experience; marketing
campaign; educational app.
 Selection of AR Platform

• 8th Wall provides powerful tools to create WebAR experiences which are optimized
for the browser. 8th Wall supports great AR features such as face effects, world
tracking, and image targets, and allows for source code control through a cloud
editor.
 Selection of AR Platform
Platform
Tool Feature Set Ease of Use Best For
Compatibility
High-quality visuals, multi-user
ARKit iOS Apple developers iOS apps
experiences
Environmental understanding, multi-
ARCore Android, iOS Android Studio/Unity Cross-platform apps
user
echo3D Multiple Cloud-based, multi-user, animations Drag and drop Web and mobile AR
Unity Multiple 3D applications, multi-platform Requires coding Complex 3D AR apps
Object/image recognition, cloud
Vuforia Multiple Compatible tools Interactive experiences
storage
Wikitude Multiple Image/object/place recognition Simple editor Professional apps
Spark AR iOS, Android Social media effects User-friendly Social media effects
Web, iOS,
8th Wall Web-based AR, multi-user Unity & JavaScript Web-based AR
Android
Kudan iOS, Android SLAM, multi-user Drag and drop Mobile games, demos

ARToolKit Multiple Basic tracking, custom markers Unity compatible Basic AR projects
 Selection of AR Platform

Microsoft HoloLens is an augmented reality (AR)/mixed reality (MR) headset

developed and manufactured by Microsoft.

https://www.microsoft.com/en-in/hololens/hardware#document-experiences
 Selection of AR Platform

Magic Leap, Inc. is an American technology company that released

a head-mounted augmented reality display, called Magic Leap One,
which superimposes 3D computer-generated imagery over real world
objects.
Target Devices
▪ Mobile Devices (smartphones, tablets):
• ARKit: Best for targeting iOS devices (iPhones, iPads).
• ARCore: Best for Android devices, but also has cross-platform capabilities.
• 8th Wall: Ideal for web-based AR experiences that need to work across
different devices without requiring app installation.
▪ Wearables (AR glasses, headsets):
• Microsoft HoloLens: Ideal for enterprise and industrial applications with
hands-free interaction.
• Magic Leap: Good for immersive experiences in both enterprise and
entertainment.
• Google Glass Enterprise Edition: Tailored for industrial and professional use
cases where hands-free, lightweight AR is required.
▪ Web-based AR:
• 8th Wall: Allows you to build AR experiences that can be accessed directly
through a browser, eliminating the need for an app download.
• Zappar: Provides a balance between ease of use and functionality, also
supporting web-based AR.
Development Environment and Expertise
▪ Unity with AR Foundation:
• If you have experience with Unity, AR Foundation provides a powerful
framework that supports both ARKit and ARCore, making it a great
option for cross-platform development.
▪ Unreal Engine:
• Suitable for developers looking to create high-fidelity AR experiences,
especially in gaming or high-end visualizations.
▪ Web-based AR Tools:
• A-Frame or Three.js: For web developers familiar with HTML and
JavaScript, these frameworks offer a way to create AR experiences that run
in browsers.
▪ Easy-to-use Platforms:
• Zappar or Lens Studio: These platforms are more user-friendly and
require less technical expertise, making them suitable for creators without
a strong programming background.
General Information
▪ ARKit/ARCore: Best for mobile-based AR experiences on iOS and
Android.
▪ Unity with AR Foundation: Great for cross-platform development
with a need for high flexibility.
▪ Microsoft HoloLens/MRTK: Ideal for enterprise and industrial use
cases with spatial computing needs.
▪ 8th Wall/Zappar: Excellent for web-based AR, with easy access and
no need for app downloads.
▪ Vuforia: Versatile for various applications, especially in industrial
settings.
▪ Unreal Engine: Suitable for high-fidelity, immersive AR experiences.
Integrating Hardware and Software

▪ Integrating hardware and software in Augmented Reality (AR) is a

complex but essential process to ensure that AR experiences are
immersive, responsive, and effective.
▪ The integration involves aligning the capabilities of the hardware
(such as sensors, cameras, and processors) with the software (such as
AR frameworks, applications, and algorithms) to deliver seamless
AR interactions.
Understanding the Hardware Components
▪ Cameras:
• Cameras are crucial for capturing the real world to overlay digital content.
High-resolution cameras improve the quality of the AR experience.
• Depth-sensing cameras (like LiDAR on iPhones) allow for more accurate
placement of virtual objects in 3D space, enhancing realism.
▪ Sensors:
• IMU (Inertial Measurement Unit): Combines accelerometers, gyroscopes,
and sometimes magnetometers to track the orientation and movement of
the device.
• GPS: Essential for location-based AR, enabling applications to overlay
information based on the user’s geographical position.
• Depth Sensors: Used to understand the distance between objects, enabling
more sophisticated interaction with the virtual and real world.
Cont.,
▪ Processors:
• The choice of processor impacts the performance of AR applications,
especially in terms of real-time rendering and object recognition.
• Modern mobile devices often include specialized chips (like Apple’s A-
series chips with Neural Engines) designed to handle AR tasks efficiently.
▪ Display Technology:
• Smartphones/Tablets: Utilize the existing screen for AR overlay, but have
limitations in field of view and user immersion.
• AR Glasses/Headsets (e.g., HoloLens, Magic Leap): Provide more
immersive experiences by projecting digital content directly into the user’s
view, allowing for hands-free interaction.
Selecting the Appropriate AR Software
▪ AR SDKs (Software Development Kits):
• ARKit (iOS) and ARCore (Android): These SDKs leverage the
device’s hardware to create AR experiences. They handle tasks such
as motion tracking, environmental understanding, and light
estimation.
• Vuforia: Offers robust image and object recognition, suitable for
industrial and enterprise applications.
• MRTK (Mixed Reality Toolkit) for HoloLens: Provides tools for
developing applications on Microsoft’s HoloLens, integrating with
its unique hardware features like hand tracking and spatial
mapping.
Cont.,
▪ Game Engines:
• Unity with AR Foundation: Facilitates cross-platform development by
providing a unified API that works with ARKit and ARCore.
• Unreal Engine: Known for high-quality graphics, it’s suitable for complex
and visually intensive AR applications.
▪ Custom Software Development:
• Sometimes, off-the-shelf AR SDKs may not meet all your needs. In such
cases, custom software development, possibly using libraries like OpenCV
for computer vision, may be necessary.
Integration Process
▪ Calibration and Synchronization:
• Ensuring that the hardware (e.g., cameras and sensors) and software (e.g., AR
frameworks) are properly calibrated is crucial.
• This involves aligning the software’s understanding of the physical world with
the data captured by the hardware.
• Calibration often includes synchronizing the camera’s field of view with the
software’s rendering system to ensure that virtual objects appear correctly in
the user’s view.
▪ Real-Time Data Processing:
• AR experiences require real-time processing of data from cameras and sensors
to render virtual objects accurately.
• The software must efficiently process this data to maintain a high frame rate
and low latency.
• Machine learning algorithms, often run on specialized hardware (like GPUs or
AI accelerators), can be used for tasks such as object recognition or
environment understanding.
Cont.,
▪ Environmental Understanding:
• The software needs to interpret the physical environment to allow for
realistic interaction between virtual and real-world objects.
• This includes detecting surfaces (planes), understanding light conditions,
and recognizing objects.
• Advanced AR experiences, particularly in wearables, rely on spatial
mapping and environmental meshing to create a more immersive
experience.
▪ Rendering and Visualization:
• The rendering engine needs to ensure that virtual objects are displayed
with correct perspective, occlusion, and lighting relative to the real world.
• This might involve using shaders to simulate realistic lighting effects or
adjusting the transparency of virtual objects to blend them naturally with
the physical environment.
Cont.,
▪ User Interaction:
• Integrating user input, whether through touch (on mobile devices),
gestures (on AR glasses), or voice commands, is essential for interactive
AR applications.
• The software must process these inputs in real time, ensuring that the
system responds intuitively to user actions.
 Optical & Inertial Calibration

▪ Optical and inertial calibration are crucial steps in ensuring that an

Augmented Reality (AR) system functions accurately and reliably.
▪ These calibrations align the optical sensors (e.g., cameras) and
inertial sensors (e.g., accelerometers, gyroscopes) so that the AR
software can correctly interpret the real world and overlay virtual
objects seamlessly.
Understanding the Components
▪ Optical Sensors:
• Typically, these are cameras that capture the visual information from the
real world.
• They are responsible for features like object recognition, marker detection,
and environment mapping.
• In some advanced AR devices, optical sensors might include depth
sensors like LiDAR, which capture depth information to create 3D maps of
the environment.
Understanding the Components
▪ Inertial Sensors:
• These include accelerometers, gyroscopes, and sometimes magnetometers,
which track the device's orientation, movement, and sometimes even its
absolute heading (compass direction).
• These sensors provide data on the device's motion, which is crucial for
stabilizing the AR content relative to the user’s movements.
▪ Accelerometer
Accelerometers are responsible for measuring linear acceleration
https://www.youtube.com/watch?v=i2U49usFo10&t=107s
▪ Gyroscope
Gyroscopes measure velocity or rotational movement.
https://www.youtube.com/watch?v=cquvA_IpEsA&t=212s
▪ Magnetometers
• Magnetometers determine orientation to Earth’s magnetic field.
Why Calibration is Important?
▪ Alignment:
• Ensures that the data from the optical and inertial sensors are aligned
correctly.
• Without proper calibration, the virtual objects may appear to drift, lag, or
be misaligned with the real world.
▪ Accuracy:
• Calibration improves the accuracy of pose estimation, which is critical for
placing virtual objects accurately in 3D space.
▪ Stability:
• Proper calibration reduces jitter and improves the stability of AR content,
making the experience more comfortable and believable for users.
Optical Calibration
▪ Optical calibration focuses on ensuring that the camera(s) used in the
AR system accurately capture and interpret the visual data.
• Intrinsic Calibration:
• Purpose: To determine the internal parameters of the camera, such as
focal length, optical center, and lens distortion coefficients.
• Process:
• Capture multiple images of a known calibration pattern, such as a checkerboard,
from different angles and distances.
• Use calibration software (like OpenCV’s calibration tools) to estimate the camera’s
intrinsic parameters.
• These parameters are then used to correct lens distortions and to ensure that the
images captured by the camera accurately represent the physical world.
Cont.,
▪ Extrinsic Calibration:
• Purpose: To determine the position and orientation of the camera
relative to other sensors (like depth sensors or another camera in a
stereo setup).
• Process:
• In a multi-camera setup, capture images of the same scene simultaneously
from different cameras.
• Use the captured data to calculate the relative position and orientation
between the cameras.
• This step is crucial for applications that rely on stereo vision or require
accurate spatial mapping.
Cont.,
▪ Stereo Camera Calibration:
• Purpose: Used when an AR system employs two cameras to capture
stereoscopic images, allowing for depth perception.
• Process:
• Perform intrinsic calibration on both cameras individually.
• Use stereo calibration techniques to determine the relative orientation and
position between the two cameras.
• This calibration allows the system to calculate the depth information by
comparing the two images and finding the disparities.
Inertial Calibration
▪ Inertial calibration ensures that the accelerometer, gyroscope, and
magnetometer (if present) provide accurate and consistent data.
• Accelerometer Calibration:
• Purpose: To ensure that the accelerometer correctly measures acceleration
in all three axes.
• Process:
• Place the device on a flat surface and ensure that it is perfectly still.
• Record the accelerometer readings for all three axes and compare them to the
expected values (e.g., the gravity vector should read approximately 9.81 m/s²
along the axis pointing downward).
• Adjust the sensor’s scale and offset values to match the expected output
Cont.,
▪ Gyroscope Calibration:
• Purpose: To ensure that the gyroscope accurately measures
rotational rates without drift.
• Process:
• Keep the device stationary and record the gyroscope readings.
• Since the device is not rotating, the gyroscope should read near-zero
angular velocity. Any consistent deviation (bias) is corrected through
software by subtracting this bias from future readings.
Cont.,
▪ Magnetometer Calibration (if applicable):
• Purpose: To ensure that the magnetometer correctly measures the
Earth's magnetic field, which is crucial for determining the device's
orientation relative to the Earth's magnetic north.
• Process:
• Perform a figure-eight motion with the device to recalibrate the
magnetometer.
• The software adjusts for any soft iron (distortions due to surrounding
metals) and hard iron (permanent magnetization) effects by recalibrating
the sensor’s output.
• This calibration is often done periodically or when the device enters a
new environment.
 Use case- Drone Control in AR
Drone IMU Calibration
Inertial Measurement Unit (IMU)

• The IMU is the brain behind your drone’s stability.

• It consists of accelerometers and gyroscopes that
detect changes in pitch, yaw, and roll.
• When your drone tilts or rotates, the IMU ensures it
knows its orientation relative to the Earth.
• Over time, errors can accumulate in these sensors.
• Calibrating the IMU corrects these errors, ensuring
your drone flies accurately.

Source :https://pilotinstitute.com/drone-imu-calibration-explained/#key-takeaways
 Use case- Drone Control in AR
Drone IMU Calibration
Accelerometer
Accelerometers are responsible for measuring linear acceleration, including changes in velocity and
tilt angles. They utilize the principles of inertia to detect variations in motion.
https://www.youtube.com/watch?v=i2U49usFo10&t=107s

Gyroscope
• Every drone is constantly being subjected to a number of forces coming from different directions.
These forces, such as wind, affect the drone’s yaw, pitch, and roll making the drone very hard to
control.
• Integrated gyroscopes can almost instantly detect changes in the position of a drone and
compensate for it in such a way, that it basically seems unaffected, as it re-adjusts its position
hundreds of time every second or can hover calmly in place.
• Modern gyroscopes are manufactured with components between 1 to 100 micrometers in size and
often include sensors for multiple axes in a single package.
https://www.youtube.com/watch?v=cquvA_IpEsA&t=212s

Magnetometers
Magnetometers determine the drone’s heading or orientation to Earth’s magnetic field.
 Use case- Drone Control in AR
Drone IMU Calibration

Why Is the IMU Calibration is Important?

 Use case- Drone Control in AR

Source: Konstantoudakis, K.;Christaki, K.; Tsiakmakis, D.; Sainidis,D.; Albanis, G.; Dimou, A.; Daras, P.Drone Control in AR: An IntuitiveSystem for Single-Handed Gesture control, Drone Tracking, and Contextualized
Camera Feed Visualization in Augmented Reality.Drones 2022, 6, 43. https://doi.org/10.3390/drones6020043
 Tracking
▪ Tracking in Augmented Reality (AR) refers to the technology and techniques
used to align virtual objects with the real world in a way that makes them
appear stable and interactable.
▪ Effective tracking ensures that virtual elements remain consistent with real-
world objects, positions, and movements. Here are the primary types of
tracking used in AR:
 Marker-based Tracking
• Relies on physical triggers such as predefined images, patterns, or QR codes
(called markers) placed in the environment.
• The AR system recognizes these markers through the device’s camera, and
virtual objects are overlaid on or around the marker.
• Market Detection
• Scans environment and searches for predefined markers(high contrast patterns) that the
system recognizes
• Pose Estimation
• Computer vision techniques calculate the position and orientation of the marker relative
to the camera’s viewpoint ensuring accurate alignment of contents
• Content Overlay
• With the determined marker’s pose, digital content is overlaid onto the physical marker
in real-time blending the virtual and real worlds

• Use cases: Interactive product displays, AR games, and art exhibitions

Marker-based Tracking –Use case
Education: marker-based AR can bring textbook diagrams to life providing stude

nts with an interactive and engaging learning experience

Source: https://www.youtube.com/watch?v=d8XJ83AO-HI
 Marker-based Tracking –Use case
Retail : markers on product packaging can activate detailed visualizations of
product features, assembly instructions, or interactive demonstrations, enhancing
consumer engagement and satisfaction

Source: https://www.youtube.com/watch?v=V1Aj4WCqevk https://www.youtube.com/watch?v=ZroFBG7-P7Q

Markerless Tracking (SLAM - Simultaneous Localization
and Mapping):
• Relies on the device’s sensors (camera, accelerometer, and gyroscope) to create a
map of the environment.
• SLAM maps the space in real-time, identifying surfaces and objects where virtual
content can be placed.
• Use cases:
▪ Navigation: Indoor and outdoor navigation with virtual directions
▪ Interior Design: visualize home furniture and decor before making
purchasing decisions.
▪ Healthcare: Markerless AR is used in medical training and surgery
simulations, providing a realistic and safe environment for practice.
▪ Art & Tourism: Location-based AR reveals historical info or 3D artwork.
▪ Entertainment: AR games like Pokémon Go

• https://www.linkedin.com/posts/abdulsalam-faheedat-opeyemi_ar-ar-
markerlessar-activity-7191383388786671618-
o5Ck?utm_source=share&utm_medium=member_desktop
Markerless Tracking
 GPS/Location-based Tracking

• Uses GPS data from the device to place virtual objects at specific
geographical locations.
• This is often combined with accelerometer and compass data for
better accuracy.
• Use cases:
• Location-based AR apps, tourism guides, and social AR apps that show
content in real-world locations.
GPS/Location-based Tracking
 Projection-based Tracking
▪ uses projectors to overlay AR content directly over real-life objects
using projection mapping techniques.
▪ Projection-based AR is appealing and interactive where light is blown
onto a surface and the interaction is done by touching the projected
surface.
Projection-based Tracking-Use case

https://www.youtube.com/shorts/y-LGVeiQ1l8
Projection-based Tracking
Augmented Reality Training for Industrial Assembly Work

https://www.youtube.com/watch?v=mpSG94HTMpQ
Superimposition based Tracking
▪ In this type of AR, object recognition plays a vital role because an app
cannot replace an original object with an augmented one if it cannot
identify the original object.
▪ This type of AR has been popularized by social platforms such as
Instagram, Facebook, and Instagram using filters.
Superimposition based Tracking
▪:

Source : https://zolak.tech/blog/augmented-reality-for-furniture-industry
Types of AR-Use Cases

https://digitalpromise.org/initiative/360-story-lab/360-production-
guide/investigate/augmented-reality/getting-started-with-ar/types-of-ar/
NIKE has taken over a huge LED billboard in the Shinjuku area of Tokyo, Japan to show a forced
perspective illusion celebrating the Air Max shoe.

Source:https://www.youtube.com/watch?v=mcl2nIeJh_A&list=TLGGzBkRwcLgNJQwNTExMjAyNA&t=14s
Challenges and Advances:
▪ Lighting conditions: AR tracking can be sensitive to lighting; poor
lighting can affect camera-based tracking.
▪ Occlusion: Properly occluding virtual objects behind real-world
objects remains a challenge in achieving realism.
▪ Environmental complexity: The more detailed or dynamic an
environment is, the more sophisticated the tracking system needs to
be to avoid jitter or misplacement.
AR Computer Vision
▪ AR Computer Vision is a subset of computer vision technology
applied to Augmented Reality (AR).
▪ It enables devices to interpret and understand the physical
environment in real-time, allowing virtual objects to be overlaid
seamlessly onto the real world.
▪ This integration is essential for creating immersive AR experiences.
▪ Image Recognition
• Identifies and classifies specific images or patterns from the camera feed.
This could include logos, QR codes, landmarks, or any predefined objects.
• Using techniques like feature detection ,the system matches key points in
the image with a database to recognize the object or image.
• Use cases: AR shopping (recognizing products), educational apps
(identifying objects), AR games, and content linked to brand logos.
Cont.,
▪ Object Detection and Recognition
• Identifies and locates specific objects within a scene, such as furniture, human
faces, or even hand gestures.
• By applying machine learning models (like Convolutional Neural Networks -
CNNs) trained on specific objects, the system can classify and locate objects in
the camera’s view. This is used to place virtual objects in relation to real-world
items.
• Use cases: AR filters (detecting faces for masks), virtual try-ons (clothes,
makeup), object-based information overlays in educational apps, and AR in
robotics.
▪ Simultaneous Localization and Mapping (SLAM)
• Allows devices to map an unknown environment while tracking their
position in it.
• SLAM combines data from cameras and inertial sensors (gyroscope,
accelerometer) to continuously update a map of the environment while
simultaneously tracking the camera's position within it. This is essential for
markerless AR.
• Use cases: AR navigation, virtual object placement in physical spaces (e.g.,
furniture apps), interactive AR games, and spatial understanding.
Cont.,
▪ Depth Sensing
• Determines the distance of objects from the camera, enabling 3D spatial
awareness.
• Techniques like stereo vision (using two cameras), structured light
(projecting a known pattern onto a surface), or time-of-flight (using light
pulses) are used to measure depth.
• Newer AR devices like iPhones with LiDAR sensors can measure depth
more accurately, allowing virtual objects to interact more realistically with
the physical world.
• Use cases: Accurate virtual object placement, occlusion (making virtual
objects disappear behind real ones), AR measuring tools, and immersive
games.
Cont.,
▪ Surface Detection and Plane Detection
• Identifies horizontal and vertical surfaces where virtual objects can be
anchored, such as tables, floors, or walls.
• The system looks for patterns in pixel movement (optical flow) and texture in
the environment to detect flat surfaces. Once a surface is detected, virtual
objects can be placed with stability.
• Use cases: Furniture apps, AR art installations, and object placement in AR
games.
▪ Pose Estimation
• Determines the orientation and position of an object or person relative to the
camera.
• Using key point detection and geometric algorithms, pose estimation
identifies how an object is positioned in 3D space relative to the camera.
• This is used for aligning virtual objects with real ones or for creating realistic
interactions with the user.
• Use cases: AR gaming (aligning weapons or tools in first-person perspective),
virtual avatars, motion tracking, and gesture-based interactions.
Cont.,
▪ Feature Detection and Tracking
• Detects and tracks key visual features (edges, corners, etc.) in the real-world
environment to maintain consistent positioning of virtual elements.
• Algorithms like Harris Corner Detection, Shi-Tomasi, or FAST are used to
detect stable visual features, which can then be tracked over time to anchor
AR objects in place.
• Use cases: AR annotations on real-world objects, interactive AR games, and
tracking moving objects in AR experiences.
▪ Occlusion Handling
• Ensures that virtual objects appear behind real-world objects when
appropriate, creating a more realistic and immersive experience.
• Using depth information from depth sensors or advanced algorithms, the AR
system can determine when a real object is closer to the camera than a virtual
one and thus should occlude the virtual object.
• Use cases: Realistic AR scenes where virtual characters walk behind real
objects, or AR measuring apps where virtual measurements disappear behind
physical objects.
Cont.,
▪ Scene Understanding and Semantic Segmentation
• Identifies and categorizes different parts of the environment (e.g.,
distinguishing between the floor, walls, furniture, people, and objects).
• Machine learning models, particularly CNNs or vision transformers, are
used to segment the camera feed into different object classes (e.g., sky,
road, person, car). This enables more advanced interactions with the
environment.
• Use cases: AR city navigation (placing virtual signs and information on
roads and buildings), interactive games, and contextual AR overlays for
educational purposes.
Mapping
▪ Mapping in Augmented Reality (AR) refers to the process of creating a
digital representation of the physical environment, allowing virtual
objects to be overlaid, interact, and stay anchored in the real world.
▪ This is critical for immersive AR experiences, as it enables devices to
understand and respond to the geometry, layout, and objects in a scene.
▪ Plane Detection
• Plane detection is used to find flat surfaces (both horizontal and vertical) in
the environment, such as floors, tables, and walls. Once detected, these planes
serve as anchors for virtual objects.
• Using visual cues, such as parallel lines, textures, and depth information, the
AR system identifies flat surfaces in the environment. Planes are essential for
creating stable virtual content, ensuring objects don’t “float” or move
unrealistically.
• Use cases: Virtual furniture placement (e.g., IKEA Place), object placement in
AR gaming, and virtual art on walls.
Cont.,
▪ Simultaneous Localization and Mapping (SLAM)
• SLAM is the foundational technology for most AR mapping systems. It
allows AR devices to build a map of the environment in real time while
simultaneously determining their own position within it.
• This is done using data from the device’s camera and inertial sensors (like
accelerometers and gyroscopes).
• Process:
• Feature Detection: The system identifies key features in the environment,
such as edges, textures, and distinct objects.
• Tracking: These features are tracked across frames as the device moves.
• Map Building: As the device moves through the environment, it creates a
3D map of the surroundings, updating with new features.
• Use cases: Markerless AR, indoor navigation, AR gaming, and
placing virtual objects in physical spaces (e.g., furniture apps).
Cont.,
▪ Depth Mapping
• Depth mapping provides a sense of distance between the camera and
objects in the environment.
• Devices equipped with depth sensors, such as LiDAR or structured light
sensors, can create detailed 3D maps by calculating the distance to
surfaces.
• Depth is measured by projecting infrared light or other patterns onto the
environment and analyzing how the pattern deforms when it hits
surfaces. This allows for a precise understanding of the 3D structure of the
scene.
• Use cases: Accurate object placement, occlusion (where virtual objects
disappear behind real objects), AR measurement apps, and enhancing
realism in AR games.
Cont.,
▪ Environment Understanding
• Beyond simple plane detection, some AR systems can classify different
parts of the environment (walls, ceilings, floors, objects) and understand
their semantic meaning.
• Using machine learning and computer vision, the AR system analyzes the
scene to distinguish between different objects and surfaces.
• This can involve detecting furniture, appliances, or even recognizing
specific rooms.
• Use cases: Home design apps, where furniture is placed intelligently, AR-
based interior design, or advanced gaming experiences that interact with
the user’s surroundings.
Cont.,
▪ 3D Object Mapping and Reconstruction
• In some cases, AR needs to map not only surfaces but also complex objects
within the environment. This involves recognizing and reconstructing 3D
objects to place virtual elements correctly.
• The AR system scans the object from multiple angles, generating a 3D model.
This can be done in real time, allowing for dynamic interactions between
virtual objects and real-world objects.
• Use cases: AR games where virtual objects interact with physical objects (e.g.,
a virtual character jumping on a real sofa), or educational apps where objects
can be scanned and enhanced with virtual annotations.
Cont.,
▪ Marker-based Mapping
• Marker-based AR uses predefined images, QR codes, or fiducial markers
as anchors to position virtual objects in the physical world.
• These markers are scanned by the camera and serve as a reference point
for positioning the virtual content.
• When the AR system detects a marker, it calculates the marker’s position,
orientation, and scale in 3D space. This allows virtual objects to be placed
accurately relative to the marker.
• Use cases: Interactive product displays, museum guides, or simple AR
games.
Cont.,
▪ Occlusion and Collision Mapping
• Occlusion occurs when virtual objects are hidden behind real-world
objects, enhancing realism.
• Collision mapping ensures that virtual objects don’t pass through physical
ones, providing more realistic interaction.
• Using depth sensors or computer vision, the system determines the depth
of real-world objects and calculates when a virtual object should be
obscured.
• Collision mapping involves detecting when virtual and real-world objects
should not occupy the same space.
• Use cases: Games where virtual characters interact with real-world
objects, AR-based design apps, or simulations that involve both real and
virtual elements.
Cont.,
▪ Spatial Mapping
• Spatial mapping involves creating a detailed, comprehensive 3D model of
the environment.
• This allows AR applications to place virtual objects more accurately,
accounting for walls, furniture, and other obstacles.
• Using data from cameras, depth sensors, or even external sensors, the
system creates a mesh or point cloud representation of the environment.
• This allows for more precise interactions between virtual objects and the
physical world.
• Use cases: Industrial AR (e.g., factory maintenance), architecture and
construction apps, and immersive AR experiences in entertainment.
Cont.,
▪ Cloud-based AR Mapping
• Cloud-based AR mapping allows multiple users to share the same AR
experience by synchronizing mapping data through the cloud. This makes
it possible for virtual objects to be placed in shared physical spaces,
accessible to different users.
• AR data, such as maps and anchors, are uploaded to the cloud and shared
across multiple devices. This enables multi-user AR interactions, where
everyone sees the same virtual content in the same location.
• Use cases: Multi-user AR games, collaborative design apps, or tourism
apps with shared AR experiences.
References
▪ https://medium.com/echo3d/top-10-ar-developer-tools-every-
developer-should-know-32fef1471883
▪ https://aimagazine.com/top10/top-10-augmented-reality-ar-
platforms
▪ https://daily.dev/blog/ar-tools-for-developers
▪ https://program-ace.com/blog/types-of-ar/