A Merged
A Merged
Submitted By
Ms. Nivedita. G Y
Assistant Professor
Department of ISE
CERTIFICATE
Certified that the project work phase 1 entitled Hawk_Eye – AI powered smart surveillance
with anomaly detection and behaviour analysis has been successfully completed by Pramod
Raghavendra Ilal (1RN22IS110), Nagaraj Subraya Bhat (1RN22IS090), Mohit
Shamsundar Bidanur (1RN22IS087), Manoj K S (1RN22IS082) bona fide students of RNS
Institute of Technology, Bengaluru in partial fulfillment of the requirements for the award
of degree in Bachelor of Engineering in Information Science and Engineering of
Visvesvaraya Technological University, Belagavi during academic year 2024-2025. The
project phase 1 report has been approved as it satisfies the academic requirements in respect of
project phase 1 work for the said degree.
External Viva
Name of the Examiners Signature with Date
1. _ 1. _ _
_
2. _ 2.
DECLARATION
We, Pramod Raghavendra Ilal [1RN22IS110], Nagaraj Subraya Bhat [1RN22IS090], Mohit
Shamsundar Bidanur [1RN22IS087] and Manoj K S [1RN22IS082], students of VI Semester
BE, in Information Science and Engineering, RNS Institute of Technology hereby declare that
the Project work phase 1 entitled Attendance Management System Using Face Recognition has
been carried out by us and submitted in partial fulfillment of the requirements for the VI Semester
degree of Bachelor of Engineering in Information Science and Engineering of Visvesvaraya
Technological University, Belgaum during academic year 2024-2025.
Place: Bengaluru
Date:
PRAMOD R ILAL[1RN22IS110]
MANOJ K S [1RN22IS082]
ABSTRACT
Additionally, the system performs basic behaviour analysis by interpreting abnormal activity
as a deviation from normal movement patterns, thus providing insights into suspicious or
unexpected behaviour without explicitly classifying individual actions. This behaviour
analysis contributes to a more holistic understanding of the surveillance context.
Overall, the integrated framework offers a robust, offline video surveillance solution capable
of highlighting anomalous and unattended events while supporting basic activity-based
behavioural understanding.
I
ACKNOWLEDGMENT
The fulfillment and rapture that go with the fruitful finishing of any assignment would be
inadequate without the specifying the people who made it conceivable, whose steady direction and
support delegated the endeavors with success.
We would like to express our thanks to our Principal Dr. Ramesh Babu for his support and
inspired us towards the attainment of knowledge.
We wish to place on record my words of gratitude to Dr. Suresh L, Professor and Head of the
Department, Information Science and Engineering, for being the enzyme and master mind behind our
project work.
We would also like to thank our project coordinator Ms. Nivedita G Y, Assistant Professor,
Department of ISE, RNSIT, Bangalore, for their valuable suggestions.
We would like to thank all other teaching and non-teaching staff of Information Science &
Engineering, RNSIT, Bangalore, who have directly or indirectly helped us to carry out the project
work
Place: Bengaluru
Date:
PRAMOD R ILAL[1RN22IS110]
MANOJ K S [1RN22IS082]
II
TABLE OF CONTENTS
CERTIFICATE
DECLERATION
ABSTRACT I
ACKNOWLEDGMENT II
TABLE OF CONTENTS III
LIST OF FIGURES IV
ABBREVATIONS V
1. INTRODUCTION 1
2. LITERATURE SURVEY 3
3. PROBLEM STATMENT 17
4. OBJECTIVES 18
5. METHODOLOGY 19
6. SYSTEM DESIGN 25
7. CONCLUSION 29
8. REFERENCE 30
III
LIST OF FIGURES
Fig No. Description Page No.
Figure 5.1 Structural Chart of the proposed system 19
Figure 6.1 Flowchart of the system 28
IV
LIST OF ABBREVIATIONS
AI – Artificial Intelligence
UI – User Interface
V
Chapter 1
INTRODUCTION
1.1 Overview
In the modern world, video surveillance plays a vital role in ensuring public safety and
monitoring environments like airports, train stations, and public places. Traditional systems
depend heavily on human operators to detect suspicious behavior, which can be time-
consuming, error-prone, and inefficient when monitoring multiple video feeds. To address these
limitations, this project introduces an intelligent video surveillance system using machine
learning for anomaly detection, unattended object tracking, and behavior analysis.
The system is designed to process offline video footage and identify unusual events based on
patterns learned from normal behavior. Using deep learning (ConvLSTM Autoencoders), the
model detects frame-level anomalies without needing labeled abnormal data. If an activity
deviates from expected motion patterns, it is marked as abnormal.
Additionally, the project includes an unattended object detection feature. When an object is left
unattended for a certain time, the system flags it, which is useful in public safety applications.
Behavior analysis builds upon the anomaly detection module by interpreting detected
irregularities. While it does not explicitly label activities, it helps understand what kind of event
is occurring based on deviations in behavior, contributing to smarter surveillance decisions.
The behavior analysis component attempts to categorize the nature of actions present in the
scene (e.g., walking, standing, bending) using spatial-temporal features extracted from the
videos. Additionally, the system includes an unattended object timer that flags objects left in
one location for too long, indicating possible security risks.
The project is implemented using Python, TensorFlow, OpenCV, and Streamlit for a user-
friendly interface. While real-time processing is not included, this system effectively
demonstrates the core concepts of intelligent video analysis through offline video datasets.
While real-time video processing is not implemented, this system effectively demonstrates the
core principles of intelligent video surveillance through offline video datasets. It simulates how
modern surveillance systems can enhance security and awareness in environments like
campuses, offices, and public areas using machine learning and video analytics.
The project is built using Python, TensorFlow, OpenCV, and Streamlit, offering a user-friendly
interface to visualize outputs such as abnormal event detection, identified behaviors, and object
tracking timelines. Users can upload pre-recorded videos and analyze them through the
application, gaining insights into various scene dynamics.
One of the system’s key features is the detection of unattended objects. The system continuously
monitors objects in the video and triggers a timer when an object is left unmoved for a specific
period, helping identify potential security threats such as abandoned luggage.
LITERATURE REVIEW
connectivity for centralized management, edge analytics for instant alert generation, and
an intuitive dashboard for operators. Specific use cases are discussed such as unattended
baggage detection, loitering, and crowd congestion. The authors emphasize real-time
alerting, scalability, and seamless integration with existing CCTV systems. Moreover,
they address key challenges including data imbalance, interpretability of AI decisions,
and hardware-software compatibility. The proposed solution shows promise for urban
security, airport monitoring, and public event management.
Limitations: While the paper presents a comprehensive framework for real-time AI-
powered threat detection in surveillance systems, it also acknowledges certain
limitations. These include challenges related to the accuracy of detection algorithms,
especially in complex environments, and the potential for false positives or negatives.
Additionally, the deployment of such AI systems raises concerns about privacy and the
ethical implications of constant surveillance. The authors suggest that addressing these
issues is crucial for the effective and responsible implementation of AI-powered
surveillance systems.
2.6. Title: "Towards a Conceptual Framework for AI-Driven Anomaly Detection in Smart
City IoT Networks"
Author: Heng Zeng , Manal Yunis, Ayman Khalil
Year:2024
Abstract: This paper presents a conceptual framework designed to detect anomalies and
security threats in smart city environments using AI techniques. It discusses the unique
challenges in IoT networks, such as data heterogeneity, real-time response needs, and
complex sensor interconnectivity. The proposed framework integrates machine learning
with context-aware anomaly detection, focusing on scalability and interoperability. It
includes data preprocessing, feature extraction from IoT devices, and adaptive learning
modules. The model emphasizes flexibility to accommodate different urban
infrastructures and threat types. The authors suggest integrating edge and cloud
computing for real-time processing, and outline potential use cases like traffic
monitoring, utility management, and emergency response. Future research directions
include addressing privacy concerns and improving explainability in AI decision
making.
Limitations: The paper identifies several challenges in implementing AI-driven
anomaly detection within smart city IoT networks. One significant issue is the difficulty
in obtaining high-quality, labeled datasets for training, which impacts model accuracy.
Many AI models struggle with real-time processing, making them less effective in time-
sensitive applications. Additionally, explainability remains a concern, as black-box
models often provide little insight into their decision-making process, reducing trust and
transparency. The presence of noisy or imbalanced data further complicates anomaly
detection, leading to false positives or negatives.
2.8. Title: "Deep Learning-Based Anomaly Detection in Video Surveillance: A Survey "
Author: Huu-Thanh Duong Viet-Tuan Le Vinh Truong Hoang.
Year:2023
Abstract: This survey reviews the landscape of deep learning-based anomaly detection
in video surveillance. It classifies methods based on architecture—CNNs, RNNs,
autoencoders, GANs, and hybrid models. The paper compares various algorithms in
terms of accuracy, processing speed, and scalability. It discusses benchmark datasets
such as UCSD Ped1/Ped2, Shanghai Tech, and UCF-Crime. Key challenges identified
include scarcity of labelled anomaly data, high computational cost, and the
generalization gap between training and deployment environments. The survey
highlights trends toward using self-supervised learning, attention mechanisms, and
transformer models. Future directions include real-time edge computing, privacy-
preserving analytics, and cross-domain anomaly detection.
Limitations: The paper identifies several challenges in implementing deep learning-
based anomaly detection within video surveillance systems. One significant issue is the
difficulty in obtaining high-quality, labeled datasets for training, which impacts model
accuracy. Many AI models struggle with real-time processing, making them less
effective in time-sensitive applications. Additionally, explainability remains a concern,
as black-box models often provide little insight into their decision-making process,
reducing trust and transparency. The presence of noisy or imbalanced data further
complicates anomaly detection, leading to false positives or negatives.
2.10. Title: "Video Anomaly Detection System Using Deep Convolutional and Recurrent
Neural Networks."
Author: Maryam Qasim, Elena Verdu.
Year:2023
Abstract: This paper presents a deep learning model that integrates Convolutional
Neural Networks (CNNs) for spatial feature extraction and Recurrent Neural Networks
(RNNs), particularly LSTM units, for temporal pattern recognition in video anomaly
detection. The model is designed to detect unusual behaviours such as running in
restricted areas or loitering. A two stage pipeline processes raw video frames into spatial
features and sequences, which are then analysed for deviations from learned normal
patterns. Evaluation on datasets like CUHK Avenue and Shanghai Tech shows F1-
scores above 90%. The authors address challenges like temporal coherence and real-
time inference by optimizing the model using pruning and frame sampling. The hybrid
approach achieves robust performance across diverse video scenarios, suggesting
applicability in both public and private surveillance systems.
Limitations: The paper identifies several challenges in implementing deep
convolutional and recurrent models for video anomaly detection. One significant issue
is the difficulty in obtaining high-quality, labeled datasets for training, which impacts
model accuracy. Many AI models struggle with real-time processing, making them less
effective in time-sensitive applications. Additionally, explainability remains a concern,
as black-box models often provide little insight into their decision-making process,
reducing trust and transparency. The presence of noisy or imbalanced data further
complicates anomaly detection, leading to false positives or negatives.
2.11. Title: " AI-Driven Anomaly Detection for Cybersecurity: A Survey, Taxonomy,
and Open Issues"
Year:2023
Abstract: This paper presents a comprehensive survey on AI-driven anomaly detection
techniques in the context of cybersecurity. It categorizes existing approaches into
statistical, machine learning, and deep learning-based models. The paper also introduces
a taxonomy that classifies techniques based on their learning paradigm, data type, and
application domain. Furthermore, it discusses the strengths and limitations of these
approaches, outlines current challenges, and identifies future research directions to
enhance the effectiveness of anomaly detection systems in cybersecurity.
Limitations: This paper highlights several limitations in AI-driven anomaly detection
within cybersecurity. Chief among them is the scarcity of labeled datasets, which poses
a challenge for supervised learning models. Additionally, the dynamic and evolving
nature of cyber threats introduces concept drift, making static models less effective over
time. The study also notes the vulnerability of these systems to adversarial attacks,
which can deceive models into misclassifying threats. Furthermore, scalability remains
a concern, particularly when deploying AI models in large-scale, real-time
environments.
2.13. Title: " A Comprehensive Review on Deep Learning-Based Methods for Video
Anomaly Detection "
2.14. Title: " Smart Surveillance Systems Using YOLOv8: A Scalable Approach for
Crowd and Threat Detection "
architecture. The paper does not delve into how the model handles scenarios involving
low lighting, heavy occlusion, or complex human interactions, which are common in
surveillance footage. Moreover, its effectiveness is tied to the quality and diversity of
training data. If the training dataset lacks representation of certain threat types or
behaviors, the system may fail to detect them. The approach also assumes that threat
objects are always visually identifiable, overlooking non-visual cues that might be
critical in real-world security contexts.
2.16. Title: " Anomaly Detection and Prediction of Human Actions in a Video
Surveillance Environment "
detection and prediction. The method uses spatiotemporal features and context
awareness for improved accuracy, achieving over 92% F1-score on benchmark datasets.
Limitations: While CT-D2GAN presents an innovative unsupervised approach for
capturing spatiotemporal patterns in video data, it faces several limitations. The paper
points out that existing models often fail to effectively balance local (frame-level) and
global (temporal) consistency, and CT-D2GAN aims to address this gap. However,
despite improvements, the model’s complexity may hinder real-time performance,
especially on resource-constrained devices. Furthermore, unsupervised methods like
this often struggle with precision due to the lack of ground truth during training. There
is also no discussion on how the model adapts to different environments or types of
anomalies, suggesting potential limitations in generalizability and deployment.
2.17. Title: " Video Anomaly Detection for Smart Surveillance: A Deep Learning
Approach "
2.18. Title: " A Survey on Anomalous Human Activity Recognition Using Video
Analytics "
2.19. Title: " Deep Learning for Anomaly Detection in Surveillance Videos: Challenges
and Solutions "
2.20. Title: " Predictive Analytics for Human Behavior in Surveillance Videos Using AI
Techniques "
Traditional surveillance systems heavily rely on continuous human monitoring, which is time-
consuming, prone to error, and inefficient, especially when managing multiple video feeds.
Human operators may miss critical events due to fatigue, distraction, or information overload,
resulting in delayed responses to security threats. Moreover, conventional systems lack the
intelligence to automatically detect unusual behavior, unattended objects, or potential threats in
real time.
There is a growing need for an automated, intelligent surveillance solution that can analyze
video footage in real time, detect anomalies, and interpret human behavior without constant
human supervision. Such a system should be capable of identifying events like loitering,
aggressive movements, or abandoned objects and alerting security personnel instantly to
prevent or respond to incidents swiftly.
This project, Hawk-Eye: AI-Powered Smart Surveillance with Anomaly Detection &
Behavior Analysis, addresses this need by leveraging advanced AI and machine learning
technologies such as YOLOv8 for real-time object detection, MediaPipe Pose for behavior
tracking, and LSTM-based models for anomaly detection. The goal is to enhance situational
awareness, reduce reliance on manual monitoring, and improve overall security effectiveness
across various high-risk environments
SYSTEM IMPLEMENTATION
This chapter elaborates on the systematic implementation of the Smart Surveillance System. It
encompasses the structural design, the integration of key modules, and the deployment strategies
employed for anomaly detection, behavioral analysis, and unattended object monitoring using
deep learning and computer vision techniques.
The structural chart of the proposed system outlines the sequential flow of operations and the
interaction between various components of the Smart Surveillance System. The process begins
with data input and ends with the classification and alert generation based on detected anomalies
or behaviors.
Step-wise Explanation:
o The process starts with uploading pre-recorded surveillance footage into the
system via the Streamlit interface. The video acts as the primary input and can be
from any standard CCTV feed in MP4 or AVI format.
o The system supports batch upload and segment-wise processing for enhanced
performance and better memory management.
o The uploaded video is broken down into individual frames using OpenCV. This
allows frame-by-frame analysis which is essential for temporal modeling and
behavior tracking.
o Frames are resized and normalized to maintain uniformity and ensure compatibility
with the trained model dimensions.
3. Feature Extraction:
o For each frame, spatial features (object boundaries, contours, and textures) are
extracted using convolutional layers.
o Frames with high reconstruction error beyond a set threshold are flagged as
potential anomalies, indicating a deviation from learned normal behavior.
5. Behavior Analysis:
o The system evaluates behavior by analyzing frame sequences over time. It clusters
patterns based on movement and pose dynamics without explicitly labeling
specific activities.
o The temporal analysis is supported using optical flow and recurrent observations
of keypoints and frame changes.
o This module helps the system understand whether the observed action is consistent
with regular movement patterns or indicates an unusual behavioral pattern.
o If an object remains in the same position beyond a predefined time threshold, the
system logs it as an unattended object, potentially indicating a security risk.
o The interface also includes visualization tools such as reconstruction error graphs,
frame differencing heatmaps, and behavior timelines.
The pipeline is a crucial backbone of the system, designed to ensure smooth operation from data
ingestion to final output generation. It includes the following stages:
1. Frame Loader:
• Frames are stored in a queue for asynchronous processing to avoid memory overflow.
2. Preprocessing Unit:
• Performs:
o Grayscale conversion
• Converts each frame into a format suitable for model input (e.g., numpy arrays or tensors).
• Utilizes Mean Squared Error (MSE) between input and reconstructed output to determine
anomaly levels.
• Applies object detection (e.g., background subtraction + contour tracking) to locate static
items.
• Uses a timer to check how long each object has remained in place without moving.
Behavioral heatmaps and trajectory maps are created to visualize movement consistency or
irregularity.
Encoder:
• Series of convolutional and pooling layers to compress input frame into a latent
representation.
Decoder:
• Transposed convolutional layers to reconstruct the input frame from the encoded latent
vector.
Training Strategy:
To enhance usability, a web interface built using Streamlit is provided, which includes:
Chapter 6
SYSTEM DESIGN
6.1 System Environment
The system architecture for the MERN-based Quiz Web Application is designed to ensure a
seamless interaction between the front-end and back-end, providing a robust, scalable, and
efficient platform. Here’s an overview of the architecture:
The system environment of a Smart Surveillance and Anomaly Detection System is the
physical and software infrastructure that ensures efficient, secure, and reliable operation.
• Hardware:
Powerful enough to handle video frame extraction, deep learning-based anomaly
detection, behavior analysis, and object monitoring efficiently. A system with a multi-core
processor, sufficient RAM, and preferably a dedicated GPU for faster model inference is
recommended.
• Software:
Compatible with machine learning, computer vision, and video processing libraries (like
TensorFlow, OpenCV, and Streamlit). Provides a secure and stable environment for
sensitive video data handling and anomaly detection processes.
• Network:
Reliable and with sufficient bandwidth for uploading analysis results, generating alerts,
and optionally sending notifications. If remote monitoring is enabled, it should support
smooth data transfer without interruptions.
• User Interface:
Provides an interactive, user-friendly dashboard (built with Streamlit) to view live or
processed results, alerts, and activity logs.
• Administration Interface:
Enables administrators to manage video datasets, configure detection thresholds, and
monitor system health and performance securely.
• Security:
Protects video data and system configurations through encryption, access controls, and
secure communication channels.
• Performance:
Ensures efficient frame extraction, model inference, and result generation to process
surveillance data quickly and reliably.
• Scalability:
Supports the integration of larger video datasets, additional camera feeds, and expanded
detection functionalities as required.
• Maintainability:
Designed to be easily updated with new models, detection parameters, and UI
enhancements as technology and security needs evolve.
• The user interface must be easy to use and navigate for uploading videos and viewing
analysis results.
• The administration interface must be secure and allow administrators to manage video
data, configure detection settings, and monitor system performance.
• The system must be able to collect, process, and store large volumes of video data.
• The system must be able to operate in various physical and network environments.
• The data collection and storage system is secure and ensures privacy protection.
• The system will remain scalable and maintainable for future upgrades and
enhancements.
Chapter 7
CONCLUSION
7.1 Conclusion
Hawk-Eye: AI-Powered Smart Surveillance with Anomaly Detection & Behavior Analysis
redefines the way surveillance systems function by integrating advanced artificial intelligence
techniques to detect, interpret, and respond to unusual activities. Traditional surveillance heavily
relies on manual monitoring, which is often inefficient and error-prone, especially in
environments with multiple video feeds. Hawk-Eye overcomes these limitations by automating
video analysis using machine learning models that are capable of detecting frame-level anomalies,
tracking unattended objects, and analyzing human behavior without requiring manually labeled
data. This makes the system highly scalable and adaptable for real-world security applications.
At the heart of Hawk-Eye lies a deep learning-based ConvLSTM autoencoder model, which learns
typical patterns of motion and activity from training video footage. Any significant deviation from
these patterns is flagged as anomalous, allowing the system to detect events such as sudden
movements, unauthorized presence, or unexpected object placement. This unsupervised learning
approach not only reduces the dependency on manually annotated datasets but also enables the
system to adapt to different environments with minimal retraining. Additionally, the unattended
object detection module ensures that any object left in a fixed location beyond a predefined
duration is identified and flagged, enhancing threat detection in sensitive areas such as airports,
offices, and public venues.
Although Hawk-Eye currently processes pre-recorded video files rather than real-time feeds, it
sets a strong foundation for future enhancements. Planned upgrades such as real-time video stream
support, multi-camera integration, and audio-based anomaly detection will further elevate its
capabilities. By transforming passive video monitoring into an intelligent, automated security tool,
Hawk-Eye demonstrates how AI can significantly improve situational awareness, response times,
and overall safety in both public and private spaces.
REFERENCES
[1]. Chris, E., Olabiyi, W., Favor, B., & Ramon, T. (2023). "AI-powered surveillance systems
and anomaly detection." Available at: ResearchGate
[2]. (2023). "Deep Learning-Based Anomaly Detection in Video Surveillance." Available at:
MDPI
[3]. (2023). "AI-Powered Anomaly Detection Techniques." Available at: ResearchGate
[4]. (2023). "Revolutionizing Video Surveillance: AI-Powered Anomaly Detection."
Available at: ResearchGate
[5]. (2023). "AI-powered threat detection in surveillance systems: A real-time data processing
framework." Available at: ResearchGate
[6]. (2024). "Towards a Conceptual Framework for AI-Driven Anomaly Detection in Smart
City IoT Networks." Available at: ScienceDirect
[7]. (2023). "AI-Based Anomaly Detection for Real-Time Cybersecurity." Available at:
ResearchGate
[8]. (2023). "Deep Learning-Based Anomaly Detection in Video Surveillance: A Survey."
Available at: ResearchGate
[9]. (2023). "AI-Powered Anomaly Detection for Real-Time Performance Monitoring in
Cloud Systems." Available at: ResearchGate
[10]. (2023). "Video Anomaly Detection System Using Deep Convolutional and
Recurrent Neural Networks." Available at: ScienceDirect
[11]. (2023). "AI-Driven Anomaly Detection." Available at: ResearchGate
[12]. (2023). "Smart Surveillance System for Anomaly Recognition." Available at:
ResearchGate
[13]. (2023). "A Comprehensive Review on Deep Learning-Based Methods for Video
Anomaly Detection." Available at: ScienceDirect
[14]. (2023). "Smart Surveillance Systems Using YOLOv8: A Scalable Approach for
Crowd and Threat Detection." Available at: ResearchGate
[15]. (2021). "Convolutional Transformer-Based Dual Discriminator Generative
Adversarial Networks for Video Anomaly Detection." Available at: arXiv