Architecture: ARM64 vs AMD64

Since we were running and testing our server on an M-series mac, which have ARM-based chips, we already had our dependencies for an ARM64 device. Therefore, we decided to start with trying to build for this architecture.

The only problem was that some of the requirements on a linux system were slightly different, but after a bit of trial and error, we eventually ended up with the following Dockerfile:

FROM python:3.12-slim

 # Add application sources with correct permissions for OpenShift
 USER 0
 WORKDIR /app

 # Copy requirements files
 COPY requirements.txt Pipfile Pipfile.lock ./

 # Install ALL system dependencies in one layer to reduce image size
 RUN apt-get update && apt-get install -y --no-install-recommends \
     build-essential \
     cmake \
     git \
     libssl-dev \
     ninja-build \
     libgl1-mesa-glx \
     libglib2.0-0 \
     libsm6 \
     libxext6 \
     libxrender1 \
     ffmpeg \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*

 # Install Python dependencies
 RUN pip install --no-cache-dir --upgrade pip && \
     pip install --no-cache-dir opencv-python-headless && \
     pip install --no-cache-dir -r requirements.txt

 # Copy the rest of the application code
 COPY . .

 # For security, run as non-root user
 RUN useradd -m appuser && chown -R appuser:appuser /app
 USER appuser

 # Expose port for the Litestar application
 EXPOSE 8000

 # Start the Litestar application with uvicorn
 CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

With this, we were finally able to run the docker file on an ARM64 linux machine. However, after trying to deploy this to IBM Cloud, we realized that all of the IBM Cloud hosts use AMD architecture.

We thought that most of the requirements should still be the same - that they will just be built for AMD architecture computers, but we were sorely mistaken. Amongst missing dlib requirements, long build times (and failures), and mismatched versions, we had to spend a significant amount of time trying to build for an AMD Linux system. In fact, at one point, we put it on hold while focusing on other parts of the project, even looking for alternative deployment solutions.

However, in the end, we finally managed to downgrade the python version, match the other dependency versions, and install all the necessities for dlib to run, and got a working container with AMD64. The Dockerfile for this looked like the following:

FROM python:3.9-slim

# Set up a working directory
WORKDIR /app

# Install system dependencies required for dlib, OpenCV, and general builds
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    cmake \
    wget \
    curl \
    g++ \
    gcc \
    make \
    git \
    libssl-dev \
    ninja-build \
    libgtk2.0-dev \
    libboost-all-dev \
    libgl1-mesa-glx \
    libglib2.0-0 \
    libsm6 \
    libxext6 \
    libxrender1 \
    ffmpeg \
    libopenblas-dev \
    liblapack-dev \
    libx11-dev \
    libjpeg-dev \
    libpng-dev \
    libtiff-dev \
    libavcodec-dev \
    libavformat-dev \
    libswscale-dev \
    libxext-dev \
    libatlas-base-dev \
    libhdf5-dev \
    libv4l-dev \
    libxvidcore-dev \
    libx264-dev \
    libjxl-dev \
    pkg-config \
    python3-dev \
    python3-pip \
    python3-setuptools \
    python3-wheel \
    gnupg \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

# Install a specific version of CMake (3.25.2)
RUN wget https://github.com/Kitware/CMake/releases/download/v3.25.2/cmake-3.25.2-linux-x86_64.sh -O /tmp/cmake.sh && \
    chmod +x /tmp/cmake.sh && \
    /tmp/cmake.sh --skip-license --prefix=/usr/local && \
    rm /tmp/cmake.sh && \
    cmake --version

# Install Pipenv
RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir pipenv

# Set environment variable to force Pipenv to use the same directory for the virtual environment
ENV PIPENV_VENV_IN_PROJECT=1

# Copy only Pipenv files first for better caching
COPY Pipfile Pipfile.lock ./

# Install dependencies inside the virtual environment
ARG BUILD_PKGS="wget build-essential cmake"
ARG CLANG_DEPS=""
ARG CLANG_PKGS="clang-15 lldb-15 lld-15"
RUN pipenv install --deploy --ignore-pipfile --python 3.9 && \
    pipenv run pip install --no-cache-dir "numpy<2" && \
    pipenv run pip install --no-cache-dir cmake==3.25.2

# Add LLVM repository key safely (fixes gnupg error)
RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | gpg --dearmor | tee /usr/share/keyrings/llvm-snapshot-keyring.gpg > /dev/null

# Install LLVM toolchain
RUN echo "deb [signed-by=/usr/share/keyrings/llvm-snapshot-keyring.gpg] http://apt.llvm.org/bookworm/ llvm-toolchain-bookworm main" >> /etc/apt/sources.list && \
    echo "deb-src [signed-by=/usr/share/keyrings/llvm-snapshot-keyring.gpg] http://apt.llvm.org/bookworm/ llvm-toolchain-bookworm main" >> /etc/apt/sources.list && \
    echo "deb [signed-by=/usr/share/keyrings/llvm-snapshot-keyring.gpg] http://apt.llvm.org/bookworm/ llvm-toolchain-bookworm-15 main" >> /etc/apt/sources.list && \
    echo "deb-src [signed-by=/usr/share/keyrings/llvm-snapshot-keyring.gpg] http://apt.llvm.org/bookworm/ llvm-toolchain-bookworm-15 main" >> /etc/apt/sources.list && \
    apt-get update && apt-get install -y clang-15 lldb-15 lld-15

RUN pipenv run pip install --no-cache-dir opencv-python-headless imutils face-recognition setuptools litestar

# Copy the rest of the application
COPY . .

# Ensure correct user permissions
RUN useradd -m appuser && chown -R appuser:appuser /app
USER appuser

# Expose the required port
EXPOSE 8000

# Start the Litestar application with pipenv
CMD ["pipenv", "run", "python", "-m", "litestar", "run", "--host", "0.0.0.0", "--port", "8000"]

As one can see, the number of requirements significantly increased when changing architectures, and we also needed to install the LLVM toolchain, which we will now talk about.

Why do we need the LLVM Toolchain?

Firstly, an LLVM toolchain is a collection of modular and reusable compiler and toolchain technologies, such as Clang and LLDB. The main reason we wanted to install the LLVM toolchain explicitly like this came down to 3 distinct reasons:

1. Better build compatibility: with Clang, some of our dependencies compiled much more smoothly (and quickly) than with GCC.

2. Advanced options: in general, Clang offers better optimization flags and debugging features for particular special use-cases (such as with cryptographic libraries and code which is geared towards performance).

3. Future-proofing: in the case that we needed to compile additional code using newer versions of C or C++, we wanted to ensure we already had LLVM.

Dlib and Dependencies

As we mentioned earlier, there were a lot of requirements that needed to be satisfied for dlib to run on the AMD architecture. Dlib is generally known to be like this (having a lot of dependencies), especially when it comes to GPU-intensive or image processing features.

From the second Dockerfile example above, all of the lib* dependencies were installed solely for dlib - generally, whenever there are missing packages for dlib, these end up being system dev libraries. We had to spend a lot of time going through the error logs and cross-referencing the package names that we needed to install (because the names sometimes differ between Linux flavours).

Pre-building Liboqs and Multi-stage Building

While trying to deploy to IBM Cloud, we realized that, every time we try to run the docker container, we were rebuilding liboqs from scratch. Obviously, this meant longer deployment times, and also meant that if something ever changed in the liboqs git repo (since we were cloning from there and building), we might end up breaking old code.

To counter this, we decided to already build liboqs and have it pre-built in the container so it wouldn’t have to rebuild from scratch every time the docker container is run. In order to do this, we had to do a multi-stage approach: first, clone and compile liboqs; then add the command COPY --from=builder /usr/local /usr/local, which copies the compiled files to the shared libraries directory to be used by the environment.

Going into a bit more detail for the multi-stage build, in the first stage, we installed all the necessary compiler tools and libraries, and then installed liboqs. The Dockerfile section for this was something like the code below.

FROM python:3.9-slim AS builder

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    git \
    cmake \
    ninja-build \
    build-essential \
    libssl-dev \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

WORKDIR /opt

RUN git clone --depth=1 --branch main https://github.com/open-quantum-safe/liboqs.git

WORKDIR /opt/liboqs
RUN mkdir build && cd build && \
    cmake -GNinja \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_INSTALL_PREFIX=/usr/local .. && \
    ninja && \
    ninja install

For the second and final stage, as we said before, we only copied the compiled artifacts from the builder image into a slimmer version of Python. We then kept the environment variables for the newly copied libraries, and finally installed the dependencies via pipenv as before. The changes to the code in second section are also shown below.

FROM python:3.9-slim

COPY --from=builder /usr/local /usr/local
ENV LD_LIBRARY_PATH="/usr/local/lib:${LD_LIBRARY_PATH:-}"

# the rest is the same as before

At this point, we were done containerizing our server, but we now had to actually deploy the server to IBM Cloud, which was a massive problem in and of itself.

March Update Video

Below is a video that summarizes the above information and what we have worked on in the last month or so.

To counter these hallucinations, we needed to ensure Granite’s responses only included commands that were located in our JSON. Even though we gave the model a system prompt saying something along the lines of “ensure that your response commands only include commands included in the large JSON”, we still had no luck.

We went to our IBM mentor, Shalini Harkar, with this problem and had a conversation about how we could tackle this problem. After a plethora of discussion and briefly delving into various approaches, we finally reached the conclusion of using NLP to create a similarity score between the user’s prompt and the JSON settings options.

Parsing the JSON for the Commands

So, the first thing we needed to do was to parse the JSON so that, rather than including a bunch of extra information, we could strip it down to only include the necessary setting name (e.g. zoom) and the relevant command for the current Operating System.

To find the Operating System, we used Rust and Tauri’s own system - using the config and target_os which was being built for. We implemented it using something along the lines of the following code:

#[tauri::command]
fn get_platform_info() -> String {
    #[cfg(target_os = "macos")]
    {
        return "macos".into();
    }
    #[cfg(target_os = "windows")]
    {
        return "windows".into();
    }
    #[cfg(target_os = "linux")]
    {
        return "linux".into();
    }
}

There is one caveat with the above code, though. With Llinux, we actually needed to ensure that the GUI running is GNOME or GNOME-based. This is because the gsettings commands we had saved in our JSON (since this application in its entirety is simply a prototype and doesn’t cover every single case of various OSes and GUIs), only work for GNOME-based GUIs running on Linux systems. Therefore, we used the environment variable XDG_CURRENT_DESKTOP and DESKTOP_SESSION to find out if the current GUI used GNOME. The code for that was as follows:

#[cfg(target_os = "linux")]
fn get_linux_gui() -> Option<String> {
    if let Ok(desktop) = std::env::var("XDG_CURRENT_DESKTOP") {
        if desktop.to_lowercase().contains("gnome") {
            return Some("gnome".to_string());
        }
        return Some(desktop);
    }

    if let Ok(session) = std::env::var("DESKTOP_SESSION") {
        if session.to_lowercase().contains("gnome") {
            return Some("gnome".to_string());
        }
        return Some(session);
    }

    None
}

The above function (get_linux_gui()) was actually used in the #[cfg(target_os = "linux")] part of the get_platform_info() method, so in reality, the final part of the get_platform_info() function looked like the following:

    #[cfg(target_os = "linux")]
    {
        let frontend_env = get_linux_gui();
        if let Some(env) = frontend_env {
            if env.to_lowercase().contains("gnome") {
                return "gnome".into();
            } else {
                return format!("linux-{env}").into();
            }
        } else {
            return "linux-unknown".into();
        }
    }

Attempt 1: Jaro-Winkler Similarity

The quickest way to avoid random suggestions was to compare the user’s request to each key in our JSON one at a time, using a string similarity algorithm. The very first algorithm we tried was the Jaro-Winkler algorithm. This is becuase it is very good at quickly measuring how similar two small strings are, and we wanted something that would work quickly and give the LLM the result as soon as possible to reduce the latency of the application.

However, only after implementing the algorithm did we realize that it is not actually very good when it has to deal with multiple words or phrases, let alone sentences. For example, even “increase the text size” is too long for Jaro-Winkler to give a high similarity score when being compared with the word “zoom” or “text-scaling-factor”. The partial matches with the other settings made things quite messy, and we ended up with an extremely irregular distribution of similarity scores, which had no correlation to the input.

Of course, for simple inputs such as “zoom in”, Jaro-Winkler performed splendidly well, but overall, this wasn’t a good enough algorithm for our purposes (which we realized as a result of extensive testing - including user testing). So, we had to find a different algorithm that could still generate a score quickly, while supporting relatively long phrases/sentences.

Attempt 2: Cosine Similarity

Assuming that every word or phrase (also known as lemma in computational linguistics) can be represented by a vector in some higher-dimensional space, this method tries to find the angle between the vectors and eventually calculate a value between -1 and 1 in order to classify the similarity between the two lemmas. Cosine similarity is known to be really good at capturing the meaning of sentences, hence why we chose this algorithm.

The basics steps we went through were quite similar to the steps during the Jaro-Winkler attempt, but for the sake of clarity, I will explicitly outline them below:

1) Embed the JSON keys: first, we had to convert each key in the JSON (e.g. “zoom”, “cursor-size”, etc.) into its vector representation using a word embedding model.

2) Embed the user prompt: Similarly, we had to convert the user’s query into its own vector. Since it’s mostly going to be made up of multiple words/lemmas, we needed to add all these separate vectors together, but this was done by the library that we chose to use to implement this algorithm (Rust-Stemmers).

3) Compute the cosine similarity: this was probably the easiest step, as we simply needed to convert the mathematical formula into code.

4) Choose the highest score: again, this step was fairly straightforward. From the list of scores we generated for each JSON setting, we just needed to get the absolute value for each of them and then select the key with the highest magnitude.

5) Filter the JSON: After finding the most relevant key, we just had to filter out all of the other keys from the JSON and prepend this snippet to the beginning of the user prompt.

Overall, using this algorithm significantly reduced the hallucinations that Granite was having. This is because, unlike Jaro-Winkler, cosine similarity is able to handle multiple words, synonyms, and also look at contextual meaning.

Final Steps

Eventually, using NLP lead to a lot more accurate results with the commands being generated by the LLM. However, there were still a few instances of some hallucinations, but we reckon that, at this point, those are mostly due to the fact that we are using a relatively low-parameter model of Granite, so it has quite a small context-window (the length of lemmas that the LLM can ‘remember’).

To counteract these infrequent hallucinations, we decided to add a simple safeguard - if the cosine similarity score is below a given threshold, we simply respond to the user with “Sorry, I can’t match that request to a feature”. Of course, this isn’t the best way of doing things in production, but for a small LLM model and a prototype (i.e. for our needs) it would be sufficient.

Part 1: Splicing Videos & Pushing to GitHub

Initially, we knew that we had a 5 second video recording from the user from when they registered themselves on the registration website. This video was being saved locally at the time, and the problem was that our facial recognition was in a different repository and wasn’t explicitly linked to the registration website or the server (via absolute directory paths), so there was no easy way to use the video to train the model.

Our first idea was to save the video on the server, but we realized that this would quickly take up our storage space with more users, so we decided against it. So, we decided on a simpler approach - using the 5-second video, take an image/snapshot of the video every second, so we end up with a total of 5 images to train the model with. Since it’s only 5 images, it wouldn’t take too much storage space, so we decided to go ahead with this attempt. Doing this was relatively simple using the OpenCV Python library. However, again, these images were stored in a different place to where they needed to be.

Our genius idea (NOTE: in retrospect, this was an extremely inefficient and quite simply, dumb, idea) was to save these images locally, then use a library such as PyGithub to commit and push these images to the Raspberry Pi code repository, making it easier to train the model.

After implementing this, we realized that 5 images was simply too small of a dataset to train the model on - it wasn’t extremely accurate, and didn’t always recognize the user. This led us to our next step - obtaining 20 images from the 5 second video.

Part 2: 20 Images and a More Accurate Facial Recognition Model

Using a similar approach to before, we managed to splice 20 images form the 5 second video. However, this time, we decided to take these samples at random, to ensure that no errors could occur (for example, some cameras took some extra time to boot, giving us only 3 seconds of actual video footage of the user’s face). Using a random sampling method, we reduced these errors, so, rather than having potentially 8 unusable images, we could have less.

With this, we significantly improved our accuracy - the Raspberry Pi camera almost instantly recognized a user whenever they appeared in the frame when we tested with the improved 20-image version. However, there was a major downside to this change. Namely, it took much longer to push the images to the Github repository, and also added more of a time delay while waiting for the model to be trained on the images.

We obviously didn’t want these 2 things to be the case for the final version, so we had to find another way to do this.

Part 3: Local Training & Face Encodings

Instead of going through this long-winded process with high latency, we realized that it would be easier to simply train the model on the server itself. So, we spent quite long trying to move this code to where our server code was. Eventually, this made our training much easier and quicker.

Additionally, rather than saving the images that we captured from the 5 second video, we thought that it would be better to just save the encodings of the face and the trained model. This benefitted us in 2 ways:

1. Speed: with the server doing the majority of the strenuous computation (i.e. training the model), the training time was significantly reduced since the Raspberry Pi, with less RAM, and generally lower hardware specifications than the average portable computer, wasn’t being used for this.

2. Privacy: not storing the user’s faces is a critical upgrade in the privacy of our entire system. By storing only the encodings of the faces, which are basically just a bunch of numbers that are useful for comparison, no essential personal data or images are being stored about the user.

With these encodings, the Raspberry Pi could now communicate with the server to request the necessary encodings, greatly reducing the latency of our entire system.

February Update Video

Below is a video that summarizes the above information and what we have worked on in the last month or so.

Initial Attempt

Originally, we planned to use the Raspberry Pi 5 to emulate a USB Human-Interface Device (HID) keyboard, which would allow the computer to be logged in into to recognize the Raspberry Pi as a keyboard, and receive keystrokes from it. Obviously, we had done our initial research - most Raspberry Pis have something called a “gadget mode” which allows for USB HID simulation, so we obviously thought that our Raspberry Pis could do the same.

It is important to note that, by default, Raspberry Pis are in host mode, i.e. the Pi itself enumerates and powers USB devices that are plugged into its USB-A ports. The problem, as it turned out, was that, although our model (Raspberry Pi 5) supports gadget mode, it unfortunately doesn’t support OTG mode, which allows the device to seamlessly switch between acting as a host and a gadget.

Due to the lack of this feature, we couldn’t continue to use the Raspberry Pi 5 to act as a HID keyboard - this is because we still needed the BLE and facial recognition scripts to run, which could only happen in host mode.

Second Problem

Additionally, the USB-A ports of the device didn’t support USB On-The-Go (OTG - different to OTG mode), but the USB-C port did. However, the USB-C port was how we powered the Raspberry Pi, and using an adapter on that port (in order to try to connect the device to power and to the computer to be logged in into) meant that the device didn’t actually receive enough power to support all of its functionalities.

We tried lots of things to make the Raspberry Pi work as a HID keyboard, and we did manage to register the Raspberry Pi as a keyboard eventually, but it wasn’t able to send keystrokes (due to the lack of support for USB OTG on the necessary USB ports).

Below is an image of when we got the Raspberry Pi 5 to be registered as an external USB device (keyboard):

USB OTG is what actually enables a device to act as a USB host/peripheral device, and because the Raspberry Pi 5 lacked the kind of direct OTG support we needed to emulate a USB keyboard in a fully functional manner, we weren’t able to send keystrokes as if they were coming from an actual keyboard peripheral.

Raspberry Pi Pico

Realizing that the Raspberry Pi 5 couldn’t be used as a HID Keyboard, we looked for the cheapest and best alternative, eventually finding the Raspberry Pi Pico. This is an incredibly small, microcontroller-based board that does support USB OTG, unlike the Raspberry Pi 5, and also widely supported by online communities.

In order to connect the 2 devices, we decided on a relatively simple method that is also robust: namely, UART (Universal Asynchronous Receiver/Transmitter). The setup for this was as follows:

1) Set up the hardware: this first mean connecting the Raspberry Pi Pico to a breadboard. Then, using GPIO wires, connecting the TX to the RX port on the opposite device, and finally having a wire connecting the ground pins on both devices.

2) Deciding the communication protocol: we decided to use a simple, straightforward text-based protocol over UART so that the Raspberry Pi 5 could easily tell the Raspberry Pi Pico which keystrokes to send to log the user in.

3) Setup up CircuitPython on the Raspberry Pi Pico: CircuitPython has a built-in library which allows for USB HID, letting us present the Pico to the host computer as a keyboard. Additionally, using the Busio library, we were able to setup the UART connection on both sides of the device.

Simulating a USB HID Keyboard with the Raspberry Pi Pico

Using the usb_hid library, we were easily able to define the Raspberry Pi Pico as a USB keyboard peripheral. The script for the UART and USB HID keyboard functionality looked something like the following:

import json
import board
import busio
import usb_hid
from adafruit_hid.keyboard import Keyboard
from adafruit_hid.keyboard_layout_us import KeyboardLayoutUS
from adafruit_hid.keycode import Keycode

# Initialize UART0 with baud rate 9600
uart = busio.UART(board.GP0, board.GP1 ,baudrate=9600)

# Function to send data over UART (FOR DEBUGGING ONLY)
def send_data(data):
    uart.write(data)
    print('Sent: ' + data)

# Function to receive data over UART
def receive_data():
    data = "Nothing"
    if uart.in_waiting > 0:
        data = uart.read(uart.in_waiting).decode('utf-8')
        try:
            return json.loads(data) # parse json
        except:
            print('invalid data')
    print("DATA:" + data)
    return None

keyboard = Keyboard(usb_hid.devices)
keyboard_layout = KeyboardLayoutUS(keyboard)

while True:
    response = receive_data()
    if response:
        keyboard_layout.write(username)
        keyboard.press(Keycode.ENTER)
        keyboard.release_all()
        time.sleep(2)
        # other values needed to be 'written' by the 'keyboard'
    time.sleep(1)

The code is quite simple and self-explanatory. Basically, after doing the initial UART and USB HID keyboard setup, we first check to see if the Pico has received any UART data (this happens again on each run). Then, if a valid JSON is found, the code uses the data in the JSON to decide which keystrokes need to be sent to the host computer. Finally, the Raspberry Pi Pico then emulates a USB keyboard for the computer it’s plugged into over the USB cable.

Desktop Application

Command Execution

One of our key functionalities for the desktop application was to be able to execute commands, as mentioned in the previous blog. The problem we faced was that the shell.command() function takes in only a single word (split by whitespace), whereas we were passing in the entire string. After thoroughly scouring the docs for this, we eventually realized that this is the case and we need to use the additional method .args() for additional whitespaced arguments of the bash command and, optionally, .arg() for the final argument.

The code for this ended up looking something like the following to get a function for executing commands:

match shell.command(&base_parts[0]).args(&base_parts[1..]).arg(last_value).output().await {
    Ok(output) => {
        if output.status.success() {
            let stdout_str = String::from_utf8(output.stdout).unwrap_or_default();
            println!("Command result: {:?}", stdout_str);

            if update {
                if let Err(err) = update_json_current_value(
                    &username,
                    &base_cmd_str,
                    last_value,
                    encryption_instance,
                    state,
                ).await {
                    println!("Warning: error updating JSON current value: {}", err);
                }
            }
        } else {
            println!("Exit with code: {}", output.status.code().unwrap_or_default());
        }
    }
    Err(e) => {
        println!("Failed to execute command: {} with error {}", command, e);
    }
}

Ollama & Ngrok

For the desktop application, we only managed to find accessibility commands through the terminal for GNOME-based GUIs. Therefore, we were running and testing this application on a Kali Linux VM (Virtual Machine) running on an M-series Mac. The problem with this was being able to access localhost through the VM, which wasn’t possible without changing some settings and bridging the connection (which couldn’t be changed due to some other variables).

We decided to use ollama to pull the Granite 3.0 model that we were going to use, and use the ollama-rs crate in Rust in order to access this model and use it - some of us had prior experience in doing this so we decided to go with this approach. The problem was, however, that running ollama and doing the inferencing on the virtual machine was extremely slow (due to the virtualization of the VM because of different architectures - ARM and AMD64).

So, we decided to use ngrok - a way to expose a local port to be accessed publicly. With this, ollama was now running on the Mac itself, with acceleration via the GPU on the Mac, and this was exposed on port 11434 , which was being called by the desktop application running in the VM. The problem with this was that we needed to run the server simultaneously as well, so we always needed another device using ngrok for the server in order to test the desktop application (as a premium ngrok subscription is required to expose multiple different ports over the internet).

ngrok http 11434 --host-header="localhost:11434"; # ngrok for ollama
ngrok http 8000; # ngrok for server running on port 8000

Raspberry Pi code

BLE Distance Calibration

To calculate the distance between the ESP32 and the Raspberry Pi, we used the following calculation:

$$\text{Distance}= 10^{\frac{\text{measured power} - \text{RSSI}}{10\text{N}}}$$

This requires calibrating measured power and N in order to get an accurate value. However, for an approximate distance of 25cm, our calculated distance varied between 10cm and 2.5 meters. This isn’t ideal, but we didn’t manage to recalibrate these values before today, so we will try to do this in one of the upcoming weeks.

Facial Recognition & BLE Integration

We had our facial recognition script running in a separate file from the main file with the BLE connectivity code. So, we tried to directly import the file and the methods from the facial recognition script to run it within the main file, however, there were some setup commands which clashed with each other and weren’t run. This meant that we had to rewrite the entire code so that they could all be run via methods from the imported script, and this took a while due to the variety of errors born from this.

Server/Desktop Application

Verification of the User

Since we don’t use BLE with the actual computer being logged in into, there is no explicit way to confirm that the user is who they claim to be. However, we assume that the Raspberry Pi correctly acts as a keyboard, entering the correct username and password into the computer. Therefore, in the Desktop application, instead of using BLE (as this would be some additional complexity), we simply use the whoami command to ensure that the user’s username matches what is stored in our server. This way, we ensure we do not respond with the user’s preferences JSON without it actually being them (this could happen in the case that the keyboard HID simulation fails and someone else enters their own username and password into the same computer, however this is an extremely unlikely case, as the HID keyboard simulation will almost never fail unless someone tries to disconnect the Raspberry Pi from the computer).

January Update Video

Below is a video that summarizes the above information and what we have worked on in the last month or so.

Overall Workflow

As illustrated in the diagram above, the very first step is to register the user. This is done by the ESP32 being connected to a computer, on which the registration website is opened. Then, the user can enter their credentials, and the website communicates with our server in order to store the MAC address of the ESP32 device, and relays the key back to it.

After being registered, when a user, either carrying or wearing the ESP32 device, walks up to the computer that they want to login into, the ESP32’s key will be transmitted via BLE (Bluetooth Low Energy) to the Raspberry Pi which is connected to that computer. The code on the Raspberry Pi then communicates with the server to ensure that the ESP32’s key is valid and correctly authenticated. Then, as a secondary authentication step (2-factor authentication), the camera module connected to the Raspberry Pi checks for the registered user’s face. Once this is found, the username and password of the user are entered into the device, logging the user in.

As soon as the computer loads up, the desktop application, with the AI Agent, is opened, and all saved preferences immediately load up. The user can then go on to prompt the application to make accessibility changes as necessary for them to feel comfortable using the device, and these commands are executed and the preferences are updated in the server so that they can be loaded immediately next time.

Server

Purpose

The server is where we handle all of the requests that the other components of the project make, as well as store our data. In our case, this means handling how to register an ESP32 device with a user (and their account details), as well as storing a user’s accessibility preferences which can be retrieved. This will also handle the majority of the encryption aspect, ensuring that we use a post-quantum secure encryption standard such as CRYSTALS-Kyber.

Progress

So far, we have managed to integrate the basic functionality, such as registering a user, and retrieving/updating default preference options that are stored in the database as a JSON. We didn’t spend too much time on this - it was mostly updated with any changes that were required in the other modules, as those were our main focus up to now.

Registration Website

Purpose

This website is how we register the ESP32 with a user (and their associated username and password). It scans the USB ports of the current device (the laptop on which you’ve opened up the website) and asks the user to select a device. In our demonstration video, we will explicitly go over which option needs to be selected, but any of the 2 options referring to the ESP32 device (connected via USB-C cable to the laptop) will work. The user will then be able to enter their username and password, and on clicking submit, the ESP32 should be registered to the user (the registration website sends requests to the server in order to make this happen).

Progress

As of the previous blog, most of this website was completed, but there were a few bugs, such as hydration errors. We managed to fix this after a lot of debugging to find the issue (there were some nested tags which didn’t match the format we were expecting), and we also cleaned-up the website to make it look better.

ESP32 Code

Purpose

This is the code needed on the ESP32 device which ensures that its MAC address is transmitted when necessary, and also handles the Bluetooth Low Energy communication with the Raspberry Pi as the users approaches the computer to be logged in into.

Progress

As of the previous blog, we had only managed to test the basic BLE functionality, but now, we have managed to actually communicate with the Raspberry Pi in a basic way that sends the MAC address to it from the ESP32, if the distance between the devices is sufficiently low.

Raspberry Pi Code

Purpose

This code checks for BLE signals from registered ESP32 devices and verifies their credentials, alongside constantly checking the distance from the computer. Once the distance is small enough, it uses the camera module attached to the Raspberry Pi and checks for the registered user, and if they are found, the Raspberry Pi acts as a HID (Human-Interface Device) Keyboard connected to the computer to be logged in into, and types out the user’s username and password.

Progress

So far, we have managed to get some basic functional communication with the ESP32 device via BLE. Additionally, we managed to write some code which allows us to get the encodings of a face and train a model on these encodings in order to recognize a person. The code that we’ve written so far for this facial recognition is relatively preliminary, and hasn’t really been integrated with the BLE right now - this is something we will be working on in the upcoming weeks.

Facial Recognition Functionality

It was quite tough to find a way to recognize people with the camera module, but after a lot of research, we stumbled across this blog, which helped us formulate an idea on how to go about using the camera module and recognizing people’s faces.

The idea is that, when a user is registered, we ask them for some images of themselves, which will be used to automatically train the facial recognition model, which can then be used by any Raspberry Pi to check against the database. Of course, we don’t want to store images of anyone on the database, so we will only be storing the encodings related to a person’s face i.e. numbers that are used by the model to recognize individuals.

Desktop Application

Purpose

This is the main part of our project that relates to accessibility. The application opens as soon as a user is logged in and immediately fetches their preferences from the server and executes the commands necessary to have these preferences loaded. The application also has a chatbot, which the user can talk to, and the chatbot will automatically be able to determine how to create and execute a command so that the user’s needs are met.

Progress

We have managed to create a very basic frontend design that represents the chatbot, and we have also integrated fetching the preferences from the server based on the current user’s username. We also prototyped some code for the backend which tries to execute the commands, and this mostly works. This is another one of our main focuses in the weeks going forward, as it is one of the components that will require the most work overall.

Command Execution Functionality

For executing commands, we need to be able to make sure that they are safely done - for example, if someone were to be able to somehow change the command being executed, this shouldn’t be allowed. This is quite an obvious requirement since otherwise someone could install a key logger, or attempt to delete all the data on the device, amongst other things, so obviously this safety feature is extremely important.

After some brainstorming, we figured that the best way to go about this was to ensure that the application does not have admin access, and to also whitelist the possible commands that can be run. In other words, if a command is not included within the JSON, then it cannot be run. This is fine because our JSON is stored on the server, and we know for sure that the server is secure - this is one of the initial necessities for the entire project. We also do have some other failsafes, and, if we have enough time, we might try to implement a fully functional guardrails feature (although granite does already have some safety nets).

AI Agent Functionality

For actually converting the user’s inputs into commands that can be executed, we had to try a plethora of different approaches. Eventually, the solution that seemed to work the best for us was to give the LLM a system prompt, telling it what kind of assistant it is, followed by a new user prompt for every message the user enters. The system prompt includes the entire JSON file, telling the LLM that it is supposed to search through these commands to always return a general response to the user, as well as a command to be run. This approach was quite effective, and allowed us to generally get a command that relates to what the user needs.

Client Requirements

After multiple versions of reviewing our requirements with the client, we eventually settled on the MoSCoW requirements for our project which are outlined below.

Must Have ✅

• BLE proximity connection to Raspberry Pi

  • Integration with BLE connected to Raspberry Pi for detecting user proximity to a specific device (ESP32)

• Facial recognition via Raspberry Pi (Raspberry Pi camera module)

  • Use of Raspberry Pi for real-time facial recognition, to authenticate users and link their profiles to devices

  • Ensure accuracy in low light conditions and secure handling of biometric data

• Accessibility settings fetching

  • Automatically fetch user-specific accessibility settings when the user connects to a device e.g text size, contrast, voice support

• Server

  • For hosting user profiles, managing proximity data and backend operations

• Post-quantum cryptography encryption for securing user data

  • All data transmitted between devices should be encrypted using a PQC algorithm

  • Secures sensitive user data during transmission and storage

• AI chatbot for easy settings configurations

  • Granite 3.0 IBM Model

  • Enable chatbot to help with troubleshooting/guidance

Should Have ❗

• A website to configure settings initially

  • Register new user profiles on website (facial recognition setup, preferences, etc.)

  • Save data to server

  • Secure login - 2 factor authentication

  • Responsive design for computer/tablet use

• Directional proximity management

  • To identify a specific device a user intends to interact with when there are multiple devices nearby

Could Have ⚠️

• API Layer to make system reproducible

• Multiple languages support

  • For the configuration website and IoT interactions

Won’t Have ❌

• Mobile application to remotely login and select device

• Voice recognition/authentication

  • Potential privacy concerns and complexity would be an issue

System Design

We also created a system design diagram that briefly illustrates how the entire system should work.

Diagram

Workflow

Below is a simple representation of the general flow of events that will happen when the system is run and set up:

ESP32 (MAC Address) → Registered → ESP32 gets back its key → Broadcasts key → Raspberry Pi listens in → Raspberry Pi relays key to server → Credentials are relayed back to the Raspberry Pi → Raspberry Pi performs facial recognition with the camera module → Raspberry Pi simulates HDI inputs → AI Agent boots up on login → Preferences loaded

Initial Work on MVP

We finally received the last of the hardware that this project will require today, so we were unable to work on the actual code related to the BLE connection between the two devices before now. However, we instead worked on other parts of the MVP that didn't explicitly need the devices.

To start with, we spent the first few weeks of this term conducting as much research as we could to ensure that the project is actually feasible to be completed within our limited time frame. This included looking into Granite 3.0, since we need to use that for our AI Agent which helps users set their accessibility settings; BLE protocols (amongst other protocols to figure out which one would be the best for our use case); and GNOME settings commands to figure out how to automatically set the user's preferences on login for the Rocky Operating System.

Building on our research, we then started working on the device and user registration website (shown below), and, since receiving the Raspberry Pis as of last week, we have managed to achieve basic BLE connectivity on them.

December Update Video

Below is a video that summarizes the above information and what we have worked on in the last month or so.