NEAT-AMBIENCE Project

In the NEAT-AMBIENCE project, we aim to develop innovative data management techniques to assist citizens in their daily lives. Specifically, with TrafficDator (TRAFFIC Data Analysis and predicTiOn for betteR mobility), our goal is to create methodologies and strategies for short- and long-term traffic prediction by integrating diverse data sources.

Project

"Next-gEnerATion dAta Management to foster suitable Behaviors and the resilience of cItizens against modErN ChallEnges (NEAT-AMBIENCE)", funded by the Spanish State Research Agency (PID2020-113037RB-I00 / AEI / 10.13039/501100011033).
Principal Investigator: Sergio Ilarri.

Project Structure

The TrafficDator project is structured into several key folders to organize data processing, model training, evaluation, and deployment efficiently. Below is a detailed breakdown:

1. full_processed_data

Contains preprocessed datasets ready for use in model training and evaluation. Each folder corresponds to a unique data_hash to differentiate experiments conducted with different datasets:

• MDT_complete_data.csv: The complete dataset after preprocessing.
• MDT_target_month.csv: Data corresponding to the month used for predictions.
• MDT_training.csv: Dataset prepared specifically for training purposes.

2. metrics

Includes evaluation metrics for both adapted and custom models. Each directory uses a data_hash for reproducibility:

• adapted_models: Results for models like LSTM, DCRNN, or STGODE.
  • Organized by sequence lengths and horizons.
  • Files include horizon-specific metrics.
• our_models: Evaluation metrics for custom models like TrafficDatorNet, KNN, and Gradient Boosting.

3. processed_data

Folder containing the processed data used for prediction and evaluation tasks. All subdirectories also include a data_hash:

predictions/

Subfolders organized by model (e.g., dcrnn, lstm, stgode, PatchSTG, DynAGS) storing prediction results generated by the models.

• dcrnn: Prediction results for DCRNN.
• lstm: Prediction results for LSTM.
• stgode: Prediction results for STGODE.
• patchstg: Prediction results for PatchSTG.
• agcrn: Prediction results for DynAGS.

temporal_sequences/

Folder containing preprocessed temporal sequence data, organized by sequence length and prediction horizon. All subdirectories also include a data_hash:

• seq_len12_horizon12: Data for a sequence length of 12 steps and prediction horizon of 12.

Generic Scripts (Outside Subfolders)/**

• MDT_adj_matrix.npy: Adjacency matrix used for spatio-temporal models.
• MDT_complete_data.csv: Complete dataset used for processing.
• MDT_id_longitude_latitude.csv: File with geographical coordinates (latitude and longitude) for each data point.
• MDT_target_month.csv: Target prediction values for a specific month.
• MDT_training.csv: Dataset used for model training.

4. raw_data

Contains the raw datasets before preprocessing, divided into the following categories. Each subfolder also tracks a data_hash for dataset integrity:

• meteorology_data: Meteorological datasets (e.g., meteo_2022.csv, meteo_2023.csv).
• traffic_data: Monthly traffic data split by year and month (e.g., 01_2023.csv, 02_2024.csv).
• work_calendar: Work calendar data (e.g., calendar.csv).

5. scripts

This folder contains key scripts organized for model experiments, data processing, and utility functions. Each script operates with datasets identified using data_hash. Below is the detailed folder structure:

adapted_models/

Contains scripts for adapted models, including all experimental configurations and utilities:

• experiments/: Organized by model type, this subfolder contains scripts for configuring and running experiments:

  • dcrnn/: Experiment scripts specific to the DCRNN model.
  • lstm/: Experiment scripts specific to the LSTM model.
  • stgode/: Experiment scripts specific to the STGODE model.
  • patchstg/: Experiment scripts specific to the PatchSTG model.
  • agcrn/: Experiment scripts specific to the AGCRN (DynAGS) model.

• src/: Contains the core codebase for adapted models:

  • base/, engines/, models/,utils/: Houses implementation files for foundational classes, training engines, and model architectures.
  • utils/: Includes general-purpose utility scripts:
    • generate_data.py: Scripts for generating model input data.
    • metrics.py: Code for calculating custom metrics used in evaluation.
    • predictions.py: Functions for processing and visualizing model predictions.

our_models/

Contains scripts and utilities for the custom models developed specifically for the TrafficDator project:

• trained_models/: Stores pre-trained custom models ready for evaluation or deployment.
• utils/: Contains auxiliary files like:
  • label_encoder.pkl: Pre-saved label encoder for categorical data processing.
  • preprocessor.pkl: Pre-saved preprocessing configurations for data normalization and scaling.
  • preprocessor_TrafficDatorNet.pkl: Pre-saved preprocessor specific to TrafficDatorNet.
• Generic Scripts (Outside Subfolders)/:
  • models_builder.py: Code for building and configuring custom models.
  • metrics.py: Scripts for evaluating prediction performance across different temporal horizons.

Generic Scripts (Outside Subfolders)

The following scripts are located directly in the root of the scripts/ folder and are used for generic tasks:

• adjacency_matrix.py: Handles the generation of adjacency matrices for graph-based models.
• extract_id_longitude_latitude.py: Extracts IDs, longitude, and latitude information from raw datasets.
• file_sizes.py: Computes and logs the size of various files in the dataset.
• MTD_generator.py: Generates specific traffic-related datasets for experiments.
• requirements.py: Contains the list of Python dependencies required for the project.
• traffic_intensity_statistics.py: Calculates and analyzes traffic intensity statistics.
• plot_sensor_predictions.py: Runs inference for a selected model and dataset, ranks sensors by R², and saves predicted vs. actual plots per sensor under metrics/our_models/<model>/<data_hash>/plots.

6. shared_resources

Contains shared resources, such as logs and auxiliary statistics. Includes data_hash for ensuring data consistency:

• Example: training_times.txt, which records training times for models.

Funding

This work has been developed as part of the R&D project PID2020-113037RB-I00, funded by MCIN/AEI/ 10.13039/501100011033.
Additionally, the support of the Department of Science, University, and Society of Knowledge of the Government of Aragon (Government of Aragon: COSMOS group; last group reference: T64_23R) is also appreciated.

Websites

• TrafficDator
• NEAT-AMBIENCE

Principal Researchers

• Iván Gómez — Department of Computer Science and Systems Engineering, Universidad de Zaragoza, 50018 Zaragoza, Spain.
• Sergio Ilarri — Department of Computer Science and Systems Engineering, Universsidad de Zaragoza, 50018 Zaragoza, Spain; Aragón Institute of Engineering Research (I3A), Universidad de Zaragoza, 50018 Zaragoza, Spain.

Logos