A data pipeline to ingest, transform and load data into data warehouse with Airflow, influxDB and MongoDB.
Install requirements using pip.
pip install -r requirements.txt
Initialize Airflow DB.
airflow initdb
Create an admin user to access airflow
airflow users create \
--username USER_NAME \
--firstname FIRST_NAME \
--lastname LAST_NAME \
--role Admin \
--email [email protected] \
--password PASSWORD
MongoDB
Install MongoDB community server or use the docker image.
docker pull mongo
influxDB
docker pull influxdb
Configure as per documentation on dockerhub.
Run airflow server on port 8080 or other available port.
airflow webserver --port 8080
Run airflow scheduler.
airflow scheduler
Browse Airflow UI
localhost:8080
Note: If you don't see the DAGs in the Aiflow UI, then make sure that in airflow.cfg the dags path dags_folder = /Users/project/path/dags is right.