the graphical, universal Log processor for incident response!

made with ❤️ by Mentat.

Gulp is a powerful software tool designed to streamline incident response and analysis. Its core features includes:

• Data Ingestion Plugins: Gulp can ingest data from a variety of sources, thanks to its versatile plugin system.

• OpenSearch and ECS: Gulp is built on OpenSearch and uses the Elasticsearch Common Scheme (ECS) as its ingestion format, ensuring compatibility and ease of use.

• High-Speed Multiprocessing Engine: Gulp's engine is designed for speed, offering fast ingestion and querying capabilities through multiprocessing.

• Query using SIGMA rules: Gulp supports querying using Sigma Rules, allowing for easy, one-click queries with thousands of rules in parallel.

• Collaboration Platform: Gulp includes a collaboration platform, enabling teams to work together on the same incident. Features include note-taking, highlighting, and link adding.

• An innovative UI: Gulp's user interface includes multiple on-screen per-context(i.e. a log source) zoomable timelines for visualizing events, making it easier to understand and analyze incidents.

  

  

  

• Scalable: Gulp is designed with scalability in mind. As your data and team grow, you can simply add more gulp nodes, more cores to increase parallel ingestion and query capabilities, more OpenSearch and PostgreSQL nodes. This makes Gulp a flexible solution that can adapt to your evolving needs!

• Python based: Gulp is written in Python, leveraging open-source libraries whenever possible. This maximizes ease of adoption from the community, as Python is widely used and understood.

here is a detailed datasheet

• GULP architecture

• plugins and mapping
• testing guidelines

gulp can be of course integrated with other applications !

both websocket and REST API is available!

• docker
• install from sources/dev setup
• installing extra plugins

gulp web ui

the following environment variables may be set to override configuration options.

• GULP_BIND_TO_ADDR, GULP_BIND_TO_PORT : if set, gulp will listen to this interface and port (either, the default 0.0.0.0, 8080 is used).

  • for the override to work, both GULP_BIND_TO_ADDR and GULP_BIND_TO_PORT must be specified, either the value of one alone is ignored.

• GULP_WORKING_DIR: this is the working directory for gulp (defaults to ~/.config/gulp), which contains:

  • gulp_cfg.json: the configuration, initialized with template if not present
  • plugins: optional extra plugins (have precedence over $INSTALLDIR/plugins)
  • mapping_files: optional extra mapping files (have precedence over $INSTALLDIR/mapping_files)
  • certs: optional SSL certificates for HTTPS
  • ingest_local directory to store big files for quick ingestion (ingest_local API)
  • tmp_upload folder to cache partial uploads during ingestion

• GULP_OPENSEARCH_URL: if set, overrides opensearch_url in the configuration to.

• GULP_POSTGRES_URL: if set, overrides postgres_url in the configuration.

• GULP_REDIS_URL: if set, overrides redis_url in the configuration.

• GULP_INTEGRATION_TEST: TEST ONLY, this must be set to 1 during integration testing (i.e. client api) to disable debug features which may interfere.

using the default docker-compose.yml

• gulp swagger page on http://localhost:8080/docs
• gulp web UI on http://localhost:3000
  • user/pwd: admin/admin (default gulp admin user)

• postgreSQL on localhost:5432

  • user/pwd: postgres/Gulp1234!

• adminer on http://localhost:8001 to manage postgreSQL (use --profile dev)

  • server/user/pwd: postgres/postgres/Gulp1234!

• opensearch on http://localhost:9200

  • user/pwd: admin/Gulp1234!

• elasticvue on http://localhost:8082 to visualize OpensSearch indexes (use --profile dev)

• opensearch dashboards on http://localhost:5001 for a more comprehensive OpenSearch management.

• redis on localhost:6379

  • user/pwd: default/Gulp1234!

• redis insight on http://localhost:5540 to manage redis (use --profile dev)

• vsftpd on port 21(ftp) or 21000 (sftp) to manage files in $GULP_WORKING_DIR
  • user/pwd: gulp/Gulp1234!

to use SSL, the following configuration options and files variables may be provided:

• Gulp configuration
  • opensearch_verify_certs: set to false to skip server verification
• certificate files
  • $GULP_WORKING_DIR/certs/opensearch-ca.pem: CA certificate for Gulp to connect to the Opensearch server
  • $GULP_WORKING_DIR/certs/opensearch.pem: client certificate for Gulp to connect to the Opensearch server
  • $GULP_WORKING_DIR/certs/opensearch.key: passwordless client certificate key

• Gulp configuration
  • postgres_ssl: use SSL for postgres connection, set to false to not use.
  • postgres_verify_certs: set to false to skip server verification
• certificate files
  • $GULP_WORKING_DIR/certs/postgres-ca.pem: CA certificate for Gulp to connect to the PostgreSQL server
  • $GULP_WORKING_DIR/certs/postgres.pem: client certificate for Gulp to connect to PostgreSQL server
  • $GULP_WORKING_DIR/certs/postgres.key: client certificate key

• Gulp configuration
  • https_enforce: set to true to enforce connection to Gulp only through HTTPS
  • https_enforce_client_certs: set to true to enforce check of client certificates signed by gulp-ca.pem CA
• certificate files
  • $GULP_WORKING_DIR/certs/gulp-ca.pem: Gulp CA
  • $GULP_WORKING_DIR/certs/gulp.pem: Gulp server certificate
  • $GULP_WORKING_DIR/certs/gulp.key: Gulp server certificate key

• certificate files
  • $GULP_WORKING_DIR/certs/sftpd.pem: server certificate + CA
  • $GULP_WORKING_DIR/certs/sftpd.key: server certificate key

default startup, creates collab database with an operation names test_operation on the very first run.

gulp

to detect if gulp has already run once check for ~/.config/gulp/.first_run_done and delete it to revert to first run on the next run.

deletes data related to ALL existing operations, both on collaboration database and OpenSearch.

gulp --reset-collab

deletes data related to ALL existing operations, both on collaboration database and OpenSearch, in the end creates/recreates my_operation.

gulp --reset-collab --create my_operation

acts only on my_operation: creates/recreates operation, deletes all related data both on collaboration database and OpenSearch.

gulp --create my_operation