Init #2

mmacata · 2025-06-02T11:59:36Z

This PR moves some modules and methods from actinia-core to this new library. These modules and methods are not only used by actinia-core but also by different plugins.

See actinia-org/actinia-core#627 for more details.

Co-authored-by: anikaweinmann <[email protected]>

* reactivate redis queue * rename variable * make queue type configurable * enhance dev setup for redis queue * lint * move user_auth * move user auth * move base_login * move map_layer_base * move renderer_base * move resource_base * fix relative imports * lint * first splitup: raster_colors * splitup download_cache_management * splitup ephemeral_custom_processing * splitup ephemeral_processing_with_export * splitup ephemeral_processing * splitup renderer_base * splitup persistent_processing * splitup location_management * splitup map_layer_management * splitup mapset_management * splitup persistent_mapset_merger * splitup process_validation * splitup raster_export * splitup raster_layer * splitup raster_legend * lint * splitup raster_renderer * splitup resource_storage_management * splitup strds_management * splitup strds_raster_management * splitup strds_renderer * splitup vector_layer * splitup vector_renderer * lint * make inheritance more clear * add readme * fix import

* tests: fix test descriptions This PR fixes some test descriptions and a few typos.

* linting tests + black workflow * linting * black * further black * fix unittests * fix test Co-authored-by: anikaweinmann <[email protected]>

* add job queue name to response * lint * Update src/actinia_core/models/response_models.py Co-authored-by: Markus Neteler <[email protected]> * black * undo-black Co-authored-by: Markus Neteler <[email protected]>

…odule plugin (#387) * change pc style for module description for importer and exporter in module plugin * black

adjustment of the interim results Co-authored-by: anikaweinmann <[email protected]> Co-authored-by: Carmen Tawalika <[email protected]>

* Add possibility to configure pattern to save addional mapsets in the interim resutls Co-authored-by: anikaweinmann <[email protected]>

…s (#410) * add raster VRT support and support for mapset names by interim results Co-authored-by: anikaweinmann <[email protected]>, linakrisztian <[email protected]>

* make resource resource delition asynchron * trailing and ending precommit ... * do not use pylint * trigger linting * linting * linting --------- Co-authored-by: anikaweinmann <[email protected]>

* allow separate config for worker * test tests * fix test * create download cache if not exists * create download cache if not exists * create resource storage if not exists * create log folder * further work on tests * add TODOs in tests

* started implementation for querying pixellimit for r.import commands * continued implementation of pixel limit check for rimport commands * considered resampling/reprojection for raster size + cleanup of created vrt * linting * markdown formating * add tests for pixellimit check * improved check in test + added fix for determining the estimated resolution of raster * Update tests/test_raster_import_pixellimit.py Co-authored-by: Markus Neteler <[email protected]> * Update tests/test_raster_import_pixellimit.py Co-authored-by: Markus Neteler <[email protected]> * Update tests/test_raster_import_pixellimit.py Co-authored-by: Carmen Tawalika <[email protected]> * Update tests/test_raster_import_pixellimit.py Co-authored-by: Carmen Tawalika <[email protected]> * Update src/actinia_core/processing/actinia_processing/ephemeral_processing.py Co-authored-by: Carmen Tawalika <[email protected]> * add test for importer --------- Co-authored-by: Markus Neteler <[email protected]> Co-authored-by: Carmen Tawalika <[email protected]>

* add stdin for parameters * add filter to stdout for other modules * add tests --------- Co-authored-by: anikaweinmann <[email protected]>

* lint * update lint workflow * update to new black version * mdformat

* add decorator for deprecated locations * add project endpoints * endpoints for projects * rename location * rename location * unify maintainer * fix GRASS Initialisation * black * fixes for tests and G83 * fixes * add rest of changes * Update tests/test_job_resumption.py * fix endpoints and job resumption tests * Update src/actinia_core/core/common/api_logger.py * Tests for G84 * Test pipelines for G8.3 and G8.4 * Update src/actinia_core/rest/base/resource_base.py * fix test workflow * fix tests for G83 * fix error due to lib update * fix proc name * Update src/actinia_core/endpoints.py Co-authored-by: Carmen Tawalika <[email protected]> --------- Co-authored-by: Carmen Tawalika <[email protected]> Co-authored-by: Carmen <[email protected]>

* rename location * fixes for tests and G83 * fix job resumption tests * linting * initial ruff check * update test imports * lint * F841 and Post-PR * lint * test with wip linting workflow * test post-pr * fix ref * trigger build * always-post-pr * post-pr in separate workflow * rename workflow * fix lint error * trigger pipelines * add ruff config * update post-pr workflow * Update .github/workflows/linting.yml * fix tests --------- Co-authored-by: anikaweinmann <[email protected]>

* update setup from redis to valkey * replace redis with valkey calls + remove non needed redis-remainings * replace redis import with valkey import * adjust function from redis to valkey * replace redis by valkey function * fix call of valkey commands fitting valkey-syntax * Semantic renaming: redis to kvdb (#602) * semantic renaming: redis to kvdb * remove non semantic changes * remove non semantic changes * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * remove non semantic changes * update readme * Update src/actinia_core/README.md Co-authored-by: Carmen Tawalika <[email protected]> * Update src/actinia_core/core/common/config.py Co-authored-by: Carmen Tawalika <[email protected]> * Update src/actinia_core/README.md Co-authored-by: Carmen Tawalika <[email protected]> * Update docker/actinia-core-alpine/actinia.cfg Co-authored-by: Carmen Tawalika <[email protected]> * PR review CT --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Carmen Tawalika <[email protected]> * linting: black --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Carmen Tawalika <[email protected]>

* reactivate redis queue * rename variable * make queue type configurable * enhance dev setup for redis queue * lint * move user_auth * move user auth * move base_login * move map_layer_base * move renderer_base * move resource_base * fix relative imports * lint * first splitup: raster_colors * splitup download_cache_management * splitup ephemeral_custom_processing * splitup ephemeral_processing_with_export * splitup ephemeral_processing * splitup renderer_base * splitup persistent_processing * splitup location_management * splitup map_layer_management * splitup mapset_management * splitup persistent_mapset_merger * splitup process_validation * splitup raster_export * splitup raster_layer * splitup raster_legend * lint * splitup raster_renderer * splitup resource_storage_management * splitup strds_management * splitup strds_raster_management * splitup strds_renderer * splitup vector_layer * splitup vector_renderer * lint * make inheritance more clear * add readme * fix import

* update setup from redis to valkey * replace redis with valkey calls + remove non needed redis-remainings * replace redis import with valkey import * adjust function from redis to valkey * replace redis by valkey function * fix call of valkey commands fitting valkey-syntax * Semantic renaming: redis to kvdb (#602) * semantic renaming: redis to kvdb * remove non semantic changes * remove non semantic changes * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * remove non semantic changes * update readme * Update src/actinia_core/README.md Co-authored-by: Carmen Tawalika <[email protected]> * Update src/actinia_core/core/common/config.py Co-authored-by: Carmen Tawalika <[email protected]> * Update src/actinia_core/README.md Co-authored-by: Carmen Tawalika <[email protected]> * Update docker/actinia-core-alpine/actinia.cfg Co-authored-by: Carmen Tawalika <[email protected]> * PR review CT --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Carmen Tawalika <[email protected]> * linting: black --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Carmen Tawalika <[email protected]>

* reactivate redis queue * rename variable * make queue type configurable * enhance dev setup for redis queue * lint * move user_auth * move user auth * move base_login * move map_layer_base * move renderer_base * move resource_base * fix relative imports * lint * first splitup: raster_colors * splitup download_cache_management * splitup ephemeral_custom_processing * splitup ephemeral_processing_with_export * splitup ephemeral_processing * splitup renderer_base * splitup persistent_processing * splitup location_management * splitup map_layer_management * splitup mapset_management * splitup persistent_mapset_merger * splitup process_validation * splitup raster_export * splitup raster_layer * splitup raster_legend * lint * splitup raster_renderer * splitup resource_storage_management * splitup strds_management * splitup strds_raster_management * splitup strds_renderer * splitup vector_layer * splitup vector_renderer * lint * make inheritance more clear * add readme * fix import

* update setup from redis to valkey * replace redis with valkey calls + remove non needed redis-remainings * replace redis import with valkey import * adjust function from redis to valkey * replace redis by valkey function * fix call of valkey commands fitting valkey-syntax * Semantic renaming: redis to kvdb (#602) * semantic renaming: redis to kvdb * remove non semantic changes * remove non semantic changes * Apply suggestions from code review Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * remove non semantic changes * update readme * Update src/actinia_core/README.md Co-authored-by: Carmen Tawalika <[email protected]> * Update src/actinia_core/core/common/config.py Co-authored-by: Carmen Tawalika <[email protected]> * Update src/actinia_core/README.md Co-authored-by: Carmen Tawalika <[email protected]> * Update docker/actinia-core-alpine/actinia.cfg Co-authored-by: Carmen Tawalika <[email protected]> * PR review CT --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Carmen Tawalika <[email protected]> * linting: black --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Carmen Tawalika <[email protected]>

* move modules used by plugins * move shared modules to common folder

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

ruff

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 191 in 30f827c

self.orig_mapset_path, os.R_OK | os.X_OK | os.W_OK

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 196 to 197 in 30f827c

    
           "Mapset <%s> exists in the global " 
        
           "dataset and can not be modified." % mapset

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 201 in 30f827c

"Unable to access global project <%s>" % self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 213 in 30f827c

self.user_project_path, mapset

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 219 in 30f827c

self.orig_mapset_path, os.R_OK | os.X_OK | os.W_OK

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 229 to 230 in 30f827c

    
           "Unable to access mapset <%s> " 
        
           "path %s" % (mapset, self.orig_mapset_path)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 236 in 30f827c

"Unable to access user project <%s>" % self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 241 to 242 in 30f827c

    
           def _check_target_mapset_exists(self): 
        
               """Check if the target mapset exists

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 255 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 258 to 261 in 30f827c

    
               def _check_lock_target_mapset(self): 
        
                   """ 
        
                   Check if the target mapset exists and lock it, then lock the temporary 
        
                   mapset

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 280 to 281 in 30f827c

    
           def _lock_target_mapset(self): 
        
               """Lock the target mapset

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 287 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 296 to 298 in 30f827c

    
           "Unable to lock project/mapset <%s/%s>, " 
        
           "resource is already locked" 
        
           % (self.project_name, self.target_mapset_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 301 to 302 in 30f827c

    
           "project/mapset <%s/%s> locked" 
        
           % (self.project_name, self.target_mapset_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 308 to 309 in 30f827c

    
           def _merge_mapset_into_target(self, source_mapset, target_mapset): 
        
               """Link the source mapset content into the target mapset

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 315 to 316 in 30f827c

    
           "Copy source mapset <%s> content " 
        
           "into the target mapset <%s>" % (source_mapset, target_mapset)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 338 in 30f827c

self.user_project_path, source_mapset, directory

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 359 to 360 in 30f827c

    
           "%s" % source_path, 
        
           "%s/." % target_path,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 369 in 30f827c

" stdout: %s stderr: %s" % (stdout_buff, stderr_buff)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 372 to 373 in 30f827c

    
           def _copy_merge_tmp_mapset_to_target_mapset(self): 
        
               """Copy the temporary mapset into the original project

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 379 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 383 in 30f827c

resource_id=self.target_mapset_lock_id, expiration=3600

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 388 in 30f827c

"<%s>" % self.target_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 393 in 30f827c

resource_id=self.temp_mapset_lock_id, expiration=3600

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 398 in 30f827c

"temporary mapset <%s>" % self.temp_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 402 to 406 in 30f827c

    
           "Copy temporary mapset from %s to %s" 
        
           % ( 
        
               self.temp_mapset_path, 
        
               os.path.join(self.user_project_path, self.target_mapset_name), 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 417 to 418 in 30f827c

    
           "Copy temporary mapset <%s> to target project " 
        
           "<%s>" % (self.temp_mapset_name, self.project_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 422 in 30f827c

self.user_project_path, self.target_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 425 to 426 in 30f827c

    
           "Copy temporary mapset <%s> to target project " 
        
           "<%s>" % (self.target_mapset_name, self.project_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 435 in 30f827c

["/bin/cp", "-fr", "%s" % source_path, "%s" % target_path],

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 445 in 30f827c

% (stdout_buff, stderr_buff, p.returncode)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 450 in 30f827c

"original project. Exception %s" % str(e)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 457 in 30f827c

self.temp_mapset_name, self.target_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 460 in 30f827c

os.path.join(self.user_project_path, self.temp_mapset_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 469 in 30f827c

"Remove interim results %s" % interim_dir

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 474 to 475 in 30f827c

    
           def _execute_process_list(self, process_list): 
        
               """Extend the mapset lock and execute the provided process list

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 483 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 493 to 494 in 30f827c

    
           "Unable to extend lock for mapset <%s>" 
        
           % self.target_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 506 in 30f827c

"temporary mapset <%s>" % self.temp_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 516 to 517 in 30f827c

    
           def _execute(self, skip_permission_check=False): 
        
               """Overwrite this function in subclasses

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 530 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 551 in 30f827c

pc_step, self.rdc.iteration - 1

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 585 in 30f827c

source_mapset_name=self.target_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 597 to 600 in 30f827c

    
               def _final_cleanup(self): 
        
                   """ 
        
                   Final cleanup called in the run function at the very end of processing 
        
                   """

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1 in 30f827c

# -*- coding: utf-8 -*-

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 24 to 26 in 30f827c

    
           """ 
        
           Base class for asynchronous processing 
        
           """

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 28 in 30f827c

import math

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 31 in 30f827c

import requests

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 39 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 40 in 30f827c

from flask import json

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 43 to 53 in 30f827c

    
           from actinia_core.core.common.config import global_config, DEFAULT_CONFIG_PATH 
        
           from actinia_core.core.common.process_object import Process 
        
           from actinia_core.core.grass_init import GrassInitializer 
        
           from actinia_core.core.messages_logger import MessageLogger 
        
           from actinia_core.core.kvdb_lock import KvdbLockingInterface 
        
           from actinia_core.core.resources_logger import ResourceLogger 
        
           from actinia_core.core.mapset_merge_utils import change_mapsetname 
        
           from actinia_core.core.common.process_chain import ( 
        
               get_param_stdin_part, 
        
               ProcessChainConverter, 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 57 in 30f827c

RsyncError,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 59 to 62 in 30f827c

    
           from actinia_core.core.common.exceptions import AsyncProcessTimeLimit 
        
           from actinia_core.models.response_models import ( 
        
               ProcessingResponseModel, 
        
               ExceptionTracebackModel,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 64 in 30f827c

from actinia_core.models.response_models import (

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 65 in 30f827c

create_response_from_model,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 68 in 30f827c

)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 69 in 30f827c

from actinia_core.core.interim_results import InterimResult, get_directory_size

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 82 in 30f827c

class EphemeralProcessing(object):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 129 to 130 in 30f827c

    
           def __init__(self, rdc): 
        
               """Constructor

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 150 in 30f827c

DEFAULT_CONFIG_PATH

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 297 to 298 in 30f827c

    
           def _send_resource_update(self, message, results=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 325 to 326 in 30f827c

    
           def _send_resource_finished(self, message, results=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 355 to 356 in 30f827c

    
           def _send_resource_terminated(self, message, results=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 384 to 385 in 30f827c

    
           def _send_resource_time_limit_exceeded(self, message, results=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 413 to 414 in 30f827c

    
           def _send_resource_error(self, message, results=None, exception=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 443 to 444 in 30f827c

    
           def _send_to_database(self, document, final=False): 
        
               """Send the document to the database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 458 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 482 to 484 in 30f827c

    
           self.message_logger.error( 
        
               "Unable to send webhook request. Traceback: %s" 
        
               % str(run_state)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 487 in 30f827c

def _post_to_webhook(self, document, type):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 496 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 498 to 499 in 30f827c

    
           "Send POST request to %s webhook url: %s" 
        
           % (type, self.webhook_finished)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 535 in 30f827c

if not (500 <= resp.status_code and resp.status_code < 600):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 540 in 30f827c

webhook_not_reached is False and resp.status_code not in [200, 204]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 543 in 30f827c

"Unable to access %s webhook URL %s" % (type, webhook_url)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 548 in 30f827c

process chain where to continue

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 554 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 561 in 30f827c

self.user_id, self.resource_id

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 565 in 30f827c

self.user_id, self.resource_id, iter

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 585 to 586 in 30f827c

    
                   """ 
        
                   Create the process list and check for user permissions.

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 610 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 612 in 30f827c

if old_process_chain is not None:

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 614 in 30f827c

old_process_chain

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 623 in 30f827c

self.request_data

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 630 in 30f827c

process_chain

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 666 in 30f827c

% self.process_num_limit

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 676 in 30f827c

if process.exec_type == "grass" or process.exec_type == "exec":

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 686 to 687 in 30f827c

    
           "Module or executable <%s> is not supported" 
        
           % process.executable

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 691 to 697 in 30f827c

    
           "Wrong process description, type: %s " 
        
           "module/executable: %s, args: %s" 
        
           % ( 
        
               str(process.exec_type), 
        
               str(process.executable), 
        
               str(process.executable_params), 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 706 in 30f827c

def _setup(self, init_grass=True):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 708 in 30f827c

temporary grass database and temporary file directories

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 741 to 742 in 30f827c

    
           else: 
        
               self.setup_flag = True

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 763 in 30f827c

**kwargs, fluent_sender=fluent_sender

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 776 in 30f827c

self.user_credentials["permissions"]["process_time_limit"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 796 to 797 in 30f827c

    
           def _setup_paths(self): 
        
               """Helper method to setup the paths"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 799 in 30f827c

self.user_credentials["permissions"]["cell_limit"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 802 in 30f827c

self.user_credentials["permissions"]["process_num_limit"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 806 in 30f827c

self.grass_temp_database, self.temp_grass_data_base_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 812 in 30f827c

self.temp_grass_data_base, self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 815 in 30f827c

self.grass_data_base, self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 823 in 30f827c

self.grass_user_data_base, self.user_group

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 830 in 30f827c

self.grass_user_data_base, self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 843 in 30f827c

def _create_temp_database(self, mapsets=None):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 906 in 30f827c

f"<{self.project_name}>"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 924 in 30f827c

", Exception: %s" % str(e)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 928 in 30f827c

"""Helper method to link projects mapsets

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 939 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 951 to 954 in 30f827c

    
           left_over_mapsets = [] 
        
           for mapset in mapsets: 
        
               if mapset not in mapsets_to_link: 
        
                   left_over_mapsets.append(mapset)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 991 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 998 in 30f827c

mapset_path, os.R_OK & os.X_OK

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1023 to 1024 in 30f827c

    
           "Invalid mapset <%s> in project <%s>" 
        
           % (mapset, self.project_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1029 in 30f827c

"Unable to access global project <%s>" % self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1032 in 30f827c

msg = "Unable to access user project <%s>" % self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1037 to 1039 in 30f827c

    
               self, grass_data_base, mapset_name="PERMANENT" 
        
           ): 
        
               """Sets up the GRASS environment to run modules

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1051 to 1052 in 30f827c

    
           "Initlialize GRASS grass_data_base: %s; project: %s; mapset: %s" 
        
           % (grass_data_base, self.project_name, mapset_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1073 to 1074 in 30f827c

    
           ): 
        
               """Create the temporary mapset and switch into it

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1109 in 30f827c

self.temp_project_path, temp_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1116 in 30f827c

"Rsync interim result mapset to temporary GRASS DB"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1127 in 30f827c

interim_result_mapset, self.temp_mapset_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1132 in 30f827c

"mapset"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1135 in 30f827c

os.path.dirname(self.temp_mapset_path)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1139 in 30f827c

"Rsync interim result file path to temporary GRASS DB"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1142 in 30f827c

interim_result_file_path, self.temp_file_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1147 in 30f827c

"new temporare file path"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1151 in 30f827c

"g.mapset", ["-c", "mapset=%s" % temp_mapset_name]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1159 in 30f827c

"mapset=%s" % (",".join(self.required_mapsets)),

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1165 in 30f827c

"search path: " + ",".join(self.required_mapsets)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1185 in 30f827c

self.temp_project_path, source_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1193 in 30f827c

def _cleanup(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1213 in 30f827c

f"Temporary file {tmpfile} can't be removed: {e}"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1216 in 30f827c

def _check_pixellimit_rimport(self, process_executable_params):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1223 to 1228 in 30f827c

    
           rimport_inp = [x for x in process_executable_params if "input=" in x][ 
        
               0 
        
           ].split("=")[1] 
        
           rimport_out = [x for x in process_executable_params if "output=" in x][ 
        
               0 
        
           ].split("=")[1]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1243 in 30f827c

"g.region", ["-ug"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1247 in 30f827c

"Unable to check the computational region size"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1257 to 1258 in 30f827c

    
           gdabuildvrt_params.append(vrt_out) 
        
           gdabuildvrt_params.append(rimport_inp)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1269 in 30f827c

"/usr/bin/gdalinfo", gdalinfo_params

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1289 in 30f827c

"r.import", [vrt_out, "-e"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1309 in 30f827c

[

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1313 to 1314 in 30f827c

    
               ][0].split("=")[1] 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1327 to 1329 in 30f827c

    
           [x for x in stdout_gregion.split("\n") if "nsres=" in x][ 
        
               0 
        
           ].split("=")[1]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1332 to 1334 in 30f827c

    
           [x for x in stdout_gregion.split("\n") if "ewres=" in x][ 
        
               0 
        
           ].split("=")[1]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1337 to 1346 in 30f827c

    
           if res_val: 
        
               if (res_val[0] < res_est) | (res_val[1] < res_est): 
        
                   # only check if smaller resolution set 
        
                   res_change_x = res_est / res_val[1] 
        
                   res_change_y = res_est / res_val[0] 
        
                   # approximate raster size after resampling 
        
                   # by using factor of changed resolution 
        
                   rastersize = ( 
        
                       rastersize_x * res_change_x * rastersize_y * res_change_y 
        
                   )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1353 in 30f827c

"Please set e.g. region smaller."

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1356 in 30f827c

def _check_reset_region(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1374 in 30f827c

"Unable to check the computational region size"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1393 to 1394 in 30f827c

    
           def _adjust_region_size(self, num_cells, ns_res, ew_res): 
        
               """Helper method to adjust the region size

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1408 to 1409 in 30f827c

    
           ns_res = ns_res * fak 
        
           ew_res = ew_res * fak

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1411 in 30f827c

"g.region", ["nsres=%f" % ns_res, "ewres=%f" % ew_res, "-g"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1417 in 30f827c

"%f ewres: %f error: %s" % (ns_res, ew_res, stderr_buff)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1421 in 30f827c

"%f ewres: %f [num_cells: %d]" % (ns_res, ew_res, num_cells)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1424 to 1425 in 30f827c

    
           def _increment_progress(self, num=1): 
        
               """Increment the progress step by a specific number

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1429 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1433 to 1434 in 30f827c

    
           def _add_actinia_process(self, process: Process): 
        
               """Add an actinia process to the list and dictionary

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1438 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1442 to 1443 in 30f827c

    
           def _update_num_of_steps(self, num): 
        
               """Update the number of total steps

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1448 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1453 in 30f827c

self, module_name, module_parameter, proc, poll_time

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1455 to 1456 in 30f827c

    
                   """ 
        
                   Wait for a specific process. Catch termination requests, process

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1470 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1478 to 1499 in 30f827c

    
           else: 
        
               # Sleep some time and update the resource status 
        
               time.sleep(poll_time) 
        
               termination_check_count += 1 
        
               update_check_count += 1 
        
               # Check all 10 loops for termination 
        
               if termination_check_count == 10: 
        
                   termination_check_count = 0 
        
                   # check if the resource should be terminated 
        
                   # and kill the current process 
        
                   if ( 
        
                       self.resource_logger.get_termination( 
        
                           self.user_id, self.resource_id, self.iteration 
        
                       ) 
        
                       is True 
        
                   ): 
        
                       proc.kill() 
        
                       raise AsyncProcessTermination( 
        
                           "Process <%s> was terminated " 
        
                           "by user request" % module_name 
        
                       )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1501 to 1519 in 30f827c

    
           # Send all 100 loops a status update 
        
           if update_check_count == 100: 
        
               update_check_count = 0 
        
               # Check max runtime of process 
        
               curr_time = time.time() 
        
               if (curr_time - start_time) > self.process_time_limit: 
        
                   proc.kill() 
        
                   raise AsyncProcessTimeLimit( 
        
                       "Time (%i seconds) exceeded to run executable %s" 
        
                       % (self.process_time_limit, module_name) 
        
                   ) 
        
               # Reduce the length of the command line parameters for 
        
               # lesser logging overhead 
        
               mparams = str(module_parameter) 
        
               if len(mparams) > 100: 
        
                   mparams = "%s ... %s" % (mparams[0:50], mparams[-50:]) 
        
               message = ( 
        
                   f"Running executable {module_name} with parameters " 
        
                   f"{mparams} for {curr_time - start_time} seconds"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1521 in 30f827c

self._send_resource_update(message)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1526 to 1527 in 30f827c

    
                   """ 
        
                   Run a process actinia_core.core.common.process_object.Process) with

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1555 in 30f827c

self.user_id, self.resource_id, self.iteration

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1560 to 1561 in 30f827c

    
           "Process <%s> was terminated by " 
        
           "user request" % process.executable

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1614 in 30f827c

self.user_id, self.resource_id, self.iteration

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1619 to 1620 in 30f827c

    
           "Process <%s> was terminated " 
        
           "by user request" % process.executable

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1623 to 1626 in 30f827c

    
           message = "Running module %s with parameters %s" % ( 
        
               process.executable, 
        
               str(process.executable_params), 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1676 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1679 in 30f827c

mode="w+b", delete=True, dir=self.temp_file_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1682 in 30f827c

mode="w+b", delete=True, dir=self.temp_file_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1701 in 30f827c

val_splitted[j][2:]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1709 in 30f827c

"module."

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1718 in 30f827c

filtered_func_name, filtered_par_value

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1724 in 30f827c

stdin_file = open(tmp_file, "w")

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1727 in 30f827c

stdin_file = open(tmp_file, "r")

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1755 in 30f827c

process.executable, process.executable_params, proc, poll_time

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1795 in 30f827c

"Error while running executable <%s>" % process.executable

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1804 in 30f827c

self.progress_steps, self.temp_mapset_path, self.temp_file_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1809 in 30f827c

" can be saved!"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1819 to 1820 in 30f827c

    
           ): 
        
               """Create a temporary GRASS GIS environment

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1841 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1843 in 30f827c

# Create the temp database and link the

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1850 in 30f827c

grass_data_base=self.temp_grass_data_base, mapset_name="PERMANENT"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1861 in 30f827c

def _execute(self, skip_permission_check=False):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1887 in 30f827c

skip_permission_check=skip_permission_check

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1892 in 30f827c

skip_permission_check=skip_permission_check

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1902 in 30f827c

self, process_chain=None, skip_permission_check=False

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1947 in 30f827c

pc_step, self.iteration - 1

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1961 in 30f827c

self, process_chain=None, skip_permission_check=False

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1999 in 30f827c

def _parse_module_outputs(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2001 in 30f827c

formats: table, list or kv

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2009 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2017 in 30f827c

"Unable to find process id in module output dictionary"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 2027 to 2029 in 30f827c

    
           value_list = [] 
        
           for value in values: 
        
               value_list.append(value.strip())

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2053 in 30f827c

try:

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 2055 to 2056 in 30f827c

    
           except Exception: 
        
               pass

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 2066 to 2068 in 30f827c

    
               def _execute_process_list(self, process_list): 
        
                   """ 
        
                   Run all modules or executables that are specified in the process list

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 2088 to 2090 in 30f827c

    
           def _interim_results(self): 
        
               """Check if interim results should be saved or cleaned up"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2109 in 30f827c

def _final_cleanup(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2119 in 30f827c

def run(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2135 in 30f827c

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2184 in 30f827c

message=self.finish_message, results=self.module_results

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2189 in 30f827c

message=self.run_state["terminated"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2193 in 30f827c

message=self.run_state["time limit exceeded"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Line 1 in 30f827c

# -*- coding: utf-8 -*-

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Lines 24 to 26 in 30f827c

    
           """ 
        
           Utils for processing 
        
           """

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Line 29 in 30f827c

from actinia_core.core.common.config import global_config

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Line 57 in 30f827c

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Lines 60 to 61 in 30f827c

    
           imported_class = getattr(imported_module, actinia_class) 
        
           return imported_class

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Line 69 in 30f827c

raise e

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

ruff

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 445 in 9162db6

% (stdout_buff, stderr_buff, p.returncode)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 450 in 9162db6

"original project. Exception %s" % str(e)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 457 in 9162db6

self.temp_mapset_name, self.target_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 460 in 9162db6

os.path.join(self.user_project_path, self.temp_mapset_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 469 in 9162db6

"Remove interim results %s" % interim_dir

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 474 to 475 in 9162db6

    
           def _execute_process_list(self, process_list): 
        
               """Extend the mapset lock and execute the provided process list

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 483 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 493 to 494 in 9162db6

    
           "Unable to extend lock for mapset <%s>" 
        
           % self.target_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 506 in 9162db6

"temporary mapset <%s>" % self.temp_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 516 to 517 in 9162db6

    
           def _execute(self, skip_permission_check=False): 
        
               """Overwrite this function in subclasses

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 530 in 9162db6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 551 in 9162db6

pc_step, self.rdc.iteration - 1

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Line 585 in 9162db6

source_mapset_name=self.target_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/persistent_processing.py

Lines 597 to 600 in 9162db6

    
               def _final_cleanup(self): 
        
                   """ 
        
                   Final cleanup called in the run function at the very end of processing 
        
                   """

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1 in 9162db6

# -*- coding: utf-8 -*-

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 24 to 26 in 9162db6

    
           """ 
        
           Base class for asynchronous processing 
        
           """

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 28 in 9162db6

import math

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 31 in 9162db6

import requests

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 39 in 9162db6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 40 in 9162db6

from flask import json

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 43 to 53 in 9162db6

    
           from actinia_core.core.common.config import global_config, DEFAULT_CONFIG_PATH 
        
           from actinia_core.core.common.process_object import Process 
        
           from actinia_core.core.grass_init import GrassInitializer 
        
           from actinia_core.core.messages_logger import MessageLogger 
        
           from actinia_core.core.kvdb_lock import KvdbLockingInterface 
        
           from actinia_core.core.resources_logger import ResourceLogger 
        
           from actinia_core.core.mapset_merge_utils import change_mapsetname 
        
           from actinia_core.core.common.process_chain import ( 
        
               get_param_stdin_part, 
        
               ProcessChainConverter, 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 57 in 9162db6

RsyncError,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 59 to 62 in 9162db6

    
           from actinia_core.core.common.exceptions import AsyncProcessTimeLimit 
        
           from actinia_core.models.response_models import ( 
        
               ProcessingResponseModel, 
        
               ExceptionTracebackModel,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 64 in 9162db6

from actinia_core.models.response_models import (

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 65 in 9162db6

create_response_from_model,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 68 in 9162db6

)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 69 in 9162db6

from actinia_core.core.interim_results import InterimResult, get_directory_size

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 82 in 9162db6

class EphemeralProcessing(object):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 129 to 130 in 9162db6

    
           def __init__(self, rdc): 
        
               """Constructor

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 150 in 9162db6

DEFAULT_CONFIG_PATH

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 297 to 298 in 9162db6

    
           def _send_resource_update(self, message, results=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 325 to 326 in 9162db6

    
           def _send_resource_finished(self, message, results=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 355 to 356 in 9162db6

    
           def _send_resource_terminated(self, message, results=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 384 to 385 in 9162db6

    
           def _send_resource_time_limit_exceeded(self, message, results=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 413 to 414 in 9162db6

    
           def _send_resource_error(self, message, results=None, exception=None): 
        
               """Create an HTTP response document and send it to the status database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 443 to 444 in 9162db6

    
           def _send_to_database(self, document, final=False): 
        
               """Send the document to the database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 458 in 9162db6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 482 to 484 in 9162db6

    
           self.message_logger.error( 
        
               "Unable to send webhook request. Traceback: %s" 
        
               % str(run_state)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 487 in 9162db6

def _post_to_webhook(self, document, type):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 496 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 498 to 499 in 9162db6

    
           "Send POST request to %s webhook url: %s" 
        
           % (type, self.webhook_finished)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 535 in 9162db6

if not (500 <= resp.status_code and resp.status_code < 600):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 540 in 9162db6

webhook_not_reached is False and resp.status_code not in [200, 204]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 543 in 9162db6

"Unable to access %s webhook URL %s" % (type, webhook_url)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 548 in 9162db6

process chain where to continue

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 554 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 561 in 9162db6

self.user_id, self.resource_id

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 565 in 9162db6

self.user_id, self.resource_id, iter

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 585 to 586 in 9162db6

    
                   """ 
        
                   Create the process list and check for user permissions.

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 610 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 612 in 9162db6

if old_process_chain is not None:

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 614 in 9162db6

old_process_chain

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 623 in 9162db6

self.request_data

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 630 in 9162db6

process_chain

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 666 in 9162db6

% self.process_num_limit

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 676 in 9162db6

if process.exec_type == "grass" or process.exec_type == "exec":

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 686 to 687 in 9162db6

    
           "Module or executable <%s> is not supported" 
        
           % process.executable

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 691 to 697 in 9162db6

    
           "Wrong process description, type: %s " 
        
           "module/executable: %s, args: %s" 
        
           % ( 
        
               str(process.exec_type), 
        
               str(process.executable), 
        
               str(process.executable_params), 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 706 in 9162db6

def _setup(self, init_grass=True):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 708 in 9162db6

temporary grass database and temporary file directories

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 741 to 742 in 9162db6

    
           else: 
        
               self.setup_flag = True

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 763 in 9162db6

**kwargs, fluent_sender=fluent_sender

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 776 in 9162db6

self.user_credentials["permissions"]["process_time_limit"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 796 to 797 in 9162db6

    
           def _setup_paths(self): 
        
               """Helper method to setup the paths"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 799 in 9162db6

self.user_credentials["permissions"]["cell_limit"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 802 in 9162db6

self.user_credentials["permissions"]["process_num_limit"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 806 in 9162db6

self.grass_temp_database, self.temp_grass_data_base_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 812 in 9162db6

self.temp_grass_data_base, self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 815 in 9162db6

self.grass_data_base, self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 823 in 9162db6

self.grass_user_data_base, self.user_group

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 830 in 9162db6

self.grass_user_data_base, self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 843 in 9162db6

def _create_temp_database(self, mapsets=None):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 906 in 9162db6

f"<{self.project_name}>"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 924 in 9162db6

", Exception: %s" % str(e)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 928 in 9162db6

"""Helper method to link projects mapsets

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 939 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 951 to 954 in 9162db6

    
           left_over_mapsets = [] 
        
           for mapset in mapsets: 
        
               if mapset not in mapsets_to_link: 
        
                   left_over_mapsets.append(mapset)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 991 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 998 in 9162db6

mapset_path, os.R_OK & os.X_OK

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1023 to 1024 in 9162db6

    
           "Invalid mapset <%s> in project <%s>" 
        
           % (mapset, self.project_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1029 in 9162db6

"Unable to access global project <%s>" % self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1032 in 9162db6

msg = "Unable to access user project <%s>" % self.project_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1037 to 1039 in 9162db6

    
               self, grass_data_base, mapset_name="PERMANENT" 
        
           ): 
        
               """Sets up the GRASS environment to run modules

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1051 to 1052 in 9162db6

    
           "Initlialize GRASS grass_data_base: %s; project: %s; mapset: %s" 
        
           % (grass_data_base, self.project_name, mapset_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1073 to 1074 in 9162db6

    
           ): 
        
               """Create the temporary mapset and switch into it

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1109 in 9162db6

self.temp_project_path, temp_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1116 in 9162db6

"Rsync interim result mapset to temporary GRASS DB"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1127 in 9162db6

interim_result_mapset, self.temp_mapset_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1132 in 9162db6

"mapset"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1135 in 9162db6

os.path.dirname(self.temp_mapset_path)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1139 in 9162db6

"Rsync interim result file path to temporary GRASS DB"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1142 in 9162db6

interim_result_file_path, self.temp_file_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1147 in 9162db6

"new temporare file path"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1151 in 9162db6

"g.mapset", ["-c", "mapset=%s" % temp_mapset_name]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1159 in 9162db6

"mapset=%s" % (",".join(self.required_mapsets)),

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1165 in 9162db6

"search path: " + ",".join(self.required_mapsets)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1185 in 9162db6

self.temp_project_path, source_mapset_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1193 in 9162db6

def _cleanup(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1213 in 9162db6

f"Temporary file {tmpfile} can't be removed: {e}"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1216 in 9162db6

def _check_pixellimit_rimport(self, process_executable_params):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1223 to 1228 in 9162db6

    
           rimport_inp = [x for x in process_executable_params if "input=" in x][ 
        
               0 
        
           ].split("=")[1] 
        
           rimport_out = [x for x in process_executable_params if "output=" in x][ 
        
               0 
        
           ].split("=")[1]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1243 in 9162db6

"g.region", ["-ug"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1247 in 9162db6

"Unable to check the computational region size"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1257 to 1258 in 9162db6

    
           gdabuildvrt_params.append(vrt_out) 
        
           gdabuildvrt_params.append(rimport_inp)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1269 in 9162db6

"/usr/bin/gdalinfo", gdalinfo_params

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1289 in 9162db6

"r.import", [vrt_out, "-e"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1309 in 9162db6

[

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1313 to 1314 in 9162db6

    
               ][0].split("=")[1] 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1327 to 1329 in 9162db6

    
           [x for x in stdout_gregion.split("\n") if "nsres=" in x][ 
        
               0 
        
           ].split("=")[1]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1332 to 1334 in 9162db6

    
           [x for x in stdout_gregion.split("\n") if "ewres=" in x][ 
        
               0 
        
           ].split("=")[1]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1337 to 1346 in 9162db6

    
           if res_val: 
        
               if (res_val[0] < res_est) | (res_val[1] < res_est): 
        
                   # only check if smaller resolution set 
        
                   res_change_x = res_est / res_val[1] 
        
                   res_change_y = res_est / res_val[0] 
        
                   # approximate raster size after resampling 
        
                   # by using factor of changed resolution 
        
                   rastersize = ( 
        
                       rastersize_x * res_change_x * rastersize_y * res_change_y 
        
                   )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1353 in 9162db6

"Please set e.g. region smaller."

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1356 in 9162db6

def _check_reset_region(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1374 in 9162db6

"Unable to check the computational region size"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1393 to 1394 in 9162db6

    
           def _adjust_region_size(self, num_cells, ns_res, ew_res): 
        
               """Helper method to adjust the region size

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1408 to 1409 in 9162db6

    
           ns_res = ns_res * fak 
        
           ew_res = ew_res * fak

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1411 in 9162db6

"g.region", ["nsres=%f" % ns_res, "ewres=%f" % ew_res, "-g"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1417 in 9162db6

"%f ewres: %f error: %s" % (ns_res, ew_res, stderr_buff)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1421 in 9162db6

"%f ewres: %f [num_cells: %d]" % (ns_res, ew_res, num_cells)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1424 to 1425 in 9162db6

    
           def _increment_progress(self, num=1): 
        
               """Increment the progress step by a specific number

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1429 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1433 to 1434 in 9162db6

    
           def _add_actinia_process(self, process: Process): 
        
               """Add an actinia process to the list and dictionary

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1438 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1442 to 1443 in 9162db6

    
           def _update_num_of_steps(self, num): 
        
               """Update the number of total steps

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1448 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1453 in 9162db6

self, module_name, module_parameter, proc, poll_time

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1455 to 1456 in 9162db6

    
                   """ 
        
                   Wait for a specific process. Catch termination requests, process

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1470 in 9162db6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1478 to 1499 in 9162db6

    
           else: 
        
               # Sleep some time and update the resource status 
        
               time.sleep(poll_time) 
        
               termination_check_count += 1 
        
               update_check_count += 1 
        
               # Check all 10 loops for termination 
        
               if termination_check_count == 10: 
        
                   termination_check_count = 0 
        
                   # check if the resource should be terminated 
        
                   # and kill the current process 
        
                   if ( 
        
                       self.resource_logger.get_termination( 
        
                           self.user_id, self.resource_id, self.iteration 
        
                       ) 
        
                       is True 
        
                   ): 
        
                       proc.kill() 
        
                       raise AsyncProcessTermination( 
        
                           "Process <%s> was terminated " 
        
                           "by user request" % module_name 
        
                       )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1501 to 1519 in 9162db6

    
           # Send all 100 loops a status update 
        
           if update_check_count == 100: 
        
               update_check_count = 0 
        
               # Check max runtime of process 
        
               curr_time = time.time() 
        
               if (curr_time - start_time) > self.process_time_limit: 
        
                   proc.kill() 
        
                   raise AsyncProcessTimeLimit( 
        
                       "Time (%i seconds) exceeded to run executable %s" 
        
                       % (self.process_time_limit, module_name) 
        
                   ) 
        
               # Reduce the length of the command line parameters for 
        
               # lesser logging overhead 
        
               mparams = str(module_parameter) 
        
               if len(mparams) > 100: 
        
                   mparams = "%s ... %s" % (mparams[0:50], mparams[-50:]) 
        
               message = ( 
        
                   f"Running executable {module_name} with parameters " 
        
                   f"{mparams} for {curr_time - start_time} seconds"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1521 in 9162db6

self._send_resource_update(message)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1526 to 1527 in 9162db6

    
                   """ 
        
                   Run a process actinia_core.core.common.process_object.Process) with

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1555 in 9162db6

self.user_id, self.resource_id, self.iteration

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1560 to 1561 in 9162db6

    
           "Process <%s> was terminated by " 
        
           "user request" % process.executable

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1614 in 9162db6

self.user_id, self.resource_id, self.iteration

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1619 to 1620 in 9162db6

    
           "Process <%s> was terminated " 
        
           "by user request" % process.executable

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1623 to 1626 in 9162db6

    
           message = "Running module %s with parameters %s" % ( 
        
               process.executable, 
        
               str(process.executable_params), 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1676 in 9162db6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1679 in 9162db6

mode="w+b", delete=True, dir=self.temp_file_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1682 in 9162db6

mode="w+b", delete=True, dir=self.temp_file_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1701 in 9162db6

val_splitted[j][2:]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1709 in 9162db6

"module."

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1718 in 9162db6

filtered_func_name, filtered_par_value

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1724 in 9162db6

stdin_file = open(tmp_file, "w")

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1727 in 9162db6

stdin_file = open(tmp_file, "r")

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1755 in 9162db6

process.executable, process.executable_params, proc, poll_time

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1795 in 9162db6

"Error while running executable <%s>" % process.executable

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1804 in 9162db6

self.progress_steps, self.temp_mapset_path, self.temp_file_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1809 in 9162db6

" can be saved!"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 1819 to 1820 in 9162db6

    
           ): 
        
               """Create a temporary GRASS GIS environment

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1841 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1843 in 9162db6

# Create the temp database and link the

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1850 in 9162db6

grass_data_base=self.temp_grass_data_base, mapset_name="PERMANENT"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1861 in 9162db6

def _execute(self, skip_permission_check=False):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1887 in 9162db6

skip_permission_check=skip_permission_check

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1892 in 9162db6

skip_permission_check=skip_permission_check

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1902 in 9162db6

self, process_chain=None, skip_permission_check=False

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1947 in 9162db6

pc_step, self.iteration - 1

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1961 in 9162db6

self, process_chain=None, skip_permission_check=False

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 1999 in 9162db6

def _parse_module_outputs(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2001 in 9162db6

formats: table, list or kv

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2009 in 9162db6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2017 in 9162db6

"Unable to find process id in module output dictionary"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 2027 to 2029 in 9162db6

    
           value_list = [] 
        
           for value in values: 
        
               value_list.append(value.strip())

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2053 in 9162db6

try:

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 2055 to 2056 in 9162db6

    
           except Exception: 
        
               pass

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 2066 to 2068 in 9162db6

    
               def _execute_process_list(self, process_list): 
        
                   """ 
        
                   Run all modules or executables that are specified in the process list

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Lines 2088 to 2090 in 9162db6

    
           def _interim_results(self): 
        
               """Check if interim results should be saved or cleaned up"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2109 in 9162db6

def _final_cleanup(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2119 in 9162db6

def run(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2135 in 9162db6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2184 in 9162db6

message=self.finish_message, results=self.module_results

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2189 in 9162db6

message=self.run_state["terminated"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral_processing.py

Line 2193 in 9162db6

message=self.run_state["time limit exceeded"]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Line 1 in 9162db6

# -*- coding: utf-8 -*-

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Lines 24 to 26 in 9162db6

    
           """ 
        
           Utils for processing 
        
           """

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Line 29 in 9162db6

from actinia_core.core.common.config import global_config

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Line 57 in 9162db6

"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Lines 60 to 61 in 9162db6

    
           imported_class = getattr(imported_module, actinia_class) 
        
           return imported_class

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/common/utils.py

Line 69 in 9162db6

raise e

* reactivate redis queue * rename variable * make queue type configurable * enhance dev setup for redis queue * lint * move user_auth * move user auth * move base_login * move map_layer_base * move renderer_base * move resource_base * fix relative imports * lint * first splitup: raster_colors * splitup download_cache_management * splitup ephemeral_custom_processing * splitup ephemeral_processing_with_export * splitup ephemeral_processing * splitup renderer_base * splitup persistent_processing * splitup location_management * splitup map_layer_management * splitup mapset_management * splitup persistent_mapset_merger * splitup process_validation * splitup raster_export * splitup raster_layer * splitup raster_legend * lint * splitup raster_renderer * splitup resource_storage_management * splitup strds_management * splitup strds_raster_management * splitup strds_renderer * splitup vector_layer * splitup vector_renderer * lint * make inheritance more clear * add readme * fix import

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

ruff

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Line 438 in c072d39

message = "Export raster layer <%s> with format %s" % (

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Lines 457 to 458 in c072d39

    
           "Export vector layer <%s> to PostgreSQL database" 
        
           % (file_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Line 468 in c072d39

message = "Export vector layer <%s> with format %s" % (

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Line 481 in c072d39

tmp_file=tmp_file, file_name=file_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Line 484 in c072d39

message = "Export strds layer <%s> with format %s" % (

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Line 495 in c072d39

"Unknown export format %s" % output_type

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Line 505 in c072d39

output_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Line 514 in c072d39

resource_url, file_name, output_type

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Lines 518 to 519 in c072d39

    
           def _execute(self, skip_permission_check=False): 
        
               """Overwrite this function in subclasses

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_core/processing/actinia_processing/ephemeral/ephemeral_processing_with_export.py

Lines 543 to 546 in c072d39

    
               def _final_cleanup(self): 
        
                   """ 
        
                   Overwrite this function in subclasses to perform the final cleanup 
        
                   """

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/processing/actinia_processing/ephemeral/persistent_processing.py

Line 31 in c072d39

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/processing/actinia_processing/ephemeral_processing.py

Line 38 in c072d39

from actinia_core.core.common.config import DEFAULT_CONFIG_PATH, global_config

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/processing/actinia_processing/ephemeral_processing.py

Lines 61 to 63 in c072d39

    
           from flask import json 
        
           from requests.auth import HTTPBasicAuth

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

ruff

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1060 to 1061 in f42f3f6

    
           "Initlialize GRASS grass_data_base: %s; project: %s; mapset: %s" 
        
           % (grass_data_base, self.project_name, mapset_name),

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1082 to 1083 in f42f3f6

    
           ): 
        
               """Create the temporary mapset and switch into it

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1164 in f42f3f6

["-c", "mapset=%s" % temp_mapset_name],

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1172 in f42f3f6

"mapset=%s" % (",".join(self.required_mapsets)),

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1207 in f42f3f6

def _cleanup(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1230 in f42f3f6

def _check_pixellimit_rimport(self, process_executable_params):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1237 to 1242 in f42f3f6

    
           rimport_inp = [x for x in process_executable_params if "input=" in x][ 
        
               0 
        
           ].split("=")[1] 
        
           rimport_out = [x for x in process_executable_params if "output=" in x][ 
        
               0 
        
           ].split("=")[1]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1272 to 1273 in f42f3f6

    
           gdabuildvrt_params.append(vrt_out) 
        
           gdabuildvrt_params.append(rimport_inp)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1326 in f42f3f6

[

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1330 in f42f3f6

][0].split("=")[1],

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1344 to 1346 in f42f3f6

    
           [x for x in stdout_gregion.split("\n") if "nsres=" in x][ 
        
               0 
        
           ].split("=")[1],

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1349 to 1351 in f42f3f6

    
           [x for x in stdout_gregion.split("\n") if "ewres=" in x][ 
        
               0 
        
           ].split("=")[1],

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1354 to 1363 in f42f3f6

    
           if res_val: 
        
               if (res_val[0] < res_est) | (res_val[1] < res_est): 
        
                   # only check if smaller resolution set 
        
                   res_change_x = res_est / res_val[1] 
        
                   res_change_y = res_est / res_val[0] 
        
                   # approximate raster size after resampling 
        
                   # by using factor of changed resolution 
        
                   rastersize = ( 
        
                       rastersize_x * res_change_x * rastersize_y * res_change_y 
        
                   )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1373 in f42f3f6

def _check_reset_region(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1410 to 1411 in f42f3f6

    
           def _adjust_region_size(self, num_cells, ns_res, ew_res): 
        
               """Helper method to adjust the region size

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1425 to 1426 in f42f3f6

    
           ns_res = ns_res * fak 
        
           ew_res = ew_res * fak

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1429 in f42f3f6

["nsres=%f" % ns_res, "ewres=%f" % ew_res, "-g"],

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1435 in f42f3f6

"%f ewres: %f error: %s" % (ns_res, ew_res, stderr_buff),

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1442 to 1443 in f42f3f6

    
           def _increment_progress(self, num=1): 
        
               """Increment the progress step by a specific number

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1452 to 1453 in f42f3f6

    
           def _add_actinia_process(self, process: Process): 
        
               """Add an actinia process to the list and dictionary

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1462 to 1463 in f42f3f6

    
           def _update_num_of_steps(self, num): 
        
               """Update the number of total steps

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1521 to 1522 in f42f3f6

    
           "Process <%s> was terminated " 
        
           "by user request" % module_name,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1540 in f42f3f6

mparams = "%s ... %s" % (mparams[0:50], mparams[-50:])

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1585 to 1586 in f42f3f6

    
           "Process <%s> was terminated by " 
        
           "user request" % process.executable,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1646 to 1647 in f42f3f6

    
           "Process <%s> was terminated " 
        
           "by user request" % process.executable,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1650 to 1653 in f42f3f6

    
           message = "Running module %s with parameters %s" % ( 
        
               process.executable, 
        
               str(process.executable_params), 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1755 in f42f3f6

stdin_file = open(tmp_file, "w")

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1758 in f42f3f6

stdin_file = open(tmp_file)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1829 in f42f3f6

"Error while running executable <%s>" % process.executable,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 1855 to 1856 in f42f3f6

    
           ): 
        
               """Create a temporary GRASS GIS environment

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 1898 in f42f3f6

def _execute(self, skip_permission_check=False):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 2041 in f42f3f6

def _parse_module_outputs(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 2043 in f42f3f6

formats: table, list or kv

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 2068 to 2070 in f42f3f6

    
           value_list = [] 
        
           for value in values: 
        
               value_list.append(value.strip())

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 2094 in f42f3f6

try:

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 2096 to 2097 in f42f3f6

    
           except Exception: 
        
               pass

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 2107 to 2108 in f42f3f6

    
           def _execute_process_list(self, process_list): 
        
               """Run all modules or executables that are specified in the process list

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Lines 2128 to 2129 in f42f3f6

    
           def _interim_results(self): 
        
               """Check if interim results should be saved or cleaned up"""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 2148 in f42f3f6

def _final_cleanup(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing.py

Line 2158 in f42f3f6

def run(self):

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 1 in f42f3f6

# -*- coding: utf-8 -*-

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 24 to 25 in f42f3f6

    
           """ 
        
           Asynchronous computation in specific temporary generated mapsets

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 29 in f42f3f6

from actinia_core.core.common.process_object import Process

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 31 in f42f3f6

from actinia_processing_lib.ephemeral_processing import EphemeralProcessing

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 44 to 45 in f42f3f6

    
               """ 
        
               This class processes GRASS data on the local machine in an temporary mapset

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 55 to 57 in f42f3f6

    
               def __init__(self, rdc): 
        
                   """ 
        
                   Setup the variables of this class

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 72 in f42f3f6

additional_options=[],

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 100 in f42f3f6

suffix = ".tif"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 108 in f42f3f6

executable_params=["raster=%s" % raster_name, "-g"],

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 127 in f42f3f6

"COG driver not available, using GTiff driver"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 136 to 138 in f42f3f6

    
           "input=%s" % raster_name, 
        
           "format=%s" % format, 
        
           "output=%s" % output_path,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 187 in f42f3f6

"Only GTiff driver is supported for STRDS export."

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 192 to 196 in f42f3f6

    
           "input=%s" % strds_name, 
        
           "format=%s" % format, 
        
           "output=%s" % output_path, 
        
           "directory=%s" % self.temp_file_path, 
        
           "compression=%s" % "gzip",

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 217 in f42f3f6

self, vector_name, format="GPKG", additional_options=[]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 219 to 221 in f42f3f6

    
                   """ 
        
                   Export a specific vector layer with v.out.ogr using a specific output 
        
                   format

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 242 in f42f3f6

prefix = ""

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 266 to 268 in f42f3f6

    
           "input=%s" % vector_name, 
        
           "format=%s" % format, 
        
           "output=%s" % file_name,

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 288 in f42f3f6

self.temp_file_path, archive_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 308 to 311 in f42f3f6

    
                   self, vector_name, dbstring, output_layer=None, additional_options=[] 
        
               ): 
        
                   """ 
        
                   Export a specific vector layer with v.out.postgis to a PostGIS database

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 324 in f42f3f6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 326 in f42f3f6

args = ["-l", "input=%s" % vector_name, "output=%s" % dbstring]

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 329 in f42f3f6

args.append("output_layer=%s" % output_layer)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 347 in f42f3f6

"""Export a specific file

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 374 in f42f3f6

self.temp_file_path, archive_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 393 to 395 in f42f3f6

    
               def _export_resources(self, use_raster_region=False): 
        
                   """ 
        
                   Export all resources that were listed in the process chain description.

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 404 in f42f3f6

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 410 to 411 in f42f3f6

    
               self.user_id, self.resource_id, self.iteration 
        
           )

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 416 in f42f3f6

"Resource export was terminated by user request"

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 420 in f42f3f6

if resource["export"]["type"] in [

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 425 in f42f3f6

]:

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 436 in f42f3f6

message = "Export raster layer <%s> with format %s" % (

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 455 to 456 in f42f3f6

    
           "Export vector layer <%s> to PostgreSQL database" 
        
           % (file_name)

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 466 in f42f3f6

message = "Export vector layer <%s> with format %s" % (

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 479 in f42f3f6

tmp_file=tmp_file, file_name=file_name

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 482 in f42f3f6

message = "Export strds layer <%s> with format %s" % (

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 493 in f42f3f6

"Unknown export format %s" % output_type

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 503 in f42f3f6

output_path

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Line 512 in f42f3f6

resource_url, file_name, output_type

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 516 to 517 in f42f3f6

    
           def _execute(self, skip_permission_check=False): 
        
               """Overwrite this function in subclasses

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/ephemeral_processing_with_export.py

Lines 541 to 544 in f42f3f6

    
               def _final_cleanup(self): 
        
                   """ 
        
                   Overwrite this function in subclasses to perform the final cleanup 
        
                   """

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/persistent_processing.py

Line 32 in f42f3f6

from actinia_processing_lib.exceptions import AsyncProcessError

[ruff] _{reported by reviewdog 🐶}

actinia-processing-lib/src/actinia_processing_lib/persistent_processing.py

Line 34 in f42f3f6

tests/integrationtests/test_ephemeral_processing.py

anikaweinmann and others added 30 commits November 26, 2021 11:54

try to fix uncontrolled data used in path (#280)

c033490

Co-authored-by: anikaweinmann <[email protected]>

actinia tests: fix test descriptions (#371)

a5ea4c9

* tests: fix test descriptions This PR fixes some test descriptions and a few typos.

Black (#378)

2a7c354

* linting tests + black workflow * linting * black * further black * fix unittests * fix test Co-authored-by: anikaweinmann <[email protected]>

Black (#378)

c0f0f1b

* linting tests + black workflow * linting * black * further black * fix unittests * fix test Co-authored-by: anikaweinmann <[email protected]>

Black (#378)

85a0600

* linting tests + black workflow * linting * black * further black * fix unittests * fix test Co-authored-by: anikaweinmann <[email protected]>

Black (#378)

3870c9f

* linting tests + black workflow * linting * black * further black * fix unittests * fix test Co-authored-by: anikaweinmann <[email protected]>

Black (#378)

4b9741f

* linting tests + black workflow * linting * black * further black * fix unittests * fix test Co-authored-by: anikaweinmann <[email protected]>

Add job queue name to response (#380)

da38d53

* add job queue name to response * lint * Update src/actinia_core/models/response_models.py Co-authored-by: Markus Neteler <[email protected]> * black * undo-black Co-authored-by: Markus Neteler <[email protected]>

Change pc style for module description for importer and exporter in m…

8c606bf

…odule plugin (#387) * change pc style for module description for importer and exporter in module plugin * black

Start with the adjustment of the interim results (#390)

e17aa1e

adjustment of the interim results Co-authored-by: anikaweinmann <[email protected]> Co-authored-by: Carmen Tawalika <[email protected]>

Improve interim results by include additional mapsets (#399)

e999c9a

* Add possibility to configure pattern to save addional mapsets in the interim resutls Co-authored-by: anikaweinmann <[email protected]>

Add raster VRT support and support for mapset names by interim result…

960ce05

…s (#410) * add raster VRT support and support for mapset names by interim results Co-authored-by: anikaweinmann <[email protected]>, linakrisztian <[email protected]>

Add raster VRT support and support for mapset names by interim result…

089ce41

…s (#410) * add raster VRT support and support for mapset names by interim results Co-authored-by: anikaweinmann <[email protected]>, linakrisztian <[email protected]>

Black (#458)

b942acd

* make resource resource delition asynchron * trailing and ending precommit ... * do not use pylint * trigger linting * linting * linting --------- Co-authored-by: anikaweinmann <[email protected]>

Black (#458)

d62628b

* make resource resource delition asynchron * trailing and ending precommit ... * do not use pylint * trigger linting * linting * linting --------- Co-authored-by: anikaweinmann <[email protected]>

Black (#458)

fff8601

* make resource resource delition asynchron * trailing and ending precommit ... * do not use pylint * trigger linting * linting * linting --------- Co-authored-by: anikaweinmann <[email protected]>

Allow separate config for worker Part 1 (#376)

5d6d167

* allow separate config for worker * test tests * fix test * create download cache if not exists * create download cache if not exists * create resource storage if not exists * create log folder * further work on tests * add TODOs in tests

Add stdin for parameters (#501)

98d6a47

* add stdin for parameters * add filter to stdout for other modules * add tests --------- Co-authored-by: anikaweinmann <[email protected]>

Merge branch 'main' of github.com:actinia-org/actinia-core into main

7366ec5

Lint (#535)

9431eb1

* lint * update lint workflow * update to new black version * mdformat

mmacata and others added 5 commits April 1, 2022 11:34

Move modules used by plugins (#221)

6cd94d5

* move modules used by plugins * move shared modules to common folder

github-actions bot reviewed Jun 2, 2025

View reviewed changes

mmacata mentioned this pull request Jun 2, 2025

Move modules and methods to plugin and new libs actinia-org/actinia-core#627

Merged

split up module from actinia-core

9162db6

github-actions bot reviewed Jun 2, 2025

View reviewed changes

mmacata and others added 3 commits June 2, 2025 17:14

cleanup

0c024c6

lint

ce9c1de

github-actions bot reviewed Jun 4, 2025

View reviewed changes

simplify folder structure

f42f3f6

github-actions bot reviewed Jun 4, 2025

View reviewed changes

lint

fec4f43

mmacata marked this pull request as ready for review June 6, 2025 06:14

linakrisztian approved these changes Jun 6, 2025

View reviewed changes

fix tests

bef809e

github-actions bot reviewed Jun 6, 2025

View reviewed changes

mmacata added 4 commits June 6, 2025 15:20

lint

5511cb7

fix test call

358c298

lint

a1094bb

update README

13cc529

mmacata merged commit 89a503b into main Jun 12, 2025
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Init #2

Init #2

Uh oh!

mmacata commented Jun 2, 2025 •

edited

Loading

Uh oh!

github-actions bot left a comment

Uh oh!

github-actions bot left a comment

Uh oh!

github-actions bot left a comment

Uh oh!

github-actions bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

	"Mapset <%s> exists in the global "
	"dataset and can not be modified." % mapset

	"Unable to access mapset <%s> "
	"path %s" % (mapset, self.orig_mapset_path)

	def _check_target_mapset_exists(self):
	"""Check if the target mapset exists

	def _check_lock_target_mapset(self):
	"""
	Check if the target mapset exists and lock it, then lock the temporary
	mapset

	"Unable to lock project/mapset <%s/%s>, "
	"resource is already locked"
	% (self.project_name, self.target_mapset_name)

	"project/mapset <%s/%s> locked"
	% (self.project_name, self.target_mapset_name)

	def _merge_mapset_into_target(self, source_mapset, target_mapset):
	"""Link the source mapset content into the target mapset

	"Copy source mapset <%s> content "
	"into the target mapset <%s>" % (source_mapset, target_mapset)

	def _copy_merge_tmp_mapset_to_target_mapset(self):
	"""Copy the temporary mapset into the original project

	"Copy temporary mapset from %s to %s"
	% (
	self.temp_mapset_path,
	os.path.join(self.user_project_path, self.target_mapset_name),
	)

	"Copy temporary mapset <%s> to target project "
	"<%s>" % (self.temp_mapset_name, self.project_name)

	"Copy temporary mapset <%s> to target project "
	"<%s>" % (self.target_mapset_name, self.project_name)

	def _execute_process_list(self, process_list):
	"""Extend the mapset lock and execute the provided process list

	"Unable to extend lock for mapset <%s>"
	% self.target_mapset_name

	def _execute(self, skip_permission_check=False):
	"""Overwrite this function in subclasses

	def _final_cleanup(self):
	"""
	Final cleanup called in the run function at the very end of processing
	"""

	from actinia_core.core.common.config import global_config, DEFAULT_CONFIG_PATH
	from actinia_core.core.common.process_object import Process
	from actinia_core.core.grass_init import GrassInitializer
	from actinia_core.core.messages_logger import MessageLogger
	from actinia_core.core.kvdb_lock import KvdbLockingInterface
	from actinia_core.core.resources_logger import ResourceLogger
	from actinia_core.core.mapset_merge_utils import change_mapsetname
	from actinia_core.core.common.process_chain import (
	get_param_stdin_part,
	ProcessChainConverter,
	)

	from actinia_core.core.common.exceptions import AsyncProcessTimeLimit
	from actinia_core.models.response_models import (
	ProcessingResponseModel,
	ExceptionTracebackModel,

	def _send_resource_update(self, message, results=None):
	"""Create an HTTP response document and send it to the status database

	def _send_resource_finished(self, message, results=None):
	"""Create an HTTP response document and send it to the status database

Uh oh!

Init #2

Init #2

Uh oh!

Conversation

mmacata commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mmacata commented Jun 2, 2025 •

edited

Loading

	def _send_resource_terminated(self, message, results=None):
	"""Create an HTTP response document and send it to the status database

	def _send_resource_time_limit_exceeded(self, message, results=None):
	"""Create an HTTP response document and send it to the status database

	def _send_resource_error(self, message, results=None, exception=None):
	"""Create an HTTP response document and send it to the status database

	def _send_to_database(self, document, final=False):
	"""Send the document to the database

	self.message_logger.error(
	"Unable to send webhook request. Traceback: %s"
	% str(run_state)

	"Send POST request to %s webhook url: %s"
	% (type, self.webhook_finished)

	"Module or executable <%s> is not supported"
	% process.executable

	"Wrong process description, type: %s "
	"module/executable: %s, args: %s"
	% (
	str(process.exec_type),
	str(process.executable),
	str(process.executable_params),
	)

	def _setup_paths(self):
	"""Helper method to setup the paths"""

	left_over_mapsets = []
	for mapset in mapsets:
	if mapset not in mapsets_to_link:
	left_over_mapsets.append(mapset)