CKAN Harvest extension not working in CKAN 2.11.3 using docker setup #9073
-
CKAN versionCKAN 2.11.3 Describe the bugA clear and concise description of what the bug is. Steps to reproduceSteps to reproduce the behavior: Steps2: Change docker-compose.yml with below code Step3: Update ckan->Dockerfile: Expected behaviorCKAN exenstion should be working without any error Additional detailsIf possible, please provide the full stack trace of the error raised, or add screenshots to help explain your problem. I see these error in logs after docker compose up -d |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 9 replies
-
|
Can you try installing the harvest extension in the way it is suggested in the example |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I have tried. Getting below some permission error: [ckan 2/5] RUN pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@master#egg=ckanext-harvest' && pip3 install -r /srv/app/src/ckanext-harvest/pip-requirements.txt
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @jktittalom Let's dig a bit deeper. ;-) First look into https://github.com/ckan/ckan-docker-base/blob/main/ckan-2.11/Dockerfile to understand what is done there. Furthermore @amercader gave the recommendation to install the extension via pip with dependencies, you manually cloned the extension and run a local Add this to your Dockerfile: USER root
RUN pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@master#egg=ckanext-harvest' && \
pip3 install -r ${APP_DIR}/src/ckanext-harvest/requirements.txt
USER ckanMore details about the installation are described here. I recommend not to remove ckanext-harvest before installing it, as it may remove dependencies you might need later - and this procedure might increase your docker image size. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jktittalom , Did you actually do an docker image rebuild? Please always ensure you build a new image first, when changing the Dockerfile. docker compose build --no-cache -progress=plainPaste your build logs here. Where did you define your redis and database connection details? .env? Afterwards run docker compose down -v
docker compose up -d
docker compose logs ckanPaste your run logs here again. Please use three backticks to format your log output more readable. Furthermore i recommend adding your user to a docker group, to avoid running docker compose via sudo: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @dev-rke , Thanks for quick response and suggestion. Please see below logs and other commands outputs: CKAN was running without Harvest extension... sudo docker compose down -v ============================= Build logs ========================== sudo docker compose build --no-cache --progress=plain --progress is a global compose flag, better use `docker compose --progress xx build ... #2 [ckan internal] load build definition from Dockerfile #3 [db internal] load build definition from Dockerfile #4 [nginx internal] load build definition from Dockerfile #5 [nginx internal] load metadata for docker.io/library/nginx:stable-alpine #6 [db internal] load metadata for docker.io/library/postgres:16-alpine #7 [db internal] load .dockerignore #8 [db internal] load build context #9 [ckan internal] load metadata for docker.io/ckan/ckan-base:2.11 #10 [db 1/2] FROM docker.io/library/postgres:16-alpine@sha256:af04f2185358fe933bb3f7fe2c42f0408b1bab4367c6d1b971bac3935636c21b #5 [nginx internal] load metadata for docker.io/library/nginx:stable-alpine #11 [ckan internal] load .dockerignore #12 [nginx internal] load .dockerignore #10 [db 1/2] FROM docker.io/library/postgres:16-alpine@sha256:af04f2185358fe933bb3f7fe2c42f0408b1bab4367c6d1b971bac3935636c21b #13 [nginx internal] load build context #10 [db 1/2] FROM docker.io/library/postgres:16-alpine@sha256:af04f2185358fe933bb3f7fe2c42f0408b1bab4367c6d1b971bac3935636c21b #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #15 [nginx 1/8] FROM docker.io/library/nginx:stable-alpine@sha256:8f2bcf97c473dfe311e79a510ee540ee02e28ce1e6a64e1ef89bfad32574ef10 #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #15 [nginx 1/8] FROM docker.io/library/nginx:stable-alpine@sha256:8f2bcf97c473dfe311e79a510ee540ee02e28ce1e6a64e1ef89bfad32574ef10 #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #16 [nginx 2/8] COPY setup/nginx.conf /etc/nginx/nginx.conf #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #17 [nginx 3/8] COPY setup/index.html /usr/share/nginx/html/index.html #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #18 [nginx 4/8] COPY setup/error.html /usr/share/nginx/html/error.html #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #19 [nginx 5/8] COPY setup/default.conf /etc/nginx/conf.d/ #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #20 [nginx 6/8] COPY setup/ckan.conf /etc/nginx/conf.d/ #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #10 [db 1/2] FROM docker.io/library/postgres:16-alpine@sha256:af04f2185358fe933bb3f7fe2c42f0408b1bab4367c6d1b971bac3935636c21b #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #21 [db 2/2] COPY --chown=postgres:postgres docker-entrypoint-initdb.d /docker-entrypoint-initdb.d #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #22 [db] exporting to image #23 [nginx 7/8] RUN mkdir -p /etc/nginx/certs #24 [db] resolving provenance for metadata file #25 [nginx 8/8] COPY setup/ckan-local.* /etc/nginx/certs/ #26 [nginx] exporting to image #27 [nginx] resolving provenance for metadata file #14 [ckan 1/3] FROM docker.io/ckan/ckan-base:2.11@sha256:19be56ca09ba85986da54471f6c8ac065fbc6f84d7b019bad21cacd17eafd5a6 #28 [ckan 2/3] RUN apt-get update && apt-get install -y vim #29 [ckan 3/3] RUN for d in /srv/app/patches/*; do if [ -d $d ]; then for f in #30 [ckan] exporting to image #31 [ckan] resolving provenance for metadata file ========================================= ======================== Where did you define your redis and database connection details? .env? Yes, Please see below which I have in .evn Host PortsCKAN_PORT_HOST=5000 CKAN databasesPOSTGRES_USER=postgres Test database connectionsTEST_CKAN_SQLALCHEMY_URL=postgres://ckan:ckan@db/ckan_test Dev settingsUSE_HTTPS_FOR_DEV=false CKAN coreCKAN_VERSION=2.10.0 See https://docs.ckan.org/en/latest/maintaining/configuration.html#api-token-settingsCKAN___API_TOKEN__JWT__ENCODE__SECRET=string:CHANGE_ME SolrSOLR_IMAGE_VERSION=2.10-solr9 RedisREDIS_VERSION=6 DatapusherDATAPUSHER_VERSION=0.0.21 NGINXNGINX_PORT=80 ExtensionsCKAN__PLUGINS="image_view text_view datatables_view datastore datapusher envvars harvest ckan_harvester" ========================================= ckan logs ======================== sudo docker compose up -d [+] Running 21/21 ===================================================================== ckan-1 | Setting a temporary value for ckan.datapusher.api_token |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jktittalom , your setup seems to be made up. This must not happen if you follow my guideline. Furthermore your .env file overrides ENV-Rules you specified in your Dockerfile, e.g. the CKAN__PLUGINS of your Dockerfile is ignored and the one of the .env file is used. $ docker compose exec ckan bash
$ envFor myself i did not use the ckan-base docker setup, but built it up step by step on my own - also to avoid these kind of issues. This is my last answer to your unproperly formatted dumps of character-width-related logs. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @dev-rke, Thanks for quick response and suggestion. I'll explore Y the patch related code in docker file creating issue for extension installation. Thanks a lots again. |
Beta Was this translation helpful? Give feedback.
Hi @dev-rke,
Thanks for quick response and suggestion.
Your suggestion for DockerFile seems working, Now there are no issue for harvest extension.
I'll explore Y the patch related code in docker file creating issue for extension installation.
Thanks a lots again.