[Port dspace-7_x] Make statistics autocommit much more frequently #9790
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit 5c9af97)
10 tasks
milanmajchrak
pushed a commit
to dataquest-dev/DSpace
that referenced
this pull request
Sep 23, 2024
[Port dspace-7_x] Make statistics autocommit much more frequently
milanmajchrak
added a commit
to dataquest-dev/DSpace
that referenced
this pull request
Sep 24, 2024
#769) [Port dspace-7_x] Make statistics autocommit much more frequently Co-authored-by: Tim Donohue <[email protected]>
milanmajchrak
added a commit
to dataquest-dev/DSpace
that referenced
this pull request
Dec 6, 2024
* UFAL/Removed duplicities of the bitstreams in the cmdi (#766) * Removed duplicities of the bitstreams in the cmdi. * Fixed checkstyle violation * used lindat code instead of vanilla. * Ufal/Preview issues (#764) * Ensure the content preview doesn't overload maximum length of the column in the database. And encode the input stream into UTF-8. * Do not store HTML content in the database because it could be longer than the limit of the database column. * UFAL/Encoded the UTF-8 characters from the redirect URL to UTF (#758) * Encoded the UTF-8 characters from the redirect URL to UTF * Moved ClarinUtils into Utils class * Added a new `dq` package into ComponentScan * Moved dq.Utils into DSpace utils.Utils because the components with the same name causes conflicts. * Removed *.dq component scan from the App * Merge pull request DSpace#9790 from DSpace/backport-9775-to-dspace-7_x (#769) [Port dspace-7_x] Make statistics autocommit much more frequently Co-authored-by: Tim Donohue <[email protected]> * test for bitstream with null value of sizebytes * Update README.md * UFAL/Shibboleth - load more net-id headers e.g. persistent-id (#772) * Load netid from more than one header. authentication-shibboleth.netid-header could be list, not only single value * Shibboleth login - sort the emails passed in the shibboleth email header and get the first one. * The user is redirected to the login page when it is trying to update eperson email which is already assigned to another eperson. * Sorting emails is moved into specific method and ShibbolethLoginFilter is updated following the ShibAuthentication changes * Fixed failing tests * The ClarinShibbolethLoginFilter and ClarinShibAuthentication has duplicate code, I put it into static method. * Propagate the verification token to the DB after the email is successfully sent. (#786) * UFAL/Enhanced type-bind feature (#762) * type bind is correctly rendered in the FE, but BE is still not working * Synchronized the `submission-forms_cs.xml` * Added doc into `submission-forms` about enhanced type-bind `field` * Updated `local.cfg` for tests - added type-bind property * Updated docs for the customized type-bind configuration property. * Updated MetadataValidation following the type-bind customization. * Added isAllowed function for multiple type-bind definitions * Added some docs for the new method * The values of the input wasn't loaded. * Allowed fields could be empty when they should have values. * Used isEmpty function and created constant for the `=>`. * create preview content for tar files (#759) * create preview content for tar files * Added right logs * devided extractFile funs into several separated smaller funs * added comment and removed empty line * added empty lines and removed unwanted comments * removed empty line * used consts * try incorrect identification level * log errors and removed unneeded consts --------- Co-authored-by: milanmajchrak <[email protected]> * Internal/fix failing Clarin integration test (#796) * Initial commit * Ignore the test class from where the tests has started failing. * Ignored half of tests in the ClarinShibbolethLoginFilterIT file * Ignored all tests * unignore some tests * 3 tests ignored 3 allowed * Maybe the problematic test is hidden between 3 unignored tests * two candidates * The last candidate * Ignore just wrong test and all test should pass * Clean up object created in the test. * Removed unused import. * Check the user which is going to be deleted is not null. * Rest api for handle resolution with metadata * decoded rawvalues and response json modification * used static extractMetadata funct in HandlePlugin * return dict: * removed property for test from local * Add default licenses - from ZCU update (#801) * Added flyway file to insert default licenses with license labels and mappings * Added required header * UFAL/share submission by email (#780) * Updated table workspace with share token, created endpoint to generate share token and it is sent via email * Added method to get workspaceitem via share token. * Added an endpoint for changing the submission's owner. * Added license headers * Added test for fetching item with share Token * Added tests to check the owner is changed * Added better explanation why the BE must return Page object in the search endpoint * Validate the user in the SubmissionController, it cannot be null * Updated email - some values are fetched directly from the configuration property * Updated preAuthorization method to ADD instead of WRITE (write is used for controlling authorization for modifying the Item) and updated shareURL * Authorize the submitter which is trying to take sharing item via shareToken. * Update integration test following the feature update * Import default licenses only if the license tables are empty. (#808) * Oai elg crosswalk (#798) * problem with language code * fix amount and sizeUnitOther rest-tests errors * added language coding * new language and funding project mishmash array position * added isoCodes * removed handle from item submission (#812) * UFAL/Shibboleth - netid-header should use getArrayProperty everywhere (#807) * Fetch netid as array from the cfg. Now netid as array is used everywhere. Added integration test to ask for an email when the user send only persistent-id in the shib header. * Fixed checkstyle issue * The user is not signed in without using link with the verification token from the email/ (#809) * UFAL/Shibboleth - show error in the UI when shibboleth authentication is failed (#810) * The user is not signed in without using link with the verification token from the email/ * Send a redirect to UI with specific parameter that the Shibboleth authorization wasn't successful * UFAL/Autocomplete enhancement (#768) * Added solr index `handle_title_ac` and `_comp` for the Item * Added support for searching results from specific solr indexes. 1. Updated submission-forms autocomplete definition to specify a specific index. 2. Updated configuration is provided via REST API. 3. Create a new `/suggestions` endpoint for searching values from custom solr index - it returns VocabularyEntry page. * Supported searching Item byHandle when passed a handle as parameter without handle canonical prefix. * Added autocompleteCustom `solr-subject_ac` and `handle_title_ac`. * Added autocompleteCustom `solr-publisher_ac`. * Added cfg property to define a separator from the solr value to get only display value. * Added autocompleteCustom `solr-dataProvider_ac` * Refactored code and created integration test for the SuggestionRestController * Updated suggestion integration tests because it has had a conflict with another IT * Added doc and changed `autocomplete.custom.format` to `autocomplete.custom.separator` for proper naming. * Added support for loading suggestions from the json file - need to refactor. * Refactored and added docs. * Created tests and fixed failing ones due to updated solr definition * Synchronized submission-forms_cs.xml with the original-english one * Added docs about magic constants * Added doc why the handle is updated to canonical form in the searchbyHandle endpoint * Allow searching only within the solr indexes or JSON files permitted by the configuration. * Removed normalization of handle prefix because there could be more prefixes. Expect only handle value. * Fixed if condition and config property default value. * Fixed integration tests. Allowed autocomplete custom was missing in the test cfg. * The suggestion endpoint is allowed only for authorized users * Refactored method for the normalizing the query for the discoverQuery and added unit tests. * UFAL/Changed position of rows in submission-forms.xml following v5 (#802) * Changed position of rows in submission-forms.xml following v5 * Fixed the alignment of some text * Removed rows which are not in the v5 * Removed license selector from the `teaching` and `clariah-data` collections * Updated input differences in the submission-forms.xml following the v5. --------- Co-authored-by: Juraj Roka <[email protected]> Co-authored-by: milanmajchrak <[email protected]> * Show db connection statistics in the log file or the `dbstatistics` endpoint (#815) * Show db statistics in the log file or the `dbstatistics` endpoint * Finding out why github checks are failed - undo hibernate.cfg * Disabled automatic logging * Use scheduled CRON job instead of PostConstruct * hibernate generating property true --------- Co-authored-by: Paurikova2 <[email protected]> * Translation of submission-forms to _cs (#816) * Translation of submission-forms to _cs * Translated bitstream metadata and complex input fields * Translated the rest of submission-froms_cs.xml * Fixed regex... it must contain regex value, not the message. --------- Co-authored-by: Juraj Roka <[email protected]> Co-authored-by: milanmajchrak <[email protected]> * Updated cfg to pretify the author suggestions (#819) * crosswalk-embargo (#821) * added fn for embargo * using of res policy end_date and added comments * fix string format problem with %s * integration tests are falling down * checkstyle violations * removed findHandle duplicity * added deleted line * checkstyle violations * For now the complex input field is without autocomplete for the size and contact person (#823) * Send the custom type bind `field` to the FE configuration (#822) * fix date converting to string (#825) * fix date converting to string * made const from format * checkstyle * cherry-pick clarin v7 into dtq dev (#820) * cherry-picked DataCite related changes from customer/uk * Add a script that adds a file from url to an item intended for large file workflows * Add ways to influence the bitstream name * add more options to specify an item * Expose resourceId (#1134) A BE part of #1127 - this exposes the resourceId so it can be used in the handle mgmt table * fixes ufal#1135 - findEpersonByNetId should stop searching when it finds an eperson - moved the `return eperson` inside the for cycle (after eperson non null check). - removed the eperson param (both callers were passing in `null`) * Test release without db logs (#827) * UFAL/Matomo statistics with dimension (#813) * Updated the version of matomo dependency and tried to change request from Custom Variables to Dimension * Added a custom dimension with item's handle URL * Send custom dimension also in oai tracker * Use only IPv4 address, the Matomo tracker has a problem with IPv6 * Do not change custom dimension when the Item is null * First custom dimension should have ID '1'. * Use a valid URL for Matomo tracker in the IT * Configure handle custom dimension ID in the clarin-dspace.cfg * Refactored ipv4 method to be more readable - return null --------- Co-authored-by: Juraj Roka <[email protected]> Co-authored-by: milanmajchrak <[email protected]> Co-authored-by: milanmajchrak <[email protected]> * The `dspace.name.short` is not working properly in the email, use `dspace.shortname` instead (#828) --------- Co-authored-by: Tim Donohue <[email protected]> Co-authored-by: Paurikova2 <[email protected]> Co-authored-by: Jozef Misutka <[email protected]> Co-authored-by: jurinecko <[email protected]> Co-authored-by: Juraj Roka <[email protected]> Co-authored-by: Paurikova2 <[email protected]> Co-authored-by: Ondřej Košarko <[email protected]>
milanmajchrak
added a commit
to dataquest-dev/DSpace
that referenced
this pull request
Feb 4, 2025
* UFAL/Removed duplicities of the bitstreams in the cmdi (#766) * Removed duplicities of the bitstreams in the cmdi. * used lindat code instead of vanilla. * Ufal/Preview issues (#764) * Ensure the content preview doesn't overload maximum length of the column in the database. And encode the input stream into UTF-8. * Do not store HTML content in the database because it could be longer than the limit of the database column. * UFAL/Encoded the UTF-8 characters from the redirect URL to UTF (#758) * Encoded the UTF-8 characters from the redirect URL to UTF * Moved ClarinUtils into Utils class * Added a new `dq` package into ComponentScan * Moved dq.Utils into DSpace utils.Utils because the components with the same name causes conflicts. * Removed *.dq component scan from the App * Merge pull request DSpace#9790 from DSpace/backport-9775-to-dspace-7_x (#769) [Port dspace-7_x] Make statistics autocommit much more frequently Co-authored-by: Tim Donohue <[email protected]> * test for bitstream with null value of sizebytes * Update README.md * UFAL/Shibboleth - load more net-id headers e.g. persistent-id (#772) * Load netid from more than one header. authentication-shibboleth.netid-header could be list, not only single value * Shibboleth login - sort the emails passed in the shibboleth email header and get the first one. * The user is redirected to the login page when it is trying to update eperson email which is already assigned to another eperson. * Sorting emails is moved into specific method and ShibbolethLoginFilter is updated following the ShibAuthentication changes * The ClarinShibbolethLoginFilter and ClarinShibAuthentication has duplicate code, I put it into static method. * Propagate the verification token to the DB after the email is successfully sent. (#786) * UFAL/Enhanced type-bind feature (#762) * type bind is correctly rendered in the FE, but BE is still not working * Synchronized the `submission-forms_cs.xml` * Added doc into `submission-forms` about enhanced type-bind `field` * Updated `local.cfg` for tests - added type-bind property * Updated docs for the customized type-bind configuration property. * Updated MetadataValidation following the type-bind customization. * Added isAllowed function for multiple type-bind definitions * Added some docs for the new method * The values of the input wasn't loaded. * Allowed fields could be empty when they should have values. * Used isEmpty function and created constant for the `=>`. * create preview content for tar files (#759) * create preview content for tar files * Added right logs * devided extractFile funs into several separated smaller funs * added comment and removed empty line * added empty lines and removed unwanted comments * removed empty line * used consts * try incorrect identification level * log errors and removed unneeded consts --------- Co-authored-by: milanmajchrak <[email protected]> * Internal/fix failing Clarin integration test (#796) * Initial commit * Ignore the test class from where the tests has started failing. * Ignored half of tests in the ClarinShibbolethLoginFilterIT file * Ignored all tests * unignore some tests * 3 tests ignored 3 allowed * Maybe the problematic test is hidden between 3 unignored tests * two candidates * The last candidate * Ignore just wrong test and all test should pass * Clean up object created in the test. * Removed unused import. * Check the user which is going to be deleted is not null. * Rest api for handle resolution with metadata * decoded rawvalues and response json modification * used static extractMetadata funct in HandlePlugin * return dict: * removed property for test from local * Add default licenses - from ZCU update (#801) * Added flyway file to insert default licenses with license labels and mappings * Added required header * UFAL/share submission by email (#780) * Updated table workspace with share token, created endpoint to generate share token and it is sent via email * Added method to get workspaceitem via share token. * Added an endpoint for changing the submission's owner. * Added license headers * Added test for fetching item with share Token * Added tests to check the owner is changed * Added better explanation why the BE must return Page object in the search endpoint * Validate the user in the SubmissionController, it cannot be null * Updated email - some values are fetched directly from the configuration property * Updated preAuthorization method to ADD instead of WRITE (write is used for controlling authorization for modifying the Item) and updated shareURL * Authorize the submitter which is trying to take sharing item via shareToken. * Update integration test following the feature update * Import default licenses only if the license tables are empty. (#808) * Oai elg crosswalk (#798) * problem with language code * fix amount and sizeUnitOther rest-tests errors * added language coding * new language and funding project mishmash array position * added isoCodes * removed handle from item submission (#812) * UFAL/Shibboleth - netid-header should use getArrayProperty everywhere (#807) * Fetch netid as array from the cfg. Now netid as array is used everywhere. Added integration test to ask for an email when the user send only persistent-id in the shib header. * The user is not signed in without using link with the verification token from the email/ (#809) * UFAL/Shibboleth - show error in the UI when shibboleth authentication is failed (#810) * The user is not signed in without using link with the verification token from the email/ * Send a redirect to UI with specific parameter that the Shibboleth authorization wasn't successful * UFAL/Autocomplete enhancement (#768) * Added solr index `handle_title_ac` and `_comp` for the Item * Added support for searching results from specific solr indexes. 1. Updated submission-forms autocomplete definition to specify a specific index. 2. Updated configuration is provided via REST API. 3. Create a new `/suggestions` endpoint for searching values from custom solr index - it returns VocabularyEntry page. * Supported searching Item byHandle when passed a handle as parameter without handle canonical prefix. * Added autocompleteCustom `solr-subject_ac` and `handle_title_ac`. * Added autocompleteCustom `solr-publisher_ac`. * Added cfg property to define a separator from the solr value to get only display value. * Added autocompleteCustom `solr-dataProvider_ac` * Refactored code and created integration test for the SuggestionRestController * Updated suggestion integration tests because it has had a conflict with another IT * Added doc and changed `autocomplete.custom.format` to `autocomplete.custom.separator` for proper naming. * Added support for loading suggestions from the json file - need to refactor. * Refactored and added docs. * Created tests and fixed failing ones due to updated solr definition * Synchronized submission-forms_cs.xml with the original-english one * Added docs about magic constants * Added doc why the handle is updated to canonical form in the searchbyHandle endpoint * Allow searching only within the solr indexes or JSON files permitted by the configuration. * Removed normalization of handle prefix because there could be more prefixes. Expect only handle value. * Fixed if condition and config property default value. * Fixed integration tests. Allowed autocomplete custom was missing in the test cfg. * The suggestion endpoint is allowed only for authorized users * Refactored method for the normalizing the query for the discoverQuery and added unit tests. * UFAL/Changed position of rows in submission-forms.xml following v5 (#802) * Changed position of rows in submission-forms.xml following v5 * Fixed the alignment of some text * Removed rows which are not in the v5 * Removed license selector from the `teaching` and `clariah-data` collections * Updated input differences in the submission-forms.xml following the v5. --------- Co-authored-by: Juraj Roka <[email protected]> Co-authored-by: milanmajchrak <[email protected]> * Show db connection statistics in the log file or the `dbstatistics` endpoint (#815) * Show db statistics in the log file or the `dbstatistics` endpoint * Finding out why github checks are failed - undo hibernate.cfg * Disabled automatic logging * Use scheduled CRON job instead of PostConstruct * hibernate generating property true --------- Co-authored-by: Paurikova2 <[email protected]> * Translation of submission-forms to _cs (#816) * Translation of submission-forms to _cs * Translated bitstream metadata and complex input fields * Translated the rest of submission-froms_cs.xml * Fixed regex... it must contain regex value, not the message. --------- Co-authored-by: Juraj Roka <[email protected]> Co-authored-by: milanmajchrak <[email protected]> * Updated cfg to pretify the author suggestions (#819) * crosswalk-embargo (#821) * added fn for embargo * using of res policy end_date and added comments * fix string format problem with %s * integration tests are falling down * removed findHandle duplicity * For now the complex input field is without autocomplete for the size and contact person (#823) * Send the custom type bind `field` to the FE configuration (#822) * fix date converting to string * cherry-picked DataCite related changes from customer/uk * Add a script that adds a file from url to an item * intended for large file workflows * Add ways to influence the bitstream name * add more options to specify an item * Expose resourceId (#1134) A BE part of #1127 - this exposes the resourceId so it can be used in the handle mgmt table * fixes ufal#1135 - findEpersonByNetId should stop searching when it finds an eperson - moved the `return eperson` inside the for cycle (after eperson non null check). - removed the eperson param (both callers were passing in `null`) * Test release without db logs (#827) * UFAL/Matomo statistics with dimension (#813) * Updated the version of matomo dependency and tried to change request from Custom Variables to Dimension * Added a custom dimension with item's handle URL * Send custom dimension also in oai tracker * Use only IPv4 address, the Matomo tracker has a problem with IPv6 * Do not change custom dimension when the Item is null * First custom dimension should have ID '1'. * Use a valid URL for Matomo tracker in the IT * Configure handle custom dimension ID in the clarin-dspace.cfg * Refactored ipv4 method to be more readable - return null --------- Co-authored-by: Juraj Roka <[email protected]> Co-authored-by: milanmajchrak <[email protected]> Co-authored-by: milanmajchrak <[email protected]> * The `dspace.name.short` is not working properly in the email, use `dspace.shortname` instead (#828) * Assinging to JAVA_OPTS_HANDLE must be without $ * Transaction bug - close context in finally block (#845) (#852) * Transaction bug - see differencies (#845) * Logging hibernate statistics doesn't work as expected. * Undo connection pool timeout contants * Removed a huge amount of log.info * Added abort to try catch blocks to check if it will solve IT * Call aborting the context in the finally block of the StatelessAuthenticationFilter --------- Co-authored-by: jm <jm@maz> * Empty commit to run actions * Added a warning message then the transaction is alive during initializing a new Context. * Updated the `actions/upload-artifact` to v4 becausa of `https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/` * Show warning when the transaction is alive when the dbConnection is created --------- Co-authored-by: jm <jm@maz> * metadata-added-provenance (#730) * added provenance metadata for update, remove and add * added provenance metadata for update and remove bitstream * item test json added metadata provenance * control provenance only by tests of item * update and remove license * provenance access control and upload bitstream * provenance move item to different collection * provenance make item non-discoverable * provenance for mapped collection * don't write provenance for bitstream without item * move addProvenance to parent class * separated class for provenance * create class for provenance management * added metadata item and bitstream tests * tests for metadata provenance * problem with access control test * better service management: * delete clarin license mapping * added json to resources * modified provenance patch messages * messages bags * used correct json * log exception, replace ! by non conds, added doc comments * added logs for replacing mtd * make object from provenance service * removed interface prom provenance msg provider * Refactored method `removeReadPolicies` - add provenance after removing policy * The message templates are loaded from the Enum instead of json file. * The `getMessage` methods was overloaded. * Renamed `getItem` to `findItemByBitstream` to clarify the method's purpose. * Do not throw exception when adding to provenance * The exceptions are caught in the ProvenanceProvider and changed order when calling provenance methods. * Renamed ProvenanceProvider to ProvenanceService and refactored it to be used as Spring Service. * The ProvenanceService wasn't initialized in the BundleAccessControl * Removed the code without any effect * Removed the logging which is not required * Moved calling od the provenance service to better place. --------- Co-authored-by: milanmajchrak <[email protected]> * Facet result for suggestion (#854) * Retrieve the suggestions from the facet results instead of a specific bunch of results * Removed not required log. * Healthcheck process (#853) * Checkouted files from the `healthcheck-process` branch and updated the code, also added integration test. * Changed the name of the test. * Updated the description to be consistent with another descriptions. * Refactored the calling the check.run method, use `check.report` instead. * Changed matomo url to some valid URL * Update the url to some non-existing url. * Oai indexing after res policy change (#855) * Fixed nullpointer exception - when the dso object was null and was accessed. * The condition was added to the boolean property. * Run build action every 4 hours to be sure tests are still passing. * The dc.type should not be repeatable (#861) * UFAL/Health report send to more email adresses Co-authored-by: Matus Kasak <[email protected]> * UFAL/Fixed missing `*` character in the required input field * Res policy for submitter (#862) * created res policy if eperson is submitter * allowed submitter edit metadata only in collections from cfg * test for ID collection allowed by cfg * created res policy during item installation, tests * added check if user is submitter * renamed property, default value if is not defined * added comments why the property is not commented * Do not merge flyway script with default licenses. * Removed unuseful comment --------- Co-authored-by: Tim Donohue <[email protected]> Co-authored-by: Paurikova2 <[email protected]> Co-authored-by: Jozef Misutka <[email protected]> Co-authored-by: jurinecko <[email protected]> Co-authored-by: Juraj Roka <[email protected]> Co-authored-by: Paurikova2 <[email protected]> Co-authored-by: Ondřej Košarko <[email protected]> Co-authored-by: jm <jm@maz> Co-authored-by: Kasinhou <[email protected]> Co-authored-by: Matus Kasak <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Port of #9775 by @mwoodiupui to
dspace-7_x.