fix publish to store placeholders of proper type #1952

dsenalik · 2024-08-12T20:06:15Z

Bug Fix

Issue #1948

Closes #1956

Tripal Version: 4.x

Description

This PR updates the publish function to

no longer store NULLs in the drupal field tables
no longer add rows to the drupal field table when the field has any property where drupal_store is set to TRUE but there is no value for that property in Chado. This happens for linking fields or property fields.
It also adds a getDefaultValue() to the property type classes that specifies the appropriate type of value to store as a placeholder in the drupal field tables.

Publish now stores these placeholder values in the field tables, which matches exactly what is stored when you enter an entity through the GUI.

Testing?

Get to your psql prompt and do this to create some things to publish

INSERT INTO chado.organism (genus, species) VALUES ('Tripalus', 'bogusii');
INSERT INTO chado.organism (genus, species, infraspecific_name, comment) VALUES ('Tripalus', 'fakerelli', 'luminarus', 'Adding a comment here for testing purposes');
INSERT INTO chado.contact (name) VALUES ('notype');
INSERT INTO chado.contact (name, type_id) VALUES ('hastype', 123);
INSERT INTO chado.pub (title, uniquename, type_id) VALUES ('On the origin of species', 'UID1', (SELECT cvterm_id FROM chado.cvterm WHERE name='Book'));
INSERT INTO chado.pubprop (pub_id, type_id, value) VALUES (2, (SELECT cvterm_id FROM chado.cvterm WHERE name='Author'), 'Charles Darwin');

Publish organism content type -there should be two organisms created.
View both organism pages, there should be no error, and the title should be as expected as the issue described in Empty entity title fields are not getting removed when publishing #1956 is fixed here. You should be able to edit and save them without error. On 4.x the title is incorrectly published as
Enter another organism manually, just genus and species, for comparison.
Look at the field tables and make sure there are no NULLs stored and that the published entity matches the GUI-entered entity. All organisms should have rows in all but two of these tables where cardinality is unlimited:
tripal_entity__organism_dbxref tripal_entity__organism_pub

\pset null '<NULL>'
\pset pager 0

select * from tripal_entity__organism_abbreviation;
select * from tripal_entity__organism_comment;
select * from tripal_entity__organism_common_name;
select * from tripal_entity__organism_genus;
select * from tripal_entity__organism_infraspecific_name;
select * from tripal_entity__organism_infraspecific_type;
select * from tripal_entity__organism_species;
select * from tripal_entity__organism_dbxref;
select * from tripal_entity__organism_pub;
  bundle  | deleted | entity_id | revision_id | langcode | delta | organism_abbreviation_record_id | organism_abbreviation_value 
----------+---------+-----------+-------------+----------+-------+---------------------------------+-----------------------------
 organism |       0 |         1 |           1 | und      |     0 |                               1 | 
 organism |       0 |         2 |           2 | und      |     0 |                               2 | 
 organism |       0 |         3 |           3 | und      |     0 |                               3 | 
(3 rows)

  bundle  | deleted | entity_id | revision_id | langcode | delta | organism_comment_record_id | organism_comment_value 
----------+---------+-----------+-------------+----------+-------+----------------------------+------------------------
 organism |       0 |         1 |           1 | und      |     0 |                          1 | 
 organism |       0 |         2 |           2 | und      |     0 |                          2 | 
 organism |       0 |         3 |           3 | und      |     0 |                          3 | 
(3 rows)

  bundle  | deleted | entity_id | revision_id | langcode | delta | organism_common_name_record_id | organism_common_name_value 
----------+---------+-----------+-------------+----------+-------+--------------------------------+----------------------------
 organism |       0 |         1 |           1 | und      |     0 |                              1 | 
 organism |       0 |         2 |           2 | und      |     0 |                              2 | 
 organism |       0 |         3 |           3 | und      |     0 |                              3 | 
(3 rows)

  bundle  | deleted | entity_id | revision_id | langcode | delta | organism_genus_record_id | organism_genus_value 
----------+---------+-----------+-------------+----------+-------+--------------------------+----------------------
 organism |       0 |         1 |           1 | und      |     0 |                        1 | 
 organism |       0 |         2 |           2 | und      |     0 |                        2 | 
 organism |       0 |         3 |           3 | und      |     0 |                        3 | 
(3 rows)

  bundle  | deleted | entity_id | revision_id | langcode | delta | organism_infraspecific_name_record_id | organism_infraspecific_name_value 
----------+---------+-----------+-------------+----------+-------+---------------------------------------+-----------------------------------
 organism |       0 |         1 |           1 | und      |     0 |                                     1 | 
 organism |       0 |         2 |           2 | und      |     0 |                                     2 | 
 organism |       0 |         3 |           3 | und      |     0 |                                     3 | 
(3 rows)

  bundle  | deleted | entity_id | revision_id | langcode | delta | organism_infraspecific_type_record_id | organism_infraspecific_type_type_id | organism_infraspecific_type_term_name | organism_infraspecific_type_id_space | organism_infraspecific_type_accession 
----------+---------+-----------+-------------+----------+-------+---------------------------------------+-------------------------------------+---------------------------------------+--------------------------------------+---------------------------------------
 organism |       0 |         1 |           1 | und      |     0 |                                     1 |                                   0 |                                       |                                      | 
 organism |       0 |         2 |           2 | und      |     0 |                                     2 |                                   0 |                                       |                                      | 
 organism |       0 |         3 |           3 | und      |     0 |                                     3 |                                   0 |                                       |                                      | 
(3 rows)

  bundle  | deleted | entity_id | revision_id | langcode | delta | organism_species_record_id | organism_species_value 
----------+---------+-----------+-------------+----------+-------+----------------------------+------------------------
 organism |       0 |         1 |           1 | und      |     0 |                          1 | 
 organism |       0 |         2 |           2 | und      |     0 |                          2 | 
 organism |       0 |         3 |           3 | und      |     0 |                          3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | organism_dbxref_record_id | organism_dbxref_linker_id | organism_dbxref_link | organism_dbxref_dbxref_id | organism_dbxref_dbxref_db_id | organism_dbxref_dbxref_accession | organism_dbxref_dbxref_version | organism_dbxref_dbxref_description | organism_dbxref_dbxref_db_name | organism_dbxref_dbxref_db_description | organism_dbxref_dbxref_db_urlprefix | organism_dbxref_dbxref_db_url 
--------+---------+-----------+-------------+----------+-------+---------------------------+---------------------------+----------------------+---------------------------+------------------------------+----------------------------------+--------------------------------+------------------------------------+--------------------------------+---------------------------------------+-------------------------------------+-------------------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | organism_pub_record_id | organism_pub_entity_id | organism_pub_linker_id | organism_pub_link | organism_pub_pub_id | organism_pub_pub_title | organism_pub_pub_volumetitle | organism_pub_pub_volume | organism_pub_pub_series_name | organism_pub_pub_issue | organism_pub_pub_pyear | organism_pub_pub_pages | organism_pub_pub_miniref | organism_pub_pub_uniquename | organism_pub_pub_type | organism_pub_pub_is_obsolete | organism_pub_pub_publisher | organism_pub_pub_pubplace 
--------+---------+-----------+-------------+----------+-------+------------------------+------------------------+------------------------+-------------------+---------------------+------------------------+------------------------------+-------------------------+------------------------------+------------------------+------------------------+------------------------+--------------------------+-----------------------------+-----------------------+------------------------------+----------------------------+---------------------------
(0 rows)

To test an integer field. Publish contact content type. (The pre-existing "null" contact will get pubished too)
Enter a contact through the GUI for comparison.
Verify the drupal tables and compare published vs. GUI entities.

\pset null '<NULL>'
\pset pager 0

select * from tripal_entity__contact_description;
select * from tripal_entity__contact_name;
select * from tripal_entity__contact_project;
select * from tripal_entity__contact_type;
 bundle  | deleted | entity_id | revision_id | langcode | delta | contact_description_record_id | contact_description_value 
---------+---------+-----------+-------------+----------+-------+-------------------------------+---------------------------
 contact |       0 |         4 |           4 | und      |     0 |                             1 | 
 contact |       0 |         5 |           5 | und      |     0 |                             2 | 
 contact |       0 |         6 |           6 | und      |     0 |                             3 | 
 contact |       0 |         7 |           7 | und      |     0 |                             4 | 
(4 rows)

 bundle  | deleted | entity_id | revision_id | langcode | delta | contact_name_record_id | contact_name_value 
---------+---------+-----------+-------------+----------+-------+------------------------+--------------------
 contact |       0 |         4 |           4 | und      |     0 |                      1 | 
 contact |       0 |         5 |           5 | und      |     0 |                      2 | 
 contact |       0 |         6 |           6 | und      |     0 |                      3 | 
 contact |       0 |         7 |           7 | und      |     0 |                      4 | 
(4 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | contact_project_record_id | contact_project_entity_id | contact_project_linker_id | contact_project_link | contact_project_project_id | contact_project_project_name | contact_project_project_description 
--------+---------+-----------+-------------+----------+-------+---------------------------+---------------------------+---------------------------+----------------------+----------------------------+------------------------------+-------------------------------------
(0 rows)

 bundle  | deleted | entity_id | revision_id | langcode | delta | contact_type_record_id | contact_type_type_id | contact_type_term_name | contact_type_id_space | contact_type_accession 
---------+---------+-----------+-------------+----------+-------+------------------------+----------------------+------------------------+-----------------------+------------------------
 contact |       0 |         4 |           4 | und      |     0 |                      1 |                    0 |                        |                       | 
 contact |       0 |         5 |           5 | und      |     0 |                      2 |                    0 |                        |                       | 
 contact |       0 |         6 |           6 | und      |     0 |                      3 |                    0 |                        |                       | 
 contact |       0 |         7 |           7 | und      |     0 |                      4 |                    0 |                        |                       | 
(4 rows)

To test a boolean field and property fields, publish the Publication content type.
Manually enter a publication for comparison.
Verify. Note that the default for a boolean field is f.

\pset null '<NULL>'
\pset pager 0

select * from tripal_entity__pub_abstract;
select * from tripal_entity__pub_authors;
select * from tripal_entity__pub_citation;
select * from tripal_entity__pub_dbxref;
select * from tripal_entity__pub_doi;
select * from tripal_entity__pub_eissn;
select * from tripal_entity__pub_elocation;
select * from tripal_entity__pub_is_obsolete;
select * from tripal_entity__pub_issn;
select * from tripal_entity__pub_issue;
select * from tripal_entity__pub_journal_abbreviation;
select * from tripal_entity__pub_journal_country;
select * from tripal_entity__pub_language;
select * from tripal_entity__pub_language_abbr;
select * from tripal_entity__pub_miniref;
select * from tripal_entity__pub_pages;
select * from tripal_entity__pub_pii;
select * from tripal_entity__pub_publication_date;
select * from tripal_entity__pub_publication_model;
select * from tripal_entity__pub_publication_type;
select * from tripal_entity__pub_publisher;
select * from tripal_entity__pub_pubplace;
select * from tripal_entity__pub_pyear;
select * from tripal_entity__pub_series_name;
select * from tripal_entity__pub_title;
select * from tripal_entity__pub_type;
select * from tripal_entity__pub_uniquename;
select * from tripal_entity__pub_url;
select * from tripal_entity__pub_volume;
select * from tripal_entity__pub_volumetitle;
select * from tripal_entity__study_pub;
 bundle | deleted | entity_id | revision_id | langcode | delta | pub_abstract_record_id | pub_abstract_prop_id | pub_abstract_linker_id | pub_abstract_value | pub_abstract_rank | pub_abstract_type_id 
--------+---------+-----------+-------------+----------+-------+------------------------+----------------------+------------------------+--------------------+-------------------+----------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_authors_record_id | pub_authors_prop_id | pub_authors_linker_id | pub_authors_value | pub_authors_rank | pub_authors_type_id 
--------+---------+-----------+-------------+----------+-------+-----------------------+---------------------+-----------------------+-------------------+------------------+---------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_citation_record_id | pub_citation_prop_id | pub_citation_linker_id | pub_citation_value | pub_citation_rank | pub_citation_type_id 
--------+---------+-----------+-------------+----------+-------+------------------------+----------------------+------------------------+--------------------+-------------------+----------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_dbxref_record_id | pub_dbxref_linker_id | pub_dbxref_link | pub_dbxref_dbxref_id | pub_dbxref_linker_is_current | pub_dbxref_dbxref_db_id | pub_dbxref_dbxref_accession | pub_dbxref_dbxref_version | pub_dbxref_dbxref_description | pub_dbxref_dbxref_db_name | pub_dbxref_dbxref_db_description | pub_dbxref_dbxref_db_urlprefix | pub_dbxref_dbxref_db_url 
--------+---------+-----------+-------------+----------+-------+----------------------+----------------------+-----------------+----------------------+------------------------------+-------------------------+-----------------------------+---------------------------+-------------------------------+---------------------------+----------------------------------+--------------------------------+--------------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_doi_record_id | pub_doi_prop_id | pub_doi_linker_id | pub_doi_value | pub_doi_rank | pub_doi_type_id 
--------+---------+-----------+-------------+----------+-------+-------------------+-----------------+-------------------+---------------+--------------+-----------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_eissn_record_id | pub_eissn_prop_id | pub_eissn_linker_id | pub_eissn_value | pub_eissn_rank | pub_eissn_type_id 
--------+---------+-----------+-------------+----------+-------+---------------------+-------------------+---------------------+-----------------+----------------+-------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_elocation_record_id | pub_elocation_prop_id | pub_elocation_linker_id | pub_elocation_value | pub_elocation_rank | pub_elocation_type_id 
--------+---------+-----------+-------------+----------+-------+-------------------------+-----------------------+-------------------------+---------------------+--------------------+-----------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_is_obsolete_record_id | pub_is_obsolete_value 
--------+---------+-----------+-------------+----------+-------+---------------------------+-----------------------
 pub    |       0 |         8 |           8 | und      |     0 |                         1 | f
 pub    |       0 |         9 |           9 | und      |     0 |                         2 | f
 pub    |       0 |        10 |          10 | und      |     0 |                         3 | f
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_issn_record_id | pub_issn_prop_id | pub_issn_linker_id | pub_issn_value | pub_issn_rank | pub_issn_type_id 
--------+---------+-----------+-------------+----------+-------+--------------------+------------------+--------------------+----------------+---------------+------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_issue_record_id | pub_issue_value 
--------+---------+-----------+-------------+----------+-------+---------------------+-----------------
 pub    |       0 |         8 |           8 | und      |     0 |                   1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                   2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                   3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_journal_abbreviation_record_id | pub_journal_abbreviation_prop_id | pub_journal_abbreviation_linker_id | pub_journal_abbreviation_value | pub_journal_abbreviation_rank | pub_journal_abbreviation_type_id 
--------+---------+-----------+-------------+----------+-------+------------------------------------+----------------------------------+------------------------------------+--------------------------------+-------------------------------+----------------------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_journal_country_record_id | pub_journal_country_prop_id | pub_journal_country_linker_id | pub_journal_country_value | pub_journal_country_rank | pub_journal_country_type_id 
--------+---------+-----------+-------------+----------+-------+-------------------------------+-----------------------------+-------------------------------+---------------------------+--------------------------+-----------------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_language_record_id | pub_language_prop_id | pub_language_linker_id | pub_language_value | pub_language_rank | pub_language_type_id 
--------+---------+-----------+-------------+----------+-------+------------------------+----------------------+------------------------+--------------------+-------------------+----------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_language_abbr_record_id | pub_language_abbr_prop_id | pub_language_abbr_linker_id | pub_language_abbr_value | pub_language_abbr_rank | pub_language_abbr_type_id 
--------+---------+-----------+-------------+----------+-------+-----------------------------+---------------------------+-----------------------------+-------------------------+------------------------+---------------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_miniref_record_id | pub_miniref_value 
--------+---------+-----------+-------------+----------+-------+-----------------------+-------------------
 pub    |       0 |         8 |           8 | und      |     0 |                     1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                     2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                     3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_pages_record_id | pub_pages_value 
--------+---------+-----------+-------------+----------+-------+---------------------+-----------------
 pub    |       0 |         8 |           8 | und      |     0 |                   1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                   2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                   3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_pii_record_id | pub_pii_prop_id | pub_pii_linker_id | pub_pii_value | pub_pii_rank | pub_pii_type_id 
--------+---------+-----------+-------------+----------+-------+-------------------+-----------------+-------------------+---------------+--------------+-----------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_publication_date_record_id | pub_publication_date_prop_id | pub_publication_date_linker_id | pub_publication_date_value | pub_publication_date_rank | pub_publication_date_type_id 
--------+---------+-----------+-------------+----------+-------+--------------------------------+------------------------------+--------------------------------+----------------------------+---------------------------+------------------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_publication_model_record_id | pub_publication_model_prop_id | pub_publication_model_linker_id | pub_publication_model_value | pub_publication_model_rank | pub_publication_model_type_id 
--------+---------+-----------+-------------+----------+-------+---------------------------------+-------------------------------+---------------------------------+-----------------------------+----------------------------+-------------------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_publication_type_record_id | pub_publication_type_prop_id | pub_publication_type_linker_id | pub_publication_type_value | pub_publication_type_rank | pub_publication_type_type_id 
--------+---------+-----------+-------------+----------+-------+--------------------------------+------------------------------+--------------------------------+----------------------------+---------------------------+------------------------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_publisher_record_id | pub_publisher_value 
--------+---------+-----------+-------------+----------+-------+-------------------------+---------------------
 pub    |       0 |         8 |           8 | und      |     0 |                       1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                       2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                       3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_pubplace_record_id | pub_pubplace_value 
--------+---------+-----------+-------------+----------+-------+------------------------+--------------------
 pub    |       0 |         8 |           8 | und      |     0 |                      1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                      2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                      3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_pyear_record_id | pub_pyear_value 
--------+---------+-----------+-------------+----------+-------+---------------------+-----------------
 pub    |       0 |         8 |           8 | und      |     0 |                   1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                   2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                   3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_series_name_record_id | pub_series_name_value 
--------+---------+-----------+-------------+----------+-------+---------------------------+-----------------------
 pub    |       0 |         8 |           8 | und      |     0 |                         1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                         2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                         3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_title_record_id | pub_title_value 
--------+---------+-----------+-------------+----------+-------+---------------------+-----------------
 pub    |       0 |         8 |           8 | und      |     0 |                   1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                   2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                   3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_type_record_id | pub_type_type_id | pub_type_term_name | pub_type_id_space | pub_type_accession 
--------+---------+-----------+-------------+----------+-------+--------------------+------------------+--------------------+-------------------+--------------------
 pub    |       0 |         8 |           8 | und      |     0 |                  1 |                0 |                    |                   | 
 pub    |       0 |         9 |           9 | und      |     0 |                  2 |                0 |                    |                   | 
 pub    |       0 |        10 |          10 | und      |     0 |                  3 |                0 |                    |                   | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_uniquename_record_id | pub_uniquename_value 
--------+---------+-----------+-------------+----------+-------+--------------------------+----------------------
 pub    |       0 |         8 |           8 | und      |     0 |                        1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                        2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                        3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_url_record_id | pub_url_prop_id | pub_url_linker_id | pub_url_value | pub_url_rank | pub_url_type_id 
--------+---------+-----------+-------------+----------+-------+-------------------+-----------------+-------------------+---------------+--------------+-----------------
(0 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_volume_record_id | pub_volume_value 
--------+---------+-----------+-------------+----------+-------+----------------------+------------------
 pub    |       0 |         8 |           8 | und      |     0 |                    1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                    2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                    3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_volumetitle_record_id | pub_volumetitle_value 
--------+---------+-----------+-------------+----------+-------+---------------------------+-----------------------
 pub    |       0 |         8 |           8 | und      |     0 |                         1 | 
 pub    |       0 |         9 |           9 | und      |     0 |                         2 | 
 pub    |       0 |        10 |          10 | und      |     0 |                         3 | 
(3 rows)

 bundle | deleted | entity_id | revision_id | langcode | delta | study_pub_record_id | study_pub_entity_id | study_pub_pub_id | study_pub_pub_title | study_pub_pub_volumetitle | study_pub_pub_volume | study_pub_pub_series_name | study_pub_pub_issue | study_pub_pub_pyear | study_pub_pub_pages | study_pub_pub_miniref | study_pub_pub_uniquename | study_pub_pub_type | study_pub_pub_is_obsolete | study_pub_pub_publisher | study_pub_pub_pubplace 
--------+---------+-----------+-------------+----------+-------+---------------------+---------------------+------------------+---------------------+---------------------------+----------------------+---------------------------+---------------------+---------------------+---------------------+-----------------------+--------------------------+--------------------+---------------------------+-------------------------+------------------------
(0 rows)

We don't have datetime or float fields yet so we can't test those yet 🙈
See this comment for how I tested that this does not break an existing site with published content. fix publish to store placeholders of proper type #1952 (comment)

dsenalik · 2024-08-12T22:46:34Z

We're back to draft status because the new "default values" propagate through to the formatters. I should have anticipated this since the nulls getting through was the original problem.

dsenalik · 2024-08-13T12:59:04Z

This is because publish is creating rows in the field table when there is no value defined. In the GUI, if you leave one of these blank, then there will not be a row in the field table. Publish is creating a row, and the default value in that row (the zero) displays because no value was defined in chado that could replace it.
The solution is to match what happens in the GUI, and not create a field table row when no value is defined in chado.

dsenalik · 2024-08-14T14:13:41Z

I wanted to test that this PR will not break an existing site.

On a 4.x docker I entered a contact and array design

insert into teacup.contact (name, type_id, description) values ('Borg', 42, 'You will be assimilated');
insert into teacup.arraydesign (name, manufacturer_id, platformtype_id, description) values ('AD1', 2, 42, 'Test one');

and published those.

Then I switched to this branch and added two more

insert into teacup.contact (name, type_id, description) values ('Dr. Peaks', 43, 'Science rules');
insert into teacup.arraydesign (name, manufacturer_id, platformtype_id, description) values ('AD2', 3, 43, 'Test two');

and again published.

We look at the drupal field table and see that the records entered while on the 4.x branch have NULL while the record from this branch has the correct empty value ''. There are two contacts on the 4.x branch because the null contact was published also.

sitedb=> \pset null '<null>'
sitedb=> select * from tripal_entity__contact_description;
 bundle  | deleted | entity_id | revision_id | langcode | delta | contact_description_record_id | contact_description_value 
---------+---------+-----------+-------------+----------+-------+-------------------------------+---------------------------
 contact |       0 |         1 |           1 | und      |     0 |                             1 | <null>
 contact |       0 |         2 |           1 | und      |     0 |                             2 | <null>
 contact |       0 |         4 |           1 | und      |     0 |                             3 | 
(3 rows)

The same for the arraydesign table:

sitedb=> select * from tripal_entity__array_design_description;
    bundle    | deleted | entity_id | revision_id | langcode | delta | array_design_description_record_id | array_design_description_value 
--------------+---------+-----------+-------------+----------+-------+------------------------------------+--------------------------------
 array_design |       0 |         3 |           1 | und      |     0 |                                  2 | <null>
 array_design |       0 |         5 |           1 | und      |     0 |                                  3 | 
(2 rows)

This next table shows that after this PR we don't add a row to the table if there is no underlying chado value. The value here was from the first array design on 4.x. On this branch a row was not inserted.

sitedb=> select * from tripal_entity__array_design_num_array_columns;
    bundle    | deleted | entity_id | revision_id | langcode | delta | array_design_num_array_columns_record_id | array_design_num_array_columns_value 
--------------+---------+-----------+-------------+----------+-------+------------------------------------------+--------------------------------------
 array_design |       0 |         3 |           1 | und      |     0 |                                        2 |                               <null>
(1 row)

And Finally

This also fixes the bug where we have a bunch of field headers for fields that don't have a value!

If published on the 4.x branch

If published on this branch

laceysanderson

So I tested the organism as suggested but received unexpected behaviour 🙈

There was no record in tripal_entity__organism_comment for Tripalus bogusii. This makes sense if by "no longer add rows when the underlying Chado record is empty" in the description you meant "doesn't add a record to the drupal table for a field that doesn't have a value in the linked chado record" but doesn't match what you said to expect in the testing instructions.
When you go to the published page, the title replacement no longer works as expected 🙈 Specifically, the title is Tripalus bogusii [organism_infraspecific_type] [organism_infraspecific_name]. This is likely a side effect of the fact that the tripal_entity__organism_infraspecific_name table no longer has an record for the infraspecific name whereas before it did but it was empty.

Note: Interestingly when I edit the organism and then save again without making changes, it fixes the title but does not add a record to the tripal_entity__organism_infraspecific_name or tripal_entity__organism_comment tables 🤔

Note: I did also enter an organism with a comment and can confirm that when the comment is present, the organism_comment_value column is now an empty string as expected --no null :-)

laceysanderson

Ok code review completed :-)

Also, after a cofest discussion, I confirmed that the following issues from my manual test are not caused by this PR:

There was no record in tripal_entity__organism_comment for Tripalus bogusii.

This was just outdated testing instructions 🤪 They have been updated.

When you go to the published page, the title replacement no longer works as expected

This is being fixed in issue #1956 and was not caused by this PR. This PR does exacerbate the issue though as now both type and name are not replaced in the title whereas before this PR it was only name.

tripal/src/TripalStorage/Interfaces/TripalStorageInterface.php

tripal_chado/src/Plugin/TripalStorage/ChadoStorage.php

tripal/src/Services/TripalPublish.php

tripal_chado/tests/src/Functional/Services/ChadoTripalPublishTest.php

tripal_chado/tests/src/Kernel/Services/TripalEntityLookupServiceTest.php

dsenalik · 2024-08-29T00:47:28Z

Here is something I did not understand. For fields with unlimited cardinality, when an entity is created through the GUI, if there is no record, then the drupal field table does not have a row. That's how this PR works now.

HOWEVER, if the field is cardinality 1, then the drupal field table DOES have a row!
The symptom @Lacey found is that if you create this:
INSERT INTO organism (genus, species) VALUES ('Tripalus', 'bugii');
and publish, you cannot edit this and add a comment, common name, or abbreviation.

dsenalik · 2024-08-29T18:48:16Z

Looking up cardinality is easy, but the problem that is happening in this PR is that, in the insertFieldItems() function, the $matches array no longer has records for fields that don't have a value in chado. This is because after adding the "not required" items to the initial search query, these fields are coming back with a NULL for their value. The whole match record then is removed by isFieldValid() and at this point

tripal/tripal/src/Services/TripalPublish.php

Lines 794 to 795 in 3ad2f09

    
           $num_items = count(array_keys($match[$field_name])); 
        
           for ($delta = 0; $delta < $num_items; $delta++) {

there is nothing there for that field.
In 4.x, there was a record in the $match, although it was empty except for the record_id.

laceysanderson · 2024-09-03T16:32:11Z

no longer add rows to the drupal field table when the underlying Chado record has an empty value for that field

I am seeing quite a few bugs related to this in my production site 🤔 Perhaps we shouldn't do this. Rather if there is a chado record for that field then there is a record in drupal even if it is empty.

dsenalik · 2024-09-03T19:22:39Z

I can see how this might make better sense. We would have to figure out how to handle integer fields, though, that have zero as an empty value, which caused the problem in this comment #1952 (comment) - although we could back to storing NULLs now that the formatter won't break from those.

@laceysanderson did you mean your production site problems are on 4.x or testing on this branch? If we abandon this whole thing, well that's okay.

laceysanderson · 2024-09-03T19:56:12Z

@laceysanderson did you mean your production site problems are on 4.x or testing on this branch? If we abandon this whole thing, well that's okay.

Ah yes, I should have been more clear. I have tested this branch on a clone of my production site and I think there were still issues though at this point I'm having trouble determining which issues were on 4.x and which were still on this branch 🤪 I do not want to abandon this all together as I think many of the changes are great! I just think we should go back to always having a drupal record if there is a chado record.

I can see how this might make better sense. We would have to figure out how to handle integer fields, though, that have zero as an empty value, which caused the problem in this comment #1952 (comment) - although we could back to storing NULLs now that the formatter won't break from those.

Ah yes... I think we should leave a default of 0 for integer fields and create a new issue for this. I think I would be tempted to solve it with an option in the formatter for integer fields that lets you indicate if you want to hide entries with a value of 0 or not 🤔 but there may be a better option.

dsenalik · 2024-09-09T15:17:06Z

With commit 6ade77f I have made two significant changes

Lookup cardinality. Any field with cardinality 1 is always added to the drupal field tables. Otherwise, it is only added if the chado value is not empty. This should exactly match the behavior when adding an entity through the gui.
Bypass the isFieldValid() check for publish only. This hopefully fixes what I described in comment fix publish to store placeholders of proper type #1952 (comment) - but I have yet to test thoroughly.

I will now go back and update the testing procedure yet again as I test this ...

dsenalik · 2024-09-09T15:33:07Z

Something to check, what about this NULL in a property field:
select * from tripal_entity__pub_abstract;

 bundle | deleted | entity_id | revision_id | langcode | delta | pub_abstract_record_id | pub_abstract_prop_id | pub_abstract_linker_id | pub_abstract_value | pub_abstract_rank | pub_abstract_type_id 
--------+---------+-----------+-------------+----------+-------+------------------------+----------------------+------------------------+--------------------+-------------------+----------------------
 pub    |       0 |         6 |           1 | und      |     0 |                      1 |               <NULL> |                      1 |                    |                 0 |                    0
(1 row)

and several others, authors, citation, doi...

If you enter a pub with gui and the field is blank, no row is created in the drupal field table for these, even though cardinality is 1 😖

With commit 6ade77f I am checking for cardinality 1 fields where the main property is NULL as opposed to empty, and for these do not create a drupal field table entry.

…al into tv4g1-issue1948-publish-fix

dsenalik · 2024-09-11T23:26:04Z

With commit b2f0fbf I have abandoned using cardinality. Now, add to drupal tables except for the case where there is a property with drupal_store set to TRUE, but there is not a corresponding value. This magically handles everything and I am getting the exact same results as entities entered through the GUI except for revision_id.

"revision id for an unversioned entity type is the same as the entity id"

dsenalik · 2024-09-12T01:31:18Z

I have also fixed revision_id to match how it works for entities entered through the GUI, and updated testing instructions, so my work here is done 🤞

laceysanderson

✅ Code changes look good
✅ Manually tested on a clone of my production site and it works beautifully with our project-based grant, study and experiment pages. It correctly published a complex set of multiple property types, contact relationships, and dbxref associations. It did not break or interact badly with existing content either 😌
✅ Manually tested according to the instructions on a fresh Tripal docker and all looked as expected!

Thanks for all the hard work @dsenalik!!! This is ready to merge!!! 🎉

fix publish to store placeholders of proper type

839d93b

dsenalik added the Group 1 - Tripal Content Types | Terms | Fields Any issue relating to Tripal Content including types, terms, and fields. label Aug 12, 2024

dsenalik mentioned this pull request Aug 12, 2024

Publish bug - nulls in drupal field tables #1948

Closed

dsenalik added 2 commits August 12, 2024 15:19

Fix typo

6a38d60

Merge branch '4.x' into tv4g1-issue1948-publish-fix

c3c3311

dsenalik marked this pull request as draft August 12, 2024 21:02

tests need to be taught that null was wrong

ba1294f

dsenalik marked this pull request as ready for review August 12, 2024 22:29

dsenalik marked this pull request as draft August 12, 2024 22:38

dsenalik added 2 commits August 12, 2024 18:21

revert change to getValue()

4b52f2d

typo fix

102dfab

dsenalik added 5 commits August 13, 2024 15:46

first draft don't add row if chado column is empty

ded424b

fix test to reflect new reality

b868657

reduce complexity with helper function

860ccec

fix wrong logic for add_record

7567e27

fix entity lookup to match new publish changes

349fd74

dsenalik marked this pull request as ready for review August 14, 2024 14:13

dsenalik and others added 3 commits August 14, 2024 09:32

fix datetime even though no field for it yet

ee7c2fd

remove duplicated line

e9fb217

Merge branch '4.x' into tv4g1-issue1948-publish-fix

a5dbe27

laceysanderson mentioned this pull request Aug 23, 2024

don't add to render array if no value #1949

Merged

Merge branch '4.x' into tv4g1-issue1948-publish-fix

b807dfb

laceysanderson requested changes Aug 26, 2024

View reviewed changes

laceysanderson reviewed Aug 26, 2024

View reviewed changes

dsenalik added 3 commits August 26, 2024 13:29

valid tokens should be replaced with empty string

8ea74a4

fix unreplaced title placeholders

e56a19d

Merge branch '4.x' into tv4g1-issue1948-publish-fix

16b365e

laceysanderson and others added 3 commits September 4, 2024 16:39

Merge branch '4.x' into tv4g1-issue1948-publish-fix

5a7a1c3

Merge branch '4.x' into tv4g1-issue1948-publish-fix

44ed257

add cardinality check, skip check valid

6ade77f

check for null on cardinality 1

d9e0569

dsenalik mentioned this pull request Sep 9, 2024

Ensure TripalFields can be added through UI and add some tests #1968

Merged

laceysanderson mentioned this pull request Sep 9, 2024

Empty Values for Boolean + Integer #1970

Closed

dsenalik added 4 commits September 10, 2024 09:23

Merge branch '4.x' into tv4g1-issue1948-publish-fix

2098238

different count after recent changes

5ba6df6

Merge branch 'tv4g1-issue1948-publish-fix' of github.com:/tripal/trip…

28ce1fa

…al into tv4g1-issue1948-publish-fix

abandon cardinality, now check if drupal_store fields have values

b2f0fbf

dsenalik added 4 commits September 11, 2024 20:06

revert commit 5ba6df6

302d0ea

infraspecies will be in drupal field table even if empty

7f474f3

For an unversioned entity, revision_id is the same as the entity_id

1de85a7

revert commit 5ba6df6 fully

0900f51

remove unneeded initialization

1280e13

laceysanderson approved these changes Sep 12, 2024

View reviewed changes

laceysanderson merged commit cc85a50 into 4.x Sep 12, 2024
14 of 15 checks passed

laceysanderson deleted the tv4g1-issue1948-publish-fix branch September 12, 2024 22:09

This was referenced Sep 15, 2024

Null integer fields are published as zeroes #1972

Closed

Publish update - no drupal records when no chado records #1977

Merged

laceysanderson mentioned this pull request Oct 3, 2025

Create the 4.0.0-alpha3 release. #2307

Closed

fix publish to store placeholders of proper type #1952

fix publish to store placeholders of proper type #1952

Uh oh!

Conversation

dsenalik commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bug Fix

Issue #1948

Closes #1956

Tripal Version: 4.x

Description

Testing?

Uh oh!

dsenalik commented Aug 12, 2024

Uh oh!

dsenalik commented Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dsenalik commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I wanted to test that this PR will not break an existing site.

And Finally

If published on the 4.x branch

If published on this branch

Uh oh!

laceysanderson left a comment

Choose a reason for hiding this comment

Uh oh!

laceysanderson left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dsenalik commented Aug 29, 2024

Uh oh!

dsenalik commented Aug 29, 2024

Uh oh!

laceysanderson commented Sep 3, 2024

Uh oh!

dsenalik commented Sep 3, 2024

Uh oh!

laceysanderson commented Sep 3, 2024

Uh oh!

dsenalik commented Sep 9, 2024

Uh oh!

dsenalik commented Sep 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dsenalik commented Sep 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dsenalik commented Sep 12, 2024

Uh oh!

laceysanderson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dsenalik commented Aug 12, 2024 •

edited

Loading

dsenalik commented Aug 13, 2024 •

edited

Loading

dsenalik commented Aug 14, 2024 •

edited

Loading

laceysanderson left a comment •

edited

Loading

dsenalik commented Sep 9, 2024 •

edited

Loading

dsenalik commented Sep 11, 2024 •

edited

Loading