Table 'promgen_ruleannotation' is truncated after migrate command on 0.58.x #485
Description
Some context for this is that the docker Promgen image connects to a MySQL database on the host.
I had earlier experienced an issue re:
# docker exec -it 194fa5c8c5c9 promgen migrate
SystemCheckError: System check identified some issues:
ERRORS:
promgen.Rule: (fields.E180) MySQL does not support JSONFields.
promgen.Rule: (fields.E180) MySQL does not support JSONFields.
But it was resolved by updating the MySQL DB version from 5.6 -> 5.7.44 (ref: "JSONField is supported on MariaDB 10.2.7+, MySQL 5.7.8+, Oracle, PostgreSQL, and SQLite (with the JSON1 extension enabled)."
When running the newly pulled image (0.58.1/0.58.2), I also run the migrate command (docker exec -it promgen migrate):
Operations to perform:
Apply all migrations: admin, auth, authtoken, contenttypes, promgen, sessions, sites, social_django
Running migrations:
Applying auth.0012_alter_user_first_name_max_length... OK
Applying authtoken.0003_tokenproxy... OK
Applying promgen.0021_shard_load... OK
Applying promgen.0022_rule_labels_annotations... OK
Applying social_django.0009_auto_20191118_0520... OK
Applying social_django.0010_uid_db_index... OK
Applying social_django.0011_alter_id_fields... OK
When a config change is made on the promgen ui, the celery worker outputs an error regarding the promgen_ruleannotation table not existing anymore:
Task promgen.tasks.write_rules[5d8d9910-0925-428b-b254-916d34350526] raised unexpected: ProgrammingError(1146, "Table 'promgendocker2.promgen_ruleannotation
' doesn't exist")
Traceback (most recent call last):
File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/cursors.py", line 312, in _query
db.query(q)
File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/connections.py", line 224, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.ProgrammingError: (1146, "Table 'promgendocker2.promgen_ruleannotation' doesn't exist")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/promgen38/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/promgen38/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/promgen-0.38/promgen/tasks.py", line 114, in write_rules
fp.write(prometheus.render_rules())
File "/usr/src/promgen-0.38/promgen/prometheus.py", line 80, in render_rules
'rulelabel_set',
File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 1625, in prefetch_related_objects
obj_list, additional_lookups = prefetch_one_level(obj_list, prefetcher, lookup, level)
File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 1738, in prefetch_one_level
prefetcher.get_prefetch_queryset(instances, lookup.get_current_queryset(level)))
File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 627, in get_prefetch_queryset
for rel_obj in queryset:
File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
self._fetch_all()
File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
cursor.execute(sql, params)
File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/opt/promgen38/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/cursors.py", line 312, in _query
db.query(q)
File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/connections.py", line 224, in query
_mysql.connection.query(self, query)
django.db.utils.ProgrammingError: (1146, "Table 'promgendocker2.promgen_ruleannotation' doesn't exist")
I had to rollback my promgen database in MySQL and found out that when the migrate command is executed, for whatever reason the promgen_ruleannotation table is truncated from the database:
Before docker migrate:
| auth_group |
| auth_group_permissions |
| auth_permission |
| auth_user |
| auth_user_groups |
| auth_user_user_permissions |
| authtoken_token |
| django_admin_log |
| django_content_type |
| django_migrations |
| django_session |
| django_site |
| promgen_alert |
| promgen_alerterror |
| promgen_alertlabel |
| promgen_audit |
| promgen_defaultexporter |
| promgen_exporter |
| promgen_farm |
| promgen_filter |
| promgen_host |
| promgen_probe |
| promgen_project |
| promgen_prometheus |
| promgen_rule |
| promgen_ruleannotation |
| promgen_rulelabel |
| promgen_sender |
| promgen_service |
| promgen_shard |
| promgen_url |
| social_auth_association |
| social_auth_code |
| social_auth_nonce |
| social_auth_partial |
| social_auth_usersocialauth |
+----------------------------+
36 rows in set (0.01 sec)
After docker migrate:
| auth_group |
| auth_group_permissions |
| auth_permission |
| auth_user |
| auth_user_groups |
| auth_user_user_permissions |
| authtoken_token |
| django_admin_log |
| django_content_type |
| django_migrations |
| django_session |
| django_site |
| promgen_alert |
| promgen_alerterror |
| promgen_alertlabel |
| promgen_audit |
| promgen_defaultexporter |
| promgen_exporter |
| promgen_farm |
| promgen_filter |
| promgen_host |
| promgen_probe |
| promgen_project |
| promgen_prometheus |
| promgen_rule |
| promgen_sender |
| promgen_service |
| promgen_shard |
| promgen_url |
| social_auth_association |
| social_auth_code |
| social_auth_nonce |
| social_auth_partial |
| social_auth_usersocialauth |
+----------------------------+
34 rows in set (0.00 sec)
Is this the expected behaviour? I do not know why the promgen_ruleannoation or the promgen_rulelabel tables would dissapear.
This issue is preventing me from running this version of Promgen (however if #484 gets resolved, this will be a non-issue as well).
Thanks in advanced!