Solving #690

Description

This pull request adds Load_partitions boolean switch to the Metastore creation form for HMSMetastoreLoader and HMSThriftMetastoreLoader.

Solution

I've noticed that this switch is already available in GlueDataCatalogLoader, so I've decided to extract the form field to a new module and reuse it from there in all three loaders.

Backwards compatibility

• If I just tried to change the form and opened a Metastore page that uses HMSMetastoreLoader, the front-end returned following error: [Immer] Immer only supports setting array indices and the 'length' property caused by an array of URLs still present in the DB for the loader in metastore_params field.

  • Therefore, I've included transformation for HMSMetastoreLoader loader that changes metastore_params into proper dictionary in data migrations.

• If there is an existing metastore that uses HMSMetastoreLoader or HMSThriftMetastoreLoader, it has a behavior of loading all partitions right now. After applying changes from this pull requests, Load_partitions would be turned off by default.

  • Therefore, I've included logic that adds "load_partitions": True in metastore_params for these two loaders in data migrations.

Another solution would be to create completely new loader classes (HMSMetastoreLoaderV2, HMSThriftMetastoreLoaderV2) that would not infer with the current DB data. But this approach could lead to big number of very similar loaders after some time, so I wasn't sure about it.