import / export

The import and export of a full realm(s), including users, groups, etc. to a single file is problematic - the maximum REST payload for a single body is currently limited to 10 MB and as touched on in #40798 both via the api and on the filesystem we have memory issues because we parse the entire contents into memory. We may want to limit or even discourage the usage of such formats - making exceptions only in cases where usability, performance, or transactionality would be compromised.

I don't recall if we have committed in admin api v2 to supporting a realm respresentation that contains all, or a configurable set, of sub-resources - see below for more on partial imports.

While pragmatically fine in a POST, to keep things symetrical we should not allow sub-resources to appear in POST, PUT, or PATCH. Appearing (or not) in a PUT, can be interpretted as an overwrite of the sub-resources, which will make the management of the top-level resource difficult or start to muddle PUT and PATCH operations.

update realm (PUT .../realms/name)

Using some of a realm representation payload this currently provides an operation that is similar to a json merge patch - that is most fields appear to be modified only if they are specified in the payload body, and applicable collections (attributes) are overwriten entirely. Also allows for the realm name to be changed.

Given that this is not well documented and has non-standard semantics we should not persue additional changes, such as #25326, and instead focus on deprecating this behavior as soon as possible. This will be superceded by a realm PUT or PATCH (json merge patch, or json patch) operation in the admin api v2.

partial import / export

Partial import appears to compliment update realm and supports modifying some collections under the realm, but not the top-level realm fields.

The payload is similar to a realm representation, but supports only a subset of fields, and has additional policy and ifResourceExists fields.

It also does no removals, only additions or overwrites - so it is also similar to a json merge path, but does not do removals of collections.

Similar to update realm this is not well documented and has non-standard semantics. We should probably not persue additional changes, such as #41436, and instead focus on deprecating this behavior as soon as possible.

The applicable set of resources in admin api v2 will be targetable by PUT and PATCH (initially json merge patch, or json patch). However working with collections from a parent resource PATCH in the prototype is currently difficult. With both json patch and json merge patch there are usability issues wrt to array / collection handling. With json merge patch you must completely specify the array - selectively adding or removing items is not possible. With json patch there is no understanding of identity in a collection beyond an index, so there is no better approach to add / replace / remove than to require stable sorting w/ opputunistic locking and to compute the necessary indexes on the client side when building the patch.

Primarily for performance reasons we may want to consider adding end points for bulk operations - for example a PATCH type / common endpoint that allows for upserting a list of resources. Anything that needs to be acted upon in this manner needs to its own sub-resource - this at least includes users, groups, clients, identityProvider, and identityProviderMappers.

Obviously spanning a partial import over multiple requests runs the risk of an incomplete application. While partial import seems to be for users who are opting for a UI driven (non-declaratively managed) flow to updating resources and propogating that to another environment, it's not yet clear how much we'll expect operators to sync in way that is transactional - or if we can safely assume an ability to reach the desired runtime state via a well-known ordering of updates with necessary ability to retry.

Alternatively we can more generally consider an additional patch format - similar to the Kubernetes strategic /enhanced json merge patch. While that format is non-standard, from a parent resource it does allow for merging arrays / collection based upon merge keys - but things get complicated if there are collections that do not have keys.

UI support is TBD depending upon the exact endpoints supported. A zip artifact could be considered is this will span multiple payloads and we don't want to introduce a specialized coordinating file.

cc @pedroigor @ssilvert @vmuzikar @mabartos @Pepo48 - are there any qualms with closing out the long-standing issues logged against update realm and partial import / export?