Add category moderation groups and more attributes to generic importer #32561

RubenOussoren · 2025-05-02T15:25:58Z

This PR improves the generic bulk import and base scripts by adding support for several previously unimplemented features and attributes:

Category Moderation Groups: Implements the import of group-based moderation settings for categories.
Group Attributes: Adds import support for public_admission, public_exit, allow_membership_requests, and assignable_level.
User Attributes: Adds import support for trust_level and primary_group_id.
User Option: Adds import support for the hide_profile_and_presence user option.
Multiple Tag Groups per Tag: Updates tag import logic to handle tags belonging to multiple tag groups.

…user option, and multi-tag-group support

…ricted flags

script/bulk_import/generic_bulk.rb

…import script

s3lase · 2025-05-07T10:58:20Z

script/bulk_import/base.rb

@@ -508,6 +511,7 @@ def discourse_reaction_id_from_original_id(id)
    messageable_level
    created_at
    updated_at
+    assignable_level


Does this currently work? I don't see assignable_level in the groups table

s3lase · 2025-05-07T11:04:57Z

script/bulk_import/base.rb

+    public_admission
+    public_exit
+    allow_membership_requests


Could you also extend the process_group method to set a default value for these if they're not present? In its current form, it’ll fail for intermediate DBs generated by other converters that don’t explicitly set these values.

s3lase · 2025-05-07T11:13:45Z

script/bulk_import/generic_bulk.rb

      next if existing_group_user_ids.include?([group_id, user_id])

-      { group_id: group_id, user_id: user_id }
+      { group_id: group_id, user_id: user_id, owner: row["owner"] }


Let's extend process_group_user to set default value for owner

s3lase · 2025-05-07T12:16:33Z

script/bulk_import/generic_bulk.rb

@@ -562,6 +631,7 @@ def import_user_options
        email_level: row["email_level"],
        email_messages_level: row["email_messages_level"],
        email_digests: row["email_digests"],
+        hide_profile_and_presence: row["hide_profile_and_presence"] || false,


There seems to be some work to make this setting more granular by splitting it into hide_profile and hide_presence, but hide_profile_and_presence still appears to be kept in sync as follows:

discourse/app/models/user_option.rb

Lines 228 to 235 in 4117b08

def update_hide_profile_and_presence

if hide_profile_changed? || hide_presence_changed?

self.hide_profile_and_presence = hide_profile || hide_presence

elsif hide_profile_and_presence_changed?

self.hide_profile = hide_profile_and_presence

self.hide_presence = hide_profile_and_presence

end

end

Simply setting hide_profile_and_presence might be problematic. You could:

Set both hide_profile and hide_presence to the value of hide_profile_and_presence, if it’s explicitly provided in process_user_options

Update the converter framework to support to also support hide_profile and hide_presence and replicate the logic above to keep them in sync

s3lase · 2025-05-07T12:23:54Z

script/bulk_import/generic_bulk.rb

+      group_id = group_id_from_imported_id(row["group_id"])
+
+      next unless category_id && group_id
+      next if existing_moderation_groups.include?([category_id, group_id])


Suggested change

next if existing_moderation_groups.include?([category_id, group_id])

next unless existing_moderation_groups.add?([category_id, group_id])

s3lase · 2025-05-07T12:41:15Z

script/bulk_import/generic_bulk.rb

@@ -77,6 +77,8 @@ def execute
    import_category_tag_groups
    import_category_permissions
    import_category_users
+    import_category_moderation_groups
+    update_read_restricted_flags


Tiny nit, for clarity, could the method name explicitly mention category, even though it's colocated? Maybe update_category_read_restricted?. Makes it more obvious at first glance and consistent with other similar steps

s3lase · 2025-05-07T13:36:00Z

script/bulk_import/generic_bulk.rb

+      .find_each do |category|
+        processed_count += 1
+
+        permissions = category.permissions_params


The way permissions_params is currently implemented triggers additional DB queries even when both category_groups and group are preloaded. Since it's a short method, it wouldn’t hurt to inline it.

Category.includes(category_groups: :group).find_each do |category| # ... Do stuff permissions = {} category.category_groups.each do |category_group| group = category_group.group permissions[category_group.group_name] = category_group.permission_type if group.present? end # ... Do more stuff end

s3lase · 2025-05-07T13:46:13Z

script/bulk_import/generic_bulk.rb

@@ -247,7 +249,6 @@ def import_categories
        parent_category_id:
          row["parent_category_id"] ? category_id_from_imported_id(row["parent_category_id"]) : nil,
        slug: row["slug"],
-        read_restricted: row["read_restricted"],


This is still referenced in here and there. It's redundant since we're moving to computing the value instead

s3lase · 2025-05-07T14:03:17Z

script/bulk_import/generic_bulk.rb

+      if row["tag_group_id"] && !row["tag_group_id"].empty?
+        intermediate_group_ids = JSON.parse(row["tag_group_id"])
+
+        intermediate_group_ids.each do |intermediate_group_id|
+          discourse_tag_group_id = @tag_group_mapping[intermediate_group_id]
+
+          if discourse_tag_group_id
+            TagGroupMembership.find_or_create_by!(
+              tag_id: tag.id,
+              tag_group_id: discourse_tag_group_id,
+            )
+          else
+            puts "Warning: Intermediate tag group ID #{intermediate_group_id} from row not found in @tag_group_mapping for tag '#{tag.name}'"
+          end
+        end


Did you update tag_group_id in the converter to a JSON column?

You’d typically expect tag_group_id to be an integer. If we want to support multiple tag group IDs, it might be better to add a separate tag_group_ids column and leave tag_group_id as is since we don’t have time to update all converters right now. We can just handle both columns here for now

feat(import): Add category moderation groups, user/group attributes, …

07d14d5

…user option, and multi-tag-group support

RubenOussoren requested a review from s3lase May 2, 2025 15:25

RubenOussoren self-assigned this May 2, 2025

RubenOussoren requested a review from a team as a code owner May 2, 2025 15:25

github-actions bot added the migrations-tooling PR which includes changes to migrations tooling label May 2, 2025

RubenOussoren added 3 commits May 5, 2025 12:28

feat(read_restricted): created a new method to properly set read_rest…

c3a45fe

…ricted flags

fix added method call

877c303

fix removed old read_restricted references from import_categories

306024e

s3lase reviewed May 5, 2025

View reviewed changes

script/bulk_import/generic_bulk.rb Show resolved Hide resolved

Refactor: Use core Category methods for read_restricted flag in bulk …

4117b08

…import script

s3lase requested changes May 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add category moderation groups and more attributes to generic importer #32561

Add category moderation groups and more attributes to generic importer #32561

RubenOussoren commented May 2, 2025

s3lase May 7, 2025

s3lase May 7, 2025

s3lase May 7, 2025

s3lase May 7, 2025

s3lase May 7, 2025

s3lase May 7, 2025

s3lase May 7, 2025

s3lase May 7, 2025

s3lase May 7, 2025

	def update_hide_profile_and_presence
	if hide_profile_changed? \|\| hide_presence_changed?
	self.hide_profile_and_presence = hide_profile \|\| hide_presence
	elsif hide_profile_and_presence_changed?
	self.hide_profile = hide_profile_and_presence
	self.hide_presence = hide_profile_and_presence
	end
	end

	next if existing_moderation_groups.include?([category_id, group_id])
	next unless existing_moderation_groups.add?([category_id, group_id])

Add category moderation groups and more attributes to generic importer #32561

Are you sure you want to change the base?

Add category moderation groups and more attributes to generic importer #32561

Conversation

RubenOussoren commented May 2, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment